KR20240149809A - Image sensing device and method for image processing - Google Patents

Abstract

An image processing method according to an exemplary embodiment of the present disclosure may include a step of obtaining an RGB-IR pattern image through a pixel array of an RGB-IR pattern, a step of generating a first RGB image including an IR component based on the RGB-IR pattern image, a step of generating a second RGB image from which an IR component is removed based on the RGB-IR pattern image and the first RGB image, a step of generating a third RGB image based on a brightness component of the first RGB image and a color component of the second RGB image, and a step of converting the third RGB image into a Bayer pattern image.

Description

{IMAGE SENSING DEVICE AND METHOD FOR IMAGE PROCESSING}

The technical idea of the present disclosure relates to image processing, and more specifically, to an image sensing device and an image processing method.

Image sensing devices are devices that capture images by utilizing the properties of semiconductors that react to light. Recently, with the development of the computer industry and the communication industry, the demand for high-performance image sensing devices has been increasing in various fields such as smartphones, digital cameras, game devices, the Internet of Things, robots, security cameras, and medical micro cameras. For example, as IR (Infrared) images are widely used in fields such as night surveillance, iris recognition, and face recognition, the demand for RGB-IR image sensing devices is increasing. However, the R pixels, G pixels, and B pixels of the RGB-IR image sensing device can collect not only R, G, and B but also IR, and thus color distortion may occur due to the IR component. Therefore, in order to restore accurate colors using the RGB-IR image sensing device, it is necessary to calculate the IR components collected on the R pixels, G pixels, and B pixels and remove the IR components from the visible light components and IR components, which are the entire collected light components.

The technical problem of the present disclosure is to provide a method and device for effectively removing an IR (Infrared) signal entering an RGB pixel.

The technical problem of the present disclosure is to provide a method and a device for generating an RGB image without color distortion even in an environment where IR exists.

The technical problem of the present disclosure is to provide a method and a device for using an ISP (Image Signal Processor) and an image processing system that process a Bayer pattern image without change by combining an RGB image and an IR image generated by an RGB-IR image sensing device into one.

The technical problem of the present disclosure is to provide a method and device for converting an RGB-IR pattern image into a Bayer pattern image.

The technical problem of the present disclosure is to provide a method and a device for generating an image including an IR component but without color distortion.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs from the description below.

An image processing method according to an exemplary embodiment of the present disclosure may include a step of obtaining an RGB-IR pattern image through a pixel array of an RGB-IR pattern, a step of generating a first RGB image including an IR component based on the RGB-IR pattern image, a step of generating a second RGB image from which an IR component is removed based on the RGB-IR pattern image and the first RGB image, a step of generating a third RGB image based on a brightness component of the first RGB image and a color component of the second RGB image, and a step of converting the third RGB image into a Bayer pattern image.

According to one embodiment, the step of generating the first RGB image may be characterized by including the step of generating the first RGB image by performing interpolation on an R channel, a G channel, and a B channel of an RGB-IR pattern image.

According to one embodiment, the step of generating the first RGB image may further include the step of generating an estimated image by estimating an R pixel value, a G pixel value, or a B pixel value for an IR pixel location of the RGB-IR pattern to correspond to the Bayer pattern, and the step of generating the first RGB image by performing demosaicing on the estimated image.

According to one embodiment, the step of generating the second RGB image may include the step of generating an IR image by performing interpolation for an IR channel based on the RGB-IR pattern image, and the step of generating the second RGB image based on the IR image and the first RGB image.

According to one embodiment, the step of generating the second RGB image may be characterized by including the step of generating a fourth RGB image by removing an IR component from the first RGB image using an IR image, and the step of generating the second RGB image by increasing color saturation of the fourth RGB image.

According to one embodiment, the step of generating the second RGB image may be characterized by including the step of generating a fourth RGB image by removing an IR component from the first RGB image using an IR image, and the step of generating the second RGB image by performing denoising on the fourth RGB image.

According to one embodiment, the step of generating the second RGB image may be characterized by including a step of generating the second RGB image by using a matrix that removes an IR component from the first RGB image by considering crosstalk occurring between R pixels, G pixels, B pixels, and IR pixels included in a pixel array of an RGB-IR pattern, and pixel values of the first RGB image and pixel values of the IR image.

According to one embodiment, the matrix may be characterized as being a matrix obtained by performing regression analysis by setting an RGB image with IR components removed as a target RGB image.

According to one embodiment, the step of generating the third RGB image may include the step of calculating a Y value, which is a brightness component of the first RGB image, the step of calculating a U value and a V value, which are color components of the second RGB image, and the step of calculating an R value, a G value, and a B value using the Y value, the U value, and the V values, thereby generating the third RGB image.

An image sensing device according to an exemplary embodiment of the present disclosure may include an RGB interpolation unit configured to generate a first RGB image by performing interpolation for an R channel, a G channel, and a B channel based on an RGB-IR pattern image generated through a pixel array of an RGB-IR pattern, an IR interpolation unit configured to generate an IR image by performing interpolation for an IR channel based on the RGB-IR pattern image, a brightness component processing unit configured to calculate a brightness component value of the first RGB image, a color component processing unit configured to generate a second RGB image from which an IR component is removed based on the first RGB image and the IR image, and calculate a color component value of the second RGB image, a merging unit configured to generate a third RGB image by merging the brightness component value of the first RGB image and the color component value of the second RGB image, and a pattern conversion unit configured to convert the third RGB image into a Bayer pattern image.

According to one embodiment, the RGB interpolator may be configured to generate an estimated image by estimating an R pixel value, a G pixel value, or a B pixel value for an IR pixel location of an RGB-IR pattern to correspond to a Bayer pattern, and to generate a first RGB image by performing demosaicing on the estimated image.

According to one embodiment, the color component processing unit may be characterized by being configured to generate a fourth RGB image by removing an IR component from a first RGB image using an IR image, and to generate a second RGB image by increasing color saturation of the fourth RGB image.

According to one embodiment, the color component processing unit may be characterized by being configured to generate a fourth RGB image by removing an IR component from a first RGB image using an IR image, and to generate a second RGB image by performing denoising on the fourth RGB image.

According to one embodiment, the color component processing unit may be characterized by being configured to generate a second RGB image using a matrix that removes an IR component from a first RGB image by considering crosstalk occurring between R pixels, G pixels, B pixels, and IR pixels included in a pixel array of an RGB-IR pattern, and pixel values of the first RGB image and pixel values of the IR image.

According to one embodiment, the matrix may be characterized as being a matrix obtained by performing regression analysis by setting an RGB image with IR components removed as a target RGB image.

According to one embodiment, the merging unit may be configured to generate a third RGB image by calculating an R value, a G value, and a B value using a Y value, which is a brightness component value of the first RGB image, and a U value and a V value, which are color component values of the second RGB image.

The features briefly summarized above regarding the present disclosure are merely exemplary aspects of the detailed description of the present disclosure that follows and do not limit the scope of the present disclosure.

According to an image processing method according to an exemplary embodiment of the present disclosure, an IR (Infrared) signal entering an RGB pixel can be effectively removed.

According to an image processing method according to an exemplary embodiment of the present disclosure, an RB image without color distortion can be generated even in an environment where IR exists.

According to an image processing method according to an exemplary embodiment of the present disclosure, an ISP (Image Signal Processor) and an image processing system that process a Bayer pattern image can be used without change by combining an RGB image and an IR image generated by an RGB-IR image sensing device into one.

According to an image processing method according to an exemplary embodiment of the present disclosure, an RGB-IR pattern image can be converted into a Bayer pattern image.

According to an image processing method according to an exemplary embodiment of the present disclosure, an image including an IR component but without color distortion can be generated.

The effects obtainable from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person skilled in the art to which the present disclosure belongs from the description below.

FIG. 1 is a block diagram illustrating an image sensing device according to an exemplary embodiment of the present disclosure.
FIG. 2 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.
FIG. 3 is a drawing for explaining an RGB-IR pattern according to an exemplary embodiment of the present disclosure.
FIG. 4 is a block diagram of an image sensing device according to an exemplary embodiment of the present disclosure.
FIG. 5 is a drawing for explaining an image processing method according to an exemplary embodiment of the present disclosure.
FIG. 6 is a drawing for explaining an image processing method according to an exemplary embodiment of the present disclosure.
FIG. 7 is a drawing for explaining an image processing method according to an exemplary embodiment of the present disclosure.
FIG. 8 is a block diagram illustrating an imaging sensing device according to an exemplary embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

In describing embodiments of the present disclosure, if it is determined that a detailed description of a known configuration or function may obscure the gist of the present disclosure, a detailed description thereof will be omitted. In addition, parts in the drawings that are not related to the description of the present disclosure have been omitted, and similar parts have been given similar drawing reference numerals.

In the present disclosure, when a component is said to be "connected," "coupled," or "connected" to another component, this may include not only a direct connection relationship, but also an indirect connection relationship in which another component exists in between. In addition, when a component is said to "include" or "have" another component, this does not exclude the other component unless specifically stated otherwise, but rather means that the other component may be included.

In this disclosure, the terms first, second, etc. are used only for the purpose of distinguishing one component from another component, and do not limit the order or importance among the components unless specifically stated otherwise. Accordingly, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

In the present disclosure, the components that are distinct from each other are only intended to clearly explain the characteristics of each, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to form a single hardware or software unit, or a single component may be distributed to form a plurality of hardware or software units. Accordingly, even if not mentioned separately, such integrated or distributed embodiments are also included in the scope of the present disclosure.

In the present disclosure, the components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, an embodiment composed of a subset of the components described in one embodiment is also included in the scope of the present disclosure. In addition, an embodiment including other components in addition to the components described in various embodiments is also included in the scope of the present disclosure.

In the present disclosure, expressions of positional relationships used in the present specification, such as top, bottom, left, right, etc., are described for convenience of explanation, and when the drawings illustrated in the present specification are viewed in reverse, the positional relationships described in the specification may be interpreted in the opposite way.

In the present disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

Hereinafter, exemplary embodiments of the present disclosure will be specifically described with reference to FIGS. 1 to 8.

FIG. 1 is a block diagram illustrating an image sensing device according to an exemplary embodiment of the present disclosure.

Referring to FIG. 1, an image sensing device (10) according to an exemplary embodiment of the present disclosure may include an RGB-IR image sensor (11) and a conversion unit (12). An image signal processor (ISP) (13) may be a processor that processes RGB Bayer pattern image data.

The RGB-IR image sensor (11) can output an image by digitizing light collected on a photodiode, which is a silicon surface, through an ADC (Analog-to-Digital Converter). For example, the RGB-IR image sensor (11) can include a pixel array of an RGB-IR pattern, and can generate an RGB-IR pattern image through the pixel array of the RGB-IR pattern.

The R pixel, G pixel, and B pixel of the RGB-IR image sensor (11) can collect near-infrared (IR) light in addition to visible light, and the IR component can distort the RGB color component. In other words, when an RGB image is generated by receiving both light in the RGB wavelength range and light in the IR wavelength range in an environment where IR exists, the color of the RGB image can be distorted. Therefore, a method for generating an RGB image without color distortion even in an environment where IR exists by removing an IR signal entering the RGB pixels of the RGB-IR image sensor (11) may be required. An image sensing device (10) according to an exemplary embodiment of the present disclosure can generate an RGB image without color distortion even in an environment where IR is strong by effectively removing an IR signal entering the RGB pixels. For example, the image sensing device (10) can generate a first RGB image including an IR component based on an RGB-IR pattern image generated by the RGB-IR image sensor (11), and can generate a second RGB image with the IR component removed based on the RGB-IR pattern image. In addition, the image sensing device (10) can generate a third RGB image, which is an RGB image without color distortion, based on the brightness component of the first RGB image and the color component of the second RGB image. More specific details on generating an RGB image without color distortion using the image sensing device (10) will be described later.

Since a general RGB-IR image sensor outputs RGB images and IR images separately, an image signal processor (13) used in a Bayer pattern image sensor cannot be used as is. In other words, in order to use a general RGB-IR image sensor, an image signal processor corresponding to the RGB-IR image sensor must be newly configured, which may cause problems in that the image processing system configuration becomes complicated and the implementation cost increases. An image sensing device (10) according to an exemplary embodiment of the present disclosure includes a conversion unit (12), thereby converting an RGB-IR pattern image generated by an RGB-IR image sensor (11) into a Bayer pattern image. Therefore, the image sensing device (10) does not require a separate image signal processor for the RGB-IR image sensor, and can use an image signal processor (13) that processes a Bayer pattern image that has been used in the past. In conclusion, the image sensing device (10) can generate a Bayer pattern image that includes an IR component but has no color distortion, and by generating the Bayer pattern image, a separate image signal processor for RGB-IR patterns can be used without requiring a separate image signal processor for RGB-IR patterns.

FIG. 2 is a flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.

FIG. 3 is a drawing for explaining an RGB-IR pattern according to an exemplary embodiment of the present disclosure. Hereinafter, FIG. 2 will be explained with reference to FIG. 3.

Referring to FIG. 2, an image processing method according to an exemplary embodiment of the present disclosure may, in step S210, obtain an RGB-IR pattern image through a pixel array of an RGB-IR pattern. For example, the RGB-IR pattern may be the same as the pattern (310) of FIG. 3. Specifically, the RGB-IR pattern may be the same as a pattern in which some of the G pixels in the Bayer pattern (300) are changed into IR pixels. In addition, the RGB-IR pattern may be the pattern (320) of FIG. 3. Specifically, the RGB-IR pattern may be a pattern in which some of the R pixels and B pixels in the Bayer pattern (300) are changed into IR pixels. In addition, the RGB-IR pattern may be the same as the pattern (330) of FIG. 3. Specifically, the RGB-IR pattern may be a pattern in which some of the B pixels in the Bayer pattern (300) are changed into IR pixels. However, the RGB-IR pattern is not limited to the above-described one and may have various patterns other than the patterns (310, 320, 330).

The image processing method can generate a first RGB image including an IR component based on the RGB-IR pattern image, in step S220. Specifically, the image processing method can generate the first RGB image by performing interpolation on the R channel, the G channel, and the B channel of the RGB-IR pattern image. In other words, the image processing method can generate images for each of the R channel, the G channel, and the B channel by performing interpolation. For example, the image processing method can generate the first RGB image by using a bilinear algorithm for each of the R channel, the G channel, and the B channel. In addition, the image processing method can perform interpolation by using a gradient of pixels surrounding a pixel on which interpolation is to be performed. However, the interpolation method is not limited to the above-described method, and the image processing method can generate the first RGB image by various interpolation methods. For example, the image processing method can generate the first RGB image by performing demosaicing. Specifically, the image processing method can generate an estimated image by estimating an R pixel value, a G pixel value, or a B pixel value at an IR pixel location of an RGB-IR pattern so as to correspond to a Bayer pattern, and perform demosaicing on the estimated image to generate a first RGB image. More specific details on generating the estimated image will be described later.

The image processing method can generate, in step S230, a second RGB image from which an IR component has been removed based on the RGB-IR pattern image and the first RGB image. Specifically, the image processing method can generate an IR image by performing interpolation for an IR channel based on the RGB-IR pattern image, and can generate a second RGB image based on the IR image and the first RGB image. Here, interpolation can be performed using a bilinear algorithm or a gradient of surrounding pixels of a pixel to be interpolated to generate the IR image, but is not limited thereto. More specific details of generating the second RGB image from which the IR component has been removed will be described later.

The image processing method can generate a third RGB image based on the brightness component of the first RGB image and the color component of the second RGB image in step S240. Since the RGB pixels of the RGB-IR image sensor can collect not only visible light but also IR light, the first RGB image can be an image including an IR component. Therefore, the color component of the first RGB image can be distorted. By subtracting the IR image, which is an image generated by performing interpolation on the IR channel, from the first RGB image, the distorted color component can be restored to a certain level. However, since an error has already occurred due to clipping in a portion where brightness is saturated in the first RGB image, and an error can also occur due to crosstalk occurring between the R pixel, the G pixel, the B pixel, and the IR pixel, it may not be possible to completely restore the distorted color component simply by subtracting the IR image from the first RGB image. Therefore, the image processing method can restore the color component by generating the third RGB image using the brightness component of the first RGB image and the color component of the second RGB image. For example, the image processing method can calculate the Y value, which is a brightness component of the first RGB image. In addition, the image processing method can calculate the U value and the V value, which are color components of the second RGB image. In addition, the image processing method can generate a third RGB image by calculating the R value, the G value, and the B value using the Y value, the U value, and the V value. More specific details on generating the third RGB image will be described later.

The image processing method can convert the third RGB image into a Bayer pattern image in step S250. Therefore, an image sensing device using the image processing method described above can use an image signal processor that processes a Bayer pattern image without a separate image signal processor that processes an RGB-IR pattern image.

FIG. 4 is a block diagram of an image sensing device according to an exemplary embodiment of the present disclosure.

FIG. 5 is a diagram for explaining an image processing method according to an exemplary embodiment of the present disclosure. Hereinafter, FIG. 4 will be explained with reference to FIG. 5.

Referring to FIG. 4, the image sensing device may include an RGB interpolation unit (410), an IR interpolation unit (420), a brightness component processing unit (430), a color component processing unit (440), a merging unit (450), and a pattern conversion unit (460).

The image sensing device can perform the image processing method according to the exemplary embodiment of the present disclosure described above.

For example, the RGB interpolation unit (410) can generate the first RGB image by performing interpolation for the R channel, the G channel, and the B channel based on the RGB-IR pattern image generated through the pixel array of the RGB-IR pattern. For example, the RGB interpolation unit (410) can perform a bilinear algorithm or perform interpolation based on the gradient of the surrounding pixels of the pixel to be interpolated. In addition, the RGB interpolation unit (410) can generate the estimated image by estimating the R pixel value, the G pixel value, or the B pixel value for the IR pixel position of the RGB-IR pattern so as to correspond to the Bayer pattern. For example, referring to (a) of FIG. 5, the G value for the position of the IR pixels (511 to 514) of the RGB-IR pattern (510) can be estimated so that the RGB-IR pattern (510) corresponds to the Bayer pattern (515). A bilinear algorithm may be used to estimate the G value, and a method using the gradient of the surrounding RGB pixels of the IR pixels (511 to 514) may be used, but is not limited thereto. In addition, referring to (b) of FIG. 5, the B value for the positions of the IR pixels (521 and 524) of the RGB-IR pattern (520) and the G value for the positions of the IR pixels (522 and 523) may be estimated so that the RGB-IR pattern (520) corresponds to the Bayer pattern (525). A bilinear algorithm may be used to estimate the B value and the G value, and a method using the gradient of the surrounding RGB pixels of the IR pixels (521 to 524) may be used, but is not limited thereto. In addition, referring to (c) of FIG. 5, the B value for the positions of the IR pixels (531 and 532) of the RGB-IR pattern (530) can be estimated so that the RGB-IR pattern (530) corresponds to the Bayer pattern (535). A bilinear algorithm can be used to estimate the B value, and a method using the gradient of the RGB pixels surrounding the IR pixels (531 and 532) can be used, but is not limited thereto. The RGB interpolation unit (410) can generate a first RGB image by performing demosaicing on an estimated image generated by estimating the R pixel value, the G pixel value, or the B pixel value of the IR pixel position.

The IR interpolation unit (420) can generate an IR image by performing interpolation on an IR channel based on an RGB-IR pattern image. For example, the IR interpolation unit (420) can estimate an IR value for each pixel by using a bilinear algorithm using IR pixel values of an RGB-IR pattern image. In addition, the IR interpolation unit (420) can estimate an IR value for each pixel by using a gradient of RGB pixels surrounding an IR pixel.

The brightness component processing unit (430) can calculate the brightness component value of the first RGB image. In other words, the brightness component processing unit (430) can calculate the brightness component value of the first RGB image, which is an image including an IR component.

The color component processing unit (440) can generate a second RGB image from which the IR component has been removed based on the first RGB image and the IR image. Since the RGB color of the first RGB image including the IR component is distorted by the IR component, if the first RGB image is directly used to extract the color component and the image is generated using the color component, the color of the generated image may be distorted. Therefore, the RGB image used for color component processing in the color component processing unit (440) may be a second RGB image that is an image from which the IR component has been removed from the first RGB image including the IR component. In other words, the color component processing unit (440) can calculate the color component values of the second RGB image from which the IR component has been removed. In addition, as described above, it may not be possible to completely restore the distorted color component by simply subtracting the IR image from the first RGB image due to clipping or crosstalk. Therefore, the image processing method can generate a third RGB image from which the color component is not distorted by extracting the brightness component of the first RGB image and extracting the color component of the second RGB image. Specifically, the merging unit (450) can generate a third RGB image by merging the brightness component value of the first RGB image and the color component value of the second RGB image. For example, the merging unit (450) can generate a third RGB image by calculating the R value, the G value, and the B value using the Y value, which is the brightness component value of the first RGB image, and the U value and the V value, which are the color component values of the second RGB image.

The color component processing unit (440) can generate a fourth RGB image by removing an IR component from the first RGB image using the IR image. In addition, the color component processing unit (440) can generate a second RGB image by increasing color saturation of the fourth RGB image. In addition, the color component processing unit (440) can also generate a second RGB image by performing denoising on the fourth RGB image. In addition, the color component processing unit (440) can generate a second RGB image by using a matrix for removing an IR component from the first RGB image, pixel values of the first RGB image, and pixel values of the IR image, considering crosstalk occurring between the R pixel, the G pixel, the B pixel, and the IR pixel included in the pixel array of the RGB-IR pattern. In this case, the matrix can be a matrix obtained by performing a regression analysis by setting the RGB image from which the IR component has been removed as a target RGB image. More specific details of generating the second RGB image will be described later.

The pattern conversion unit (460) can convert the third RGB image into a Bayer pattern image. The image sensing device can use an image signal processor for a Bayer pattern image by converting the third RGB image with restored color components into a Bayer pattern image.

FIG. 6 is a drawing for explaining an image processing method according to an exemplary embodiment of the present disclosure.

Referring to FIG. 6, an image processing method according to an exemplary embodiment of the present disclosure can obtain an RGB-IR pattern image (600) through a pixel array of an RGB-IR pattern.

The image processing method can generate a first RGB image (610) by performing interpolation on the R channel, G channel, and B channel of the RGB-IR pattern image (600). Since IR can be collected on the R pixel, G pixel, and B pixel of the pixel array of the RGB-IR pattern, the first RGB image (610) can include an IR component.

The image processing method can generate an IR image (620) by performing interpolation on an IR channel of an RGB-IR pattern image (600).

The image processing method can generate the second RGB image (630) by removing the IR component from the first RGB image (610) using the IR image (620). For example, the image processing method can generate the second RGB image (630) by subtracting the IR image (620) from the first RGB image (610). However, since visible light and IR have different wavelengths, their focal lengths are different, and this may cause crosstalk between pixels. Therefore, the second RGB image (630) with the IR removed may not be generated simply by subtracting the IR image (620) from the first RGB image (610). In this case, the following mathematical formula may be used to generate the second RGB image (630).

Here, R, G and B may be R values, G values and B values of the second RGB image, respectively. In addition, R', G' and B' may be R values, G values and B values of the first RGB image (610), respectively. In addition, IR' may be IR values of the IR image (620). In addition, the coefficient , and may be coefficients determined experimentally.

In addition, since a simple first-order formula may not take into account influences (e.g., crosstalk) between R pixels, G pixels, B pixels, and IR pixels, a second RGB image (630) may be generated through matrix operations. For example, a second RGB image (630) may be generated based on the following mathematical formula.

Here, a00 to a23 are transformation matrix coefficients, and the second RGB image (630), which is a target image to be generated by the first RGB image (610) and the IR image (620), can be obtained through regression analysis. The method for generating the second RGB image (630) is not limited to the above-described method, and the image processing method can generate the second RGB image (630) in various ways.

The image processing method can calculate the brightness component value of the first RGB image (610). For example, the image processing method can calculate the Y value of the first RGB image (610), which is an RGB image including an IR component. Specifically, the image processing method can calculate the Y value of the first RGB image (610) using the following formula.

However, the method of calculating the brightness component value in the image processing method is not limited to what has been described above. For example, the image processing method may obtain the brightness component value of the first RGB image (610) by converting the color domain of the first RGB image (610) into the HSV (Hue, Saturation, Value) domain based on the R value, G value, and B value of the first RGB image (610) and calculating the V value. In addition, the image processing method may obtain the brightness component value of the first RGB image (610) by converting the color domain of the first RGB image (610) into the HSL (Hue, Saturation, Lightness) domain based on the R value, G value, and B value of the first RGB image (610) and calculating the L value. In addition, the image processing method can obtain the brightness component value of the first RGB image (610) by converting the color domain of the first RGB image (610) into the HSI (Hue, Saturation, Intensity) domain and calculating the I value based on the R value, G value, and B value of the first RGB image (610).

In addition, the image processing method can calculate the color component values of the second RGB image (630). As described above, since the RGB color of the first RGB image mixed with the IR component is distorted by the IR component, if the first RGB image is directly used to extract the color component and the image is generated using the color component, the color of the generated image may be distorted. Therefore, the image processing method can generate the second RGB image by removing the IR component from the first RGB image including the IR component, and calculate the color component values of the second RGB image. For example, the image processing method can calculate the U value and the V value of the second RGB image (630). Specifically, the image processing method can calculate the U value and the V value of the second RGB image (630) using the following formula.

However, the method of calculating the color component value in the image processing method is not limited to what has been described above. For example, the image processing method may obtain the color component value of the second RGB image (630) by converting the color domain of the second RGB image (630) into the HSV domain based on the R value, the G value, and the B value of the second RGB image (630) and calculating the H value and the S value. In addition, the image processing method may obtain the color component value of the second RGB image (630) by converting the color domain of the second RGB image (630) into the HSL domain based on the R value, the G value, and the B value of the second RGB image (630) and calculating the H value and the S value. In addition, the image processing method may obtain the color component value of the second RGB image (630) by converting the color domain of the second RGB image (630) into the HSI domain based on the R value, the G value, and the B value of the second RGB image (630) and calculating the H value and the S value.

The image processing method can generate a third RGB image (640) by merging the brightness component of the first RGB image (610) and the color component of the second RGB image (630). For example, the image processing method can calculate the R value R'', the G value G'', and the B value B'' of the third RGB image (640) using the Y value of the first RGB image (610), the U value, and the V value of the second RGB image (630) by using the following formula.

However, the method of generating an RGB image based on the brightness component value and the color component value in the image processing method is not limited to what has been described above. For example, the image processing method can generate an R'' value, a G'' value, and a B'' value by using the inverse matrix of the matrix used to obtain the Y value, the U value, and the V value.

The image processing method can convert a third RGB image (640) into a Bayer pattern image (650).

FIG. 7 is a drawing for explaining an image processing method according to an exemplary embodiment of the present disclosure.

Referring to FIG. 7, an image processing method according to an exemplary embodiment of the present disclosure can generate a second RGB image (740) based on a first RGB image (710) and an IR image (720). The image processing method can generate the second RGB image (740) by removing an IR component from the first RGB image (710) using the IR image (720). In addition, the image processing method can generate a fourth RGB image (730) by removing an IR component from the first RGB image (710) using the IR image (720), and generate the second RGB image (740) based on the fourth RGB image (730). For example, in an environment with a lot of IR, noise may be generated due to IR or the color saturation of the image may be reduced. Therefore, the image processing method can generate the second RGB image (740) by increasing the color saturation of the fourth RGB image (730) from which the IR component is removed. For example, the image processing method may generate the second RGB image (740) by converting the color domain of the fourth RGB image (730) into the HSV domain, increasing the S value, which is a color saturation value, and then converting the color domain back into the RGB domain. However, the method of increasing the color saturation of the image processing method is not limited to what has been described above.

Additionally, the image processing method can generate a second RGB image (740) by performing denoising on the fourth RGB image (730).

FIG. 8 is a block diagram illustrating an imaging sensing device according to an exemplary embodiment of the present disclosure.

Referring to FIG. 8, the image sensing device (100) may be a CIS (Complementary Metal Oxide Semiconductor Image Sensor) that converts an optical signal into an electrical signal. The image sensing device (100) may be turned on/off, have an operation mode, and have sensitivity, etc., controlled by an image signal processor (ISP) (50).

The image sensing device (100) may include a pixel array (110), a row driver (120), a correlated double sampler (CDS) 130, an analog-to-digital converter (ADC) 140, an output buffer (150), a column driver (160), and a timing controller (180). Here, each configuration of the image sensing device (100) is merely exemplary, and at least some of the configurations may be added or omitted as needed.

The pixel array (110) may include a plurality of pixels arranged in a plurality of rows and a plurality of columns. In one embodiment, the plurality of pixels may be arranged in a two-dimensional pixel array including rows and columns. For example, the pixel array (110) may have an RGB-IR pattern. In another embodiment, the plurality of unit image pixels may be arranged in a three-dimensional pixel array. The plurality of pixels may convert an optical signal into an electrical signal on a pixel-by-pixel basis or on a pixel-by-pixel-group basis, and pixels within a pixel group may share at least a specific internal circuit. The pixel array (110) may receive a driving signal including a row selection signal, a pixel reset signal, and a transmission signal from the row driver (120), and a corresponding pixel of the pixel array (110) may be activated to perform an operation corresponding to the row selection signal, the pixel reset signal, and the transmission signal by the driving signal.

The row driver (120) can activate the pixel array (110) to perform specific operations on pixels included in a corresponding row based on commands and control signals supplied by the timing controller (180). In one embodiment, the row driver (120) can select at least one pixel arranged in at least one row of the pixel array (110). The row driver (120) can generate a row selection signal to select at least one row among a plurality of rows. The row driver (120) can sequentially enable a pixel reset signal and a transmission signal for pixels corresponding to the at least one selected row. Accordingly, an analog reference signal and an image signal generated from each of the pixels of the selected row can be sequentially transmitted to the correlated double sampler (130). Here, the reference signal may be an electrical signal provided to the correlated double sampler (130) when the sensing node (e.g., the floating diffusion node) of the pixel is reset, and the image signal may be an electrical signal provided to the correlated double sampler (130) when the photocharge generated by the pixel is accumulated in the sensing node. The reference signal representing the reset noise unique to the pixel and the image signal representing the intensity of incident light may be collectively referred to as pixel signals.

CMOS image sensors can utilize correlated double sampling to remove unwanted offset values of pixels, such as fixed pattern noise, by sampling a pixel signal twice to remove the difference between the two samples. For example, correlated double sampling removes unwanted offset values by comparing pixel output voltages acquired before and after photocharges generated by incident light are accumulated at a sensing node, so that only pixel output voltages based on incident light can be measured. In one embodiment, the correlated double sampler (130) can sequentially sample and hold a reference signal and an image signal provided to each of a plurality of column lines from the pixel array (110). That is, the correlated double sampler (130) can sample and hold levels of a reference signal and an image signal corresponding to each of the columns of the pixel array (110).

The correlated double sampler (130) can transmit the reference signal and image signal of each column to the ADC (140) as a correlated double sampling signal based on a control signal from the timing controller (180).

The ADC (140) can convert the correlated double sampling signal for each column output from the correlated double sampler (130) into a digital signal and output it. In one embodiment, the ADC (140) can be implemented as a ramp-compare type ADC. The ramp-compare type ADC can include a comparison circuit that compares a ramp signal that rises or falls over time with an analog pixel signal, and a counter that performs a counting operation until the ramp signal matches the analog pixel signal. In one embodiment, the ADC (140) can convert the correlated double sampling signal generated by the correlated double sampler (130) for each of the columns into a digital signal and output it.

The ADC (140) may include a plurality of column counters corresponding to each of the columns of the pixel array (110). Each column of the pixel array (110) is connected to each column counter, and image data may be generated by converting a correlated double sampling signal corresponding to each of the columns into a digital signal using the column counters. According to another embodiment, the ADC (140) may include one global counter, and may convert a correlated double sampling signal corresponding to each of the columns into a digital signal using a global code provided by the global counter.

The output buffer (150) can temporarily hold and output image data of each column provided from the ADC (140). The output buffer (150) can temporarily store image data output from the ADC (140) based on a control signal of the timing controller (180). The output buffer (150) can operate as an interface that compensates for the difference in transmission (or processing) speed between the image sensing device (100) and another device connected thereto.

The column driver (160) can select a column of the output buffer (150) based on a control signal of the timing controller (180), and control the output buffer (150) so that image data temporarily stored in the selected column of the output buffer (150) is sequentially output. In one embodiment, the column driver (160) can receive an address signal from the timing controller (180), and the column driver (160) can generate a column selection signal based on the address signal to select a column of the output buffer (150), thereby controlling image data to be output from the selected column of the output buffer (150) to the outside (e.g., the image signal processor (50)).

The conversion unit (170) can convert image data regarding the RGB-IR pattern into image data regarding the Bayer pattern. Accordingly, the image sensing device (100) can generate image data regarding the Bayer pattern.

The timing controller (180) can control at least one of the row driver (120), the correlated dual sampler (130), the ADC (140), the output buffer (150), and the column driver (160).

The timing controller (180) may provide clock signals required for the operation of each component of the image sensing device (100), control signals for timing control, and address signals for selecting a row or column to at least one of the row driver (120), the correlated dual sampler (130), the ADC (140), the output buffer (150), and the column driver (160). According to one embodiment, the timing controller (180) may include a logic control circuit, a phase lock loop (PLL) circuit, a timing control circuit, and a communication interface circuit.

The image signal processor (50) processes image data input from the image sensing device (100), and can control each component of the photographing device (10) according to the processing result or according to an external input signal. The image signal processor (50) can reduce noise on the image data and perform image signal processing for improving image quality, such as gamma correction, color filter array interpolation, color matrix, color correction, and color enhancement. In addition, the image data generated by performing image signal processing for improving image quality can be compressed to generate an image file, or the image data can be restored from the image file. The compression format of the image can be a reversible format or an irreversible format. As an example of the compression format, in the case of a still image, the JPEG (Joint Photographic Experts Group) format or the JPEG 2000 format can be used. In addition, in the case of a moving image, a moving image file can be generated by compressing a plurality of frames according to the MPEG (Moving Picture Experts Group) standard. Image files can be created, for example, according to the Exif (Exchangeable image file format) standard.

Image data output from the image signal processor (50) may be stored in the internal memory or external memory of the image sensing system or displayed through a display according to a user's request or automatically.

Additionally, the image signal processor (50) can perform blur processing, edge emphasis processing, image interpretation processing, image recognition processing, image effect processing, etc.

In addition, the image signal processor (50) can perform display image signal processing for display. For example, the image signal processor (50) can perform brightness level adjustment, color correction, contrast adjustment, outline emphasis adjustment, screen split processing, character image generation, and image synthesis processing.

The above description is merely an illustrative description of the technical idea of the present disclosure, and those skilled in the art to which the present disclosure pertains may make various modifications and variations without departing from the essential characteristics of the present disclosure. Therefore, the embodiments disclosed in the present disclosure are not intended to limit the technical idea of the present disclosure but to explain it, and the scope of the technical idea of the present disclosure is not limited by these embodiments. The protection scope of the present disclosure should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present disclosure.

Claims (16)

A step of acquiring an RGB-IR pattern image through a pixel array of RGB-IR patterns;
A step of generating a first RGB image including an IR component based on the above RGB-IR pattern image;
A step of generating a second RGB image from which the IR component has been removed based on the RGB-IR pattern image and the first RGB image;
A step of generating a third RGB image based on the brightness component of the first RGB image and the color component of the second RGB image; and
An image processing method comprising the step of converting the third RGB image into a Bayer pattern image. In the first paragraph,
The step of generating the first RGB image is:
An image processing method, characterized by comprising a step of generating the first RGB image by performing interpolation on the R channel, G channel, and B channel of the RGB-IR pattern image. In the first paragraph,
The step of generating the first RGB image is:
A step of generating an estimated image by estimating an R pixel value, a G pixel value, or a B pixel value for an IR pixel location of the RGB-IR pattern so as to correspond to a Bayer pattern; and
An image processing method, characterized by further comprising a step of generating the first RGB image by performing demosaicing on the estimated image. In the first paragraph,
The step of generating the above second RGB image is:
A step of generating an IR image by performing interpolation on the IR channel of the above RGB-IR pattern image; and
An image processing method, characterized by comprising a step of generating the second RGB image based on the IR image and the first RGB image. In the fourth paragraph,
The step of generating the above second RGB image is:
A step of generating a fourth RGB image by removing the IR component from the first RGB image using the IR image; and
An image processing method, characterized by comprising a step of generating the second RGB image by increasing color saturation of the fourth RGB image. In the fourth paragraph,
The step of generating the above second RGB image is:
A step of generating a fourth RGB image by removing the IR component from the first RGB image using the IR image; and
An image processing method, characterized by comprising a step of generating the second RGB image by performing denoising on the fourth RGB image. In the fourth paragraph,
The step of generating the above second RGB image is:
An image processing method, characterized by comprising: a matrix for removing the IR component from the first RGB image by considering crosstalk occurring between R pixels, G pixels, B pixels and IR pixels included in the pixel array of the RGB-IR pattern; and a step of generating the second RGB image by using pixel values of the first RGB image and pixel values of the IR image. In Article 7,
The above matrix is,
An image processing method characterized in that the matrix is obtained by performing regression analysis by setting the RGB image from which the IR component has been removed as a target RGB image. In the first paragraph,
The step of generating the third RGB image is:
A step of calculating the Y value, which is the brightness component of the first RGB image;
A step of calculating the U value and the V value, which are the color components of the second RGB image; and
An image processing method characterized by comprising a step of generating the third RGB image by calculating the R value, the G value, and the B value using the Y value, the U value, and the V value. An RGB interpolation unit configured to generate a first RGB image by performing interpolation for an R channel, a G channel, and a B channel based on an RGB-IR pattern image generated through a pixel array of an RGB-IR pattern;
An IR interpolation unit configured to generate an IR image by performing interpolation for an IR channel based on the above RGB-IR pattern image;
A brightness component processing unit configured to calculate a brightness component value of the first RGB image;
A color component processing unit configured to generate a second RGB image from which an IR component has been removed based on the first RGB image and the IR image, and to calculate color component values of the second RGB image;
A merging unit configured to generate a third RGB image by merging the brightness component value of the first RGB image and the color component value of the second RGB image; and
An image sensing device comprising a pattern conversion unit configured to convert the third RGB image into a Bayer pattern image. In Article 10,
The above RGB interpolation part,
An image sensing device characterized in that it is configured to generate an estimated image by estimating an R pixel value, a G pixel value, or a B pixel value for an IR pixel location of the RGB-IR pattern so as to correspond to a Bayer pattern, and to generate the first RGB image by performing demosaicing on the estimated image. In Article 10,
The above color component processing unit,
An image sensing device characterized in that it is configured to generate a fourth RGB image by removing an IR component from the first RGB image using the IR image, and to generate the second RGB image by increasing color saturation of the fourth RGB image. In Article 10,
The above color component processing unit,
An image sensing device characterized in that it is configured to generate a fourth RGB image by removing an IR component from the first RGB image using the IR image, and to generate the second RGB image by performing denoising on the fourth RGB image. In Article 10,
The above color component processing unit,
An image sensing device characterized in that the device comprises a matrix for removing the IR component from the first RGB image by considering crosstalk occurring between the R pixels, G pixels, B pixels and IR pixels included in the pixel array of the RGB-IR pattern, and configured to generate the second RGB image using pixel values of the first RGB image and pixel values of the IR image. In Article 14,
The above matrix is,
An image sensing device characterized in that the matrix is obtained by performing regression analysis by setting the RGB image from which the IR component has been removed as a target RGB image. In Article 10,
The above merger part,
An image sensing device characterized in that it is configured to generate the third RGB image by calculating the R value, the G value, and the B value using the Y value, which is the brightness component value of the first RGB image, and the U value and the V value, which are the color component values of the second RGB image.

Images