JP2023107583A - Image encoding device, image decoding device, control method and program thereof - Google Patents

Abstract

An object of the present invention is to efficiently encode RAW image data having an infrared component in addition to three primary colors. Kind Code: A1 An image encoding device for encoding an image obtained by an imaging sensor having three primary color filters of R, G, and B in the visible light wavelength region and an infrared filter, the intensity distribution of transmission wavelengths of the infrared filter. a generator for generating difference data indicating the difference between the image data of any of the three primary color filters determined according to and the image data obtained through the infrared filter; the image data of each of the three primary color filters; and an encoding unit that encodes data. [Selection diagram] Fig. 1

Description

The present invention relates to image coding technology.

2. Description of the Related Art In recent years, imaging apparatuses represented by digital cameras, digital camcorders, and the like employ CCD sensors or CMOS sensors as imaging elements. These sensors have a color filter array (hereinafter referred to as CFA) formed on the sensor surface and detect one color component per pixel. A typical CFA is a Bayer array in which 2×2 pixels (one red (R), two green (G0, G1), and one blue (B)) are periodically arranged as shown in FIG. be. Bayer array image data (hereinafter referred to as RAW data) is naturally obtained from the Bayer array sensor.

Since human visual characteristics are highly sensitive to luminance components, in general Bayer arrays, green components, which are abundant in luminance components, are compared to red and blue components, as shown in Fig. 3. A configuration that allocates twice the number of pixels is used. Since RAW data has only one color component information per pixel, it is necessary to generate red, blue, and green information per pixel using a process called demosaicing. In general, image data of RGB signals obtained by demosaicing or YUV signals obtained by converting from RGB signals are encoded and recorded. However, due to demosaicing, one pixel has three color components, requiring a data amount three times larger than that of RAW data. Therefore, a method of directly encoding and recording RAW data before demosaicing has been proposed. there is

For example, Patent Literature 1 discloses a method of encoding RAW data after separating it into four planes R, G0, B, and G1. Further, in Patent Document 2, RAW data is separated into four planes of R, G0, B, and G1 in the same manner as in Patent Document 1, and then approximately divided into luminance (Y) and color differences (Co, Cg, Dg). A method for transforming and encoding is shown.

On the other hand, a sensor capable of simultaneously capturing a color image and an infrared image has been developed for purposes such as monitoring and inspection. Unlike FIG. 3, these sensors are arrayed with IR (infrared) pixels in addition to R, G, and B pixels as shown in FIG.

JP-A-2003-125209 JP 2006-121669 A

The plane conversion methods disclosed in Patent Documents 1 and 2 are premised on the Bayer array shown in FIG. 3, and are not compatible with pixels having IR in addition to RGB as shown in FIG.

An object of the present invention is to provide a technique for efficiently encoding RAW of a pixel array having components other than visible light in addition to RGB.

In order to solve this problem, for example, the image encoding device of the present invention has the following configuration. i.e.
An image encoding device for encoding an image obtained by an imaging sensor having three primary color filters of R, G, and B in the visible light wavelength region and an infrared filter,
generating means for generating difference data indicating a difference between the image data of any one of the three primary color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter and the image data obtained through the infrared filter; ,
It has image data for each of the three primary color filters and encoding means for encoding the difference data.

According to the present invention, RAW image data having infrared components in addition to three primary colors can be efficiently encoded.

1 is a block diagram showing a configuration example of an image encoding device according to a first embodiment; FIG. 1 is a block diagram showing a configuration example of an image decoding device according to a first embodiment; FIG. The figure which shows a Bayer array. The figure which shows an RGB-IR arrangement|sequence. FIG. 4 is a diagram for explaining an example of infrared rays having three different wavelength intensity distributions and the relationship between the wavelengths of ultraviolet rays and visible rays according to the first embodiment; A diagram for explaining a wavelet transform. FIG. 10 is a diagram for explaining a case in which the effective wavelength region of IR used is between IR1 and IR2 in the second embodiment; A diagram showing an RGB-NIR-FIR arrangement. FIG. 10 is a diagram for explaining a case in which the effective wavelength region of IR used is between IR1 and IR2 and near IR1 in the second embodiment; FIG. 11 is a diagram for explaining an IRH plane in the third embodiment; FIG. FIG. 2 is a hardware configuration diagram of an information processing apparatus according to a modification of the first embodiment; 4 is a flowchart showing a processing procedure of image encoding; 4 is a flowchart showing a processing procedure of image decoding;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

[First Embodiment]
FIG. 1 is a block configuration diagram of an image encoding device according to the first embodiment.

As shown in FIG. 1, the image encoding device has an imaging unit 100, a RAW encoding unit 110, and a control unit 150 that controls them. RAW encoding section 110 includes plane transform section 101 , frequency transform section 102 , quantization section 103 and entropy encoding section 104 . The control unit 150 includes a CPU, a ROM storing programs executed by the CPU, and a RAM used as a work area by the CPU.

Also, in this embodiment, JPEG2000 is used as an example of the encoding method, but the type of encoding method does not matter.

In the above configuration, encoding processing of an input image will be described with reference to FIG.

The imaging unit 100 is a general imaging optical unit including an optical lens, a diaphragm mechanism, a shutter mechanism, an imager, and the like. The imager is of a type that performs color separation for imaging, and is, for example, a CCD type or MOS type having a color filter for color separation on its surface. This imager separates the colors of the formed optical image and converts them into electrical signals corresponding to the amount of light. The color separation color filter of the imager in the embodiment will be explained as a filter of RGB-IR (Infrared Ray) arrangement shown in FIG. R, G, and B in the drawing indicate three primary color filters in the visible light wavelength region, and IR indicates an infrared filter.

The plane conversion unit 101 separates the RAW image data of the RGB-IR pixel array into the R plane, G plane, B plane, and IR plane as shown in FIG. The R plane is a plane composed of a single R component, the B plane is a plane composed of a single B component, the G plane is a plane composed of a single G component, and the IR plane is a single IR. A plane composed of components. Let W be the number of pixels in the horizontal direction of the RAW image, and H be the number of pixels in the vertical direction. It has a size of 2 pixels. Then, the plane conversion unit 101 outputs the R, G, and B planes obtained by the decomposition. As for the IR plane, the plane conversion unit 101 follows an instruction from the control unit 150 to generate and output an IRH plane indicating difference data between the color plane of another color component and the IR plane.

FIG. 5 shows an example of infrared light having three different wavelength intensity distributions and the relationship with the wavelengths of ultraviolet light and visible light. In FIG. 5, the B component filter has strength (high transmittance) in the B wavelength region, the G component filter has strength in the G wavelength region, and the R wavelength region. indicates that it is the filter of the R component that has the intensity at .

Also, the IR wavelength region includes three wavelength regions IR1, IR2, IR3. A filter having an intensity in the wavelength region IR1 is defined as an IR1 component filter. IR1 has a wavelength intensity distribution in a region adjacent to the R wavelength region. The wavelength region IR2 is an infrared region near the wavelength region of R next to the wavelength region IR1, and a filter having an intensity in the wavelength region IR2 is a filter of the IR2 component. The wavelength region IR3 has a wavelength intensity distribution in the wavelength region furthest from the R component in the IR wavelength region. A filter having an intensity in the wavelength region IR3 is defined as an IR3 component filter. In the embodiment, the transmission center wavelength of the wavelength region IR1 is around 780 nm, the transmission center wavelength of the wavelength region IR2 is around 870 nm, and the transmission center wavelength of the wavelength region IR3 is around 940 nm. Also, in an embodiment, the full widths at half maximum of the wavelength intensity distributions are each about 50 nm. The wavelength region IR1 has a high correlation with R, the wavelength region IR2 has a high correlation with B, and the wavelength region IR3 has a high correlation with G.

Assume that the control unit 150 acquires information indicating that the IR filter of the imaging unit 100 has an intensity in the wavelength region of the wavelength intensity distribution IR1, for example, via an operation unit (not shown). In this case, the control unit 150 sets the plane conversion unit 101 to calculate the difference between the IR plane and the R plane. In response to this, the plane conversion unit 101 calculates a difference plane, ie, an IRH plane, according to the following equation (1).
IRH=R-IR×ir_gain1 (1)

IR1 has a high correlation with R as explained above. Therefore, the plane conversion unit 101 performs gain adjustment using a gain irgain1 for matching the IR level to R. Then, the plane conversion unit 101 calculates an IRH plane indicating the difference between the R plane and the adjusted IR plane "IR×ir_gain1". Although the values (differential pixel values) forming the IRH plane have positive and negative signs, many small values close to zero appear, and a high compression rate can be expected.

Also, assume that information indicating that the IR filter of the imaging unit 100 has an intensity in the wavelength region of the wavelength intensity distribution IR2 is set to the control unit 150 . In this case, the control unit 150 sets the plane dividing unit 101 to calculate the difference between the IR plane and the B plane. In response to this, the plane conversion unit 101 calculates the IRH plane according to the following equation (2).
IRH=B-IR×ir_gain2 (2)
IR2 has a high correlation with B as explained above. Therefore, the plane conversion unit 101 performs gain adjustment using a gain irgain2 for adjusting the IR level to B. FIG. Then, the plane conversion unit 101 calculates an IRH plane representing the difference between the B plane and the adjusted IR plane "IR×ir_gain2". Although the values (differential pixel values) forming the IRH plane have positive and negative signs, many small values close to zero appear, and a high compression rate can be expected.

Further, when information indicating that the IR filter of the imaging unit 100 has an intensity in the wavelength region of the wavelength intensity distribution IR3 is set to the control unit 150, the control unit 150 causes the plane conversion unit 101 to: The IR plane is set so as to obtain the difference from the G plane. In response to this, the plane conversion unit 101 calculates the IRH plane according to the following equation (3).
IRH=G-IR×ir_gain3 (3)
As explained earlier, IR3 has a high correlation with G. Therefore, the plane conversion unit 101 performs gain adjustment using a gain irgain3 for adjusting the IR level to G. Then, the plane conversion unit 101 calculates an IRH plane representing the difference between the G plane and the adjusted IR plane "IR×ir_gain3". Although the values (differential pixel values) forming the IRH plane have positive and negative signs, many small values close to zero appear, and a high compression rate can be expected.

As described above, the plane conversion unit 101 determines the color component for which the difference is to be obtained, according to the effective wavelength region of the IR filter of the imaging unit 100, and generates the IRH plane representing the difference. The plane converter 101 then supplies the R, G, B planes and the IRH plane to the frequency converter 102 .

The frequency transform unit 102 obtains a plurality of subbands by performing frequency transform (wavelet transform in the embodiment) on each plane supplied from the plane transform unit 101 . Frequency transform section 102 then sends transform coefficients in each subband to quantization section 104 .

Here, wavelet transform will be described with reference to FIG. FIG. 6 shows an example of wavelet transform, which is also adopted in JPEG2000, and performs subband decomposition at only one level (once). As shown in FIG. 6, an input plane 400 is split into low frequency component and high frequency component data using a vertical low pass filter 401 and a vertical high pass filter 402 for the vertical component. Data extracted through vertical low-pass filter 401 and vertical high-pass filter 402 are down-sampled by 2:1 through down-sampling circuits 403 and 404, respectively. As a result, the downsampling circuits 403 and 404 output data with half the vertical resolution. The data output from the down-sampling circuits 403 and 404 are divided into low frequency component data and high frequency component data using horizontal low pass filter 405, horizontal high pass filter 406, horizontal low pass filter 407 and horizontal high pass filter 408, respectively. Further, they are down-sampled by 2:1 through down-sampling circuits 409, 410, 411, and 412, respectively. As a result, an output image 413 can be obtained. The output image 413 is composed of blocks LL, HL, LH, and HH respectively obtained through the filtering process described above. For the sake of simplification of notation, these blocks will be referred to as LL, HL, LH, and HH as necessary below. Also, L indicates a low frequency region, and H indicates a high frequency region. For example, HH indicates a high-frequency subband in both horizontal and vertical directions. The output image 413 is an image obtained by reducing the resolution of the input image 400 by half vertically and horizontally in the LL area. Regions HH, HL, and LH are high frequency regions. The illustrated LL, HL, LH, and HH regions are commonly referred to as sub-bands. Note that wavelet transform may be performed recursively on subband LL.

The quantization unit 104 quantizes the transform coefficients sent from the frequency transform unit 102 using a preset quantization parameter for each subband, and quantizes the transform coefficients included in the subbands of each plane data. Entropy encoding section 105 is supplied with the transform coefficients after encoding.

Entropy coding section 105 performs entropy coding such as EBCOT (Embedded Block Coding with Optimized Truncation) on the wavelet coefficients and quantization parameters quantized by quantization section 104 for each subband. Then, entropy coding section 105 generates a header containing information necessary for decoding. Entropy coding section 105 then outputs the coded data following this header. The information stored in the header includes the size of the RAW image and the number of bits per component. information (or information indicating that the IRH plane is the difference between the IR plane and which color component plane) is included. Note that the output target of the entropy encoding unit 105 is not particularly limited, but is, for example, a storage medium such as an HDD or a memory card.

Next, a method of decoding the encoded image data generated by the above procedure will be described.

FIG. 2 is a block configuration diagram of the image decoding device in the embodiment. As illustrated, the image decoding device includes an entropy decoding unit 200, an inverse quantization unit 201, a frequency inverse transform unit 202, a plane inverse transform unit 203, a header analysis unit 204, and a control unit 250 that controls the entire apparatus. have.

The header analysis unit 204 analyzes the header (file header) of the encoded stream to be decoded. Then, the header analysis unit 204 extracts information indicating which of IR1 to IR3 filters the IR filter of the imaging device (imaging unit 100) used when capturing the image to be decoded is, The information is supplied to the plane inverse transformation unit 203 .

The entropy decoding unit 200 decodes the encoded data following the header, that is, the encoded data of the wavelet coefficients and quantization parameters encoded by EBCOT (Embedded Block Coding with Optimized Truncation) or the like, and the inverse quantization unit 101 transfer to

The inverse quantization unit 201 inversely quantizes the data (quantized frequency transform coefficients) supplied from the entropy decoding unit 200 using a quantization parameter to generate frequency transform coefficients. The inverse quantization unit 201 then supplies the frequency transform coefficients to the inverse frequency transform unit 202 .

The inverse frequency transform unit 202 applies inverse frequency transform (inverse wavelet transform in the embodiment) to the frequency transform coefficients supplied from the inverse quantization unit 201 to generate R, G, B and IRH planes. The frequency inverse transforming unit 202 then supplies the R, G, B and IRH planes to the plane inverse transforming unit 203 .

The plane inverse transform unit 203 performs inverse plane transform on the R, G, B, and IRH planes supplied from the frequency inverse transform unit 202, reconstructs and outputs RAW image data in the RGB-IR array. Details of the inverse plane transform are described below.

The plane inverse transform unit 203 receives the information supplied from the header analysis unit 204, that is, the IR filter of the imaging device (imaging unit 100) used when capturing the image to be decoded. An IR plane is generated from the IRH plane based on the information indicating whether there is any.
When the IR1 filter is used The plane inverse transform unit 203 restores the IR plane from the R plane and the IRH plane according to the following equation (4): IR = (R-IRH)/irgain1 (4)
When the IR2 filter is used The plane inverse conversion unit 203 restores the IR plane from the R plane and the IRH plane according to the following equation (5): IR = (B-IRH)/irgain2 (5)
When the IR3 filter is used The plane inverse transform unit 203 restores the IR plane from the G plane and the IRH plane according to the following equation (6): IR = (G-IRH)/irgain3 (6)

The plane inverse transform unit 203 restores the IR plane either as described above. After that, the plane inverse conversion unit 203 generates RAW image data by arranging the pixels in the R, G, B, and IR planes as shown in FIG.

As described above, according to the present embodiment, high-efficiency encoding is achieved by taking the difference between the IR of the RAW image data in the RGB-IR array and the highly correlated color component before encoding. be possible. In the embodiment, gain adjustment is performed on IR to obtain the difference, but the difference may be obtained after gain adjustment is performed on RGB.

In the above description, it is assumed that the image decoding device has gains irgain1, irgain2, and orgain3 that are used by the image encoding device. However, when the image encoding device generates encoded data, the gain used may be stored in the header.

[Modification of First Embodiment]
An example in which the image encoding process of the first embodiment is realized by an application program of an information processing apparatus represented by a personal computer will be described as a modification of the first embodiment.

FIG. 11 is a hardware configuration diagram of an information processing apparatus in a modified example. The information processing apparatus has a CPU 1101, a ROM 1102, a RAM 1103, an HDD 1104, an operation unit (keyboard, mouse, etc.) 1105, a display unit 1106, and an I/F 1107.

In the above configuration, when the apparatus is powered on, the CPU 1101 executes the boot program in the ROM 1102, loads the OS (operating system) from the HDD 1104 into the RAM 1103, and transfers the processing to the OS. Various processing can be performed. Then, according to the user's instruction from the operation unit 1105, the image encoding application or the image decoding application is loaded from the HDD 1104 to the RAM 1103, and the application is executed under the OS. , functions as an image decoding device.

Here, the processing procedure of the CPU 1101 when the information processing device functions as the image encoding device will be described with reference to the flowchart of FIG. Note that the RGB-IR array RAW image data file to be encoded is received from an external imaging device via the I/F 1107 and is already stored in the HDD 1104 . It is also assumed that the file is selected as an encoding target file.

In S1201, the CPU 1101 reads the RAW image data file to be encoded from the HDD 1104 and inputs the RAW image data. In step S<b>1202 , the CPU 1101 generates R, G, B, and IR planes from the input RAW image data and stores them in the RAM 103 .

In S1203, the CPU 1101 determines which of the IR1 to IR3 filters is the IR filter of the imaging unit of the imaging apparatus used when capturing the RAW image data. If information specifying an IR filter is stored in the file header of the RAW image data, determination is made from that information. Alternatively, the user may make the determination according to the selection operation of any one of the IR1 to IR3 filters.

If the CPU 1101 determines that the IR1 filter is used, the process proceeds to S1204; if it determines that the IR2 filter is used, the process proceeds to S1205; and if it determines that the IR3 filter is used, the process proceeds to S1205. branch to

When the process branches to S1204, the CPU 1101 generates an IRH plane according to the formula (1) shown above. When the process branches to S1205, the CPU 1101 generates an IRH plane according to equation (2) shown above. Then, when the process branches to S1206, the CPU 1101 generates an IRH plane according to the formula (3) shown above.

In S1207, the CPU 1101 executes frequency transform (wavelet transform in this embodiment) on each of the R, G, B, and IRH planes to generate transform coefficients. Then, in S1208, the CPU 1101 performs quantization processing on the transform coefficients of each plane. In S1209, the CPU 1101 entropy-encodes the quantized transform coefficients to generate encoded data.

Finally, in S1210, the CPU 1101 creates in the HDD 1104 a file header containing information necessary for decoding, such as information indicating the type of IR filter, and writes encoded data of each plane so as to follow the file header. (It is desirable that the encoded data of the IRH be last) is stored in the HDD 1104 to create an encoded image data file.
be.

Next, the processing procedure of the CPU 1101 when the information processing device functions as the image decoding device will be described with reference to the flowchart of FIG. It is assumed that the encoded image data file to be decoded in the HDD 1104 has already been selected by the user. Also, an example in which the RAW image data obtained by decoding is stored in the HDD 1104 as a file will be described. When displaying an infrared image on the display unit 1106, the IR plane obtained by decoding may be displayed as a monochrome image. When an RGB color image is to be displayed, an image of three color components of R, G, and B may be generated for each pixel from the R, G, and B planes obtained by decoding, and then displayed.

In S1301, the CPU 1101 analyzes the header of the encoded image data file to be decoded, and determines which of the IR1 to IR3 filters was used to capture the image. Then, in S1302, the CPU 1101 reads encoded data from the encoded image data file to be decoded, performs entropy decoding processing, and obtains quantized transform coefficients for each of the R, G, B, and IRH planes. In S1303, the CPU 1101 performs inverse quantization processing to obtain frequency transform coefficients for each of the R, G, B, and IRH planes. Then, in S1304, the CPU 1101 performs inverse frequency transform (inverse wavelet transform in the embodiment) on the frequency transform coefficients of the R, G, B, and IRH planes to restore the R, G, B, and IRH planes. do.

In S1305, the CPU 1101 determines which one of IR1 to IR3 is the IR filter of the image to be decoded based on the analysis result of S1301. If the CPU 1101 determines that the IR1 filter is used, the process proceeds to S1306; if it determines that the IR2 filter is used, the process proceeds to S1307; and if it determines that the IR3 filter is used, the process proceeds to S1308. branch to

When the process branches to S1306, the CPU 1101 generates an IR plane according to equation (4) shown above. When the process branches to S1307, the CPU 1101 generates an IR plane according to Equation (5) shown above. Then, when the process branches to S1308, the CPU 1101 generates an IR plane according to the formula (3) shown above.

In S1309, the CPU 1101 arranges the pixels of the IR plane generated in any one of S1306 to S1308 and the R, G, and B planes restored in S1304 as shown in FIG. RAW image data is reconstructed.

Then, in S1310, the CPU 1101 stores the reconstructed RAW image data in the HDD as a file. At this time, information indicating the type of IR filter is also stored in the file header.

As described above, the processing corresponding to the first embodiment can be realized by the CPU executing a computer program. It should be added that the second and third embodiments described below can be realized by the CPU executing a computer program.

[Second embodiment]
Next, an image coding apparatus according to a second embodiment will be described with reference to FIGS. 1, 4 and 7. FIG. In the second embodiment, the configuration of the image coding apparatus is the same as in the first embodiment, but the conversion processing by the plane conversion unit 101 is different from that in the first embodiment. Other operations are assumed to be the same as those of the first embodiment, and description thereof will be omitted.

In the second embodiment, the effective IR wavelength region of the RGB-IR arrangement is the middle of IR1 and IR2 (the center wavelength of the intensity distribution of the transmission wavelength is approximately the middle of the range of 780 nm to 870 nm), as shown in FIG. 820 nm) will be described.

The plane conversion unit 101 generates the R, G, and IR planes from the RAW image data of the RGB-IR pixel array as shown in FIG. 4, and then calculates the IRH plane according to the following equation (7).
IRH=(R+B)/2−IR×irgain4 (7)
Here, irgain4 is a gain for adjusting the IR level to "(R+B)/2".

Since the processing related to subsequent encoding is the same as in the first embodiment, the description thereof will be omitted. However, in the header of the generated encoded image data file, information indicating that the IR filter has a specificity as shown in FIG. 4 is stored.

Also, the configuration related to the decoding process may be the same as that of FIG. 2 of the first embodiment. That is, when decoding, if the information about IR stored in the header represents FIG. will be formed.
IR={(R+B)/2-IRH}/irgain4 (8)
If the IR effective wavelength region of the RAW image data of the RGB-IR array belongs to the middle between IR1 and IR2, plane conversion is performed in consideration of the ratio of R that has a strong correlation to IR1 and B that has a strong correlation to IR2. In the present embodiment, the case of being located between IR1 and IR2 has been described as an example, but when the effective wavelength region is closer to IR1 as shown in FIG. Multiply by a stronger ratio to perform a plane transform.
IRH=(R×0.7+B×0.3)−IR×irgain5 (9)
In this case, when decoding, the IR plane can be regenerated according to the following equation (10) using the previously decoded R, B, and IRH planes.
IR={(R×0.7+B×0.3)−IRH}/irgain5 (10)
Also, the weighting coefficients shown in equations (9) and (10) differ depending on how much the effective wavelength region is closer to IR1 or IR2. Therefore, when it is closer to IR2, plane conversion is performed by multiplying B by a stronger weighting factor.

In addition, in the second embodiment, the effective wavelength region of IR is between IR1 and IR2. The same idea can be applied even in the case of

[Third Embodiment]
An image coding apparatus according to the third embodiment will be described with reference to FIGS. 1, 8, and 10. FIG. In the third embodiment, the configuration of the image coding apparatus is the same as in the first embodiment, but the conversion processing performed by the plane conversion unit 101 is different from that in the first embodiment. In addition, unlike the first embodiment and the second embodiment, the imaging sensor uses two types of filters: a near infrared filter (NIR: Near Infrared Ray) and a far infrared filter (FIR: Far Infrared Ray). There is a difference. The use of the NIR filter increases the visibility in dark places, and the use of the FIR filter also serves as a human sensor that detects intrusion, so it is expected that the monitoring performance will be further improved.

Since other operations are the same as those of the first embodiment, description thereof is omitted. Also, in the third embodiment, an example in which IR1 is used as the effective wavelength region of NIR is shown.

FIG. 8 shows the pixel array of the sensor in the third embodiment. Similar to FIG. 4, in addition to R, G, and B, the pixels are arranged with IR (infrared) pixels, but the difference is that the near infrared (NIR) and far infrared (FIR) pixels are alternately arranged. ing.

When the plane conversion unit 101 receives information indicating that this sensor is used from the control unit 150, the plane conversion unit 101 separates the RAW image data of the RGB-IR arrangement into the R, G, B planes and the FIR+NIR plane. Then, IRH representing the difference plane is calculated according to the following equations (11) and (12).
・If the IR pixel is at the NIR position, IRHr=R−NIR×irgain1 (11)
・If the IR pixel is located at the FIR position, IRHg=G-FIR×irgain6 (12)

The plane conversion unit 101 determines the color component for which the difference is to be taken depending on whether it is NIR or FIR to be used as described above, forms an IRH plane in a mixed form of IRHr and IRg calculated as shown in FIG. , B, and IRH to the frequency conversion unit 102 . Since subsequent processing is the same as that of the first embodiment, description thereof is omitted.

During decoding, the R, G, and IRH planes previously decoded are used to regenerate IR components according to the following equations (13) and (14).
・When the decoded pixel position is NIR,
NIR={R-IRHr}/irgain1 (13)
・When the decoded pixel position is FIR,
FIR={G-IRHg}/irgain6 (14)

As described above, in the case of a pixel array in which NIR and FIR are mixed, the color component for taking the difference is changed depending on whether the IR pixel is NIR or FIR. Also, instead of forming a separate plane, an IRH plane in which different calculation results are mixed is formed, and processing after plane conversion is performed. By forming an IRH plane in which different calculation results are mixed in this way, the number of pixels to be processed in R, G, B, and IRH becomes the same. Therefore, in a case where the processing after plane conversion is parallelized and executed, the processing can be made uniform.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

DESCRIPTION OF SYMBOLS 100... Imaging part 101... Plane conversion part 102... Frequency conversion part 103... Quantization part 104... Entropy encoding part 110... RAW encoding part 150... Control part 204... Header analysis part 200... Entropy decoding unit 201 Inverse quantization unit 202 Inverse frequency transform unit 203 Inverse plane transform unit 250 Control unit

Claims (10)

An image encoding device for encoding an image obtained by an imaging sensor having three primary color filters of R, G, and B in the visible light wavelength region and an infrared filter,
generating means for generating difference data indicating a difference between the image data of any one of the three primary color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter and the image data obtained through the infrared filter; ,
An image encoding apparatus comprising: image data for each of the three primary color filters; and encoding means for encoding the difference data. adjusting means for adjusting the image data obtained by the infrared filter with a gain corresponding to the type of filter used to generate the difference data among the three primary color filters;
2. The image encoding apparatus according to claim 1, wherein said generating means generates said difference data using image data obtained by said infrared filter after being adjusted by said adjusting means. The generating means is
when the central transmission wavelength of the infrared filter is near 780 nm, generating the difference data using the R component image data of the three primary colors;
when the central transmission wavelength of the infrared filter is near 870 nm, generating the difference data using the image data of the B component of the three primary colors;
3. The image encoding apparatus according to claim 1, wherein when the central transmission wavelength of the infrared filter is around 940 nm, the difference data is generated using the image data of the G component of the three primary colors. The generating means is
When the central transmission wavelength of the infrared filter is in the range of 780 nm to 870 nm, the difference data is generated using the image data of the weighted average of the R component image data and the B component image data of the three primary colors. ,
When the central transmission wavelength of the infrared filter is in the range of 870 nm to 940 nm, the difference data is generated using weighted average image data of the B component and image data of the three primary colors and the image data of the G component. 3. The image coding apparatus according to claim 1, wherein: The infrared filters in the imaging sensor include two types of near-infrared filters and far-infrared filters,
The generating means calculates the difference between the pixels obtained by passing through the near-infrared filter and the pixels obtained by passing through the R component filter, and the pixels obtained by passing through the far-infrared filter: 3. The image coding apparatus according to claim 1, wherein the differential data is generated by calculating a difference from pixels obtained through the G component filter. An image decoding device for decoding coded data of RAW image data obtained by an imaging sensor having three primary color filters of R, G, and B in the visible light wavelength region and an infrared filter,
Acquisition means for analyzing the header of the encoded data and acquiring information representing the intensity distribution of the transmission wavelength of the infrared filter;
decoding means for decoding the encoded data to generate image data for each of the three primary color filters and differential data for the infrared filter;
a restoring means for restoring the image data obtained from the infrared filter from the image data of any one of the three primary color filters specified by the information obtained by the obtaining means and the difference data;
and output means for outputting the image data of each of the three primary color filters obtained by the decoding means and the image data obtained by the infrared filter obtained by the restoration means as RAW image data. Device. A control method for an image encoding device that encodes an image obtained by an imaging sensor having three primary color filters of R, G, and B in the visible light wavelength region and an infrared filter,
a generating step of generating difference data indicating a difference between the image data of any one of the three primary color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter and the image data obtained through the infrared filter; ,
A control method for an image encoding device, comprising: image data of each of the three primary color filters; and an encoding step of encoding the difference data. An image decoding device for decoding coded data of RAW image data obtained by an imaging sensor having three primary color filters of R, G, and B in the visible light wavelength region and an infrared filter,
an acquiring step of analyzing the header of the encoded data and acquiring information representing the intensity distribution of the transmission wavelength of the infrared filter;
a decoding step of decoding the encoded data to generate image data of each of the three primary color filters and differential data of the infrared filter;
a restoring step of restoring the image data obtained from the infrared filter from the image data of any one of the three primary color filters specified by the information obtained in the obtaining step and the difference data;
and an output step of outputting the image data of each of the three primary color filters obtained in the decoding step and the image data obtained by the infrared filter obtained in the restoration step as RAW image data. How to control the device. A computer program which, when read and executed by a computer, causes the computer to execute each step of the method for controlling an image encoding apparatus according to claim 7. A computer program which, when read and executed by a computer, causes the computer to execute each step of the control method for an image decoding apparatus according to claim 8.

Images