JP2022114740A - Image processing device, image processing method and image processing program - Google Patents

Abstract

To provide an image processing device, an image processing method, and an image processing program capable of suppressing image quality deterioration caused by the presence of deteriorated pixels in a pixel block.SOLUTION: An image processing device includes a difference value acquisition unit that acquires a difference value between a pixel value of a target pixel and a pixel value of each of a plurality of peripheral pixels for a pixel block that includes the target pixel and the plurality of peripheral pixels, a pixel value correction unit that corrects pixel values of the peripheral pixels on the basis of the acquired difference value, and a filter processing unit that determines a median value from among pixel values of pixels in the pixel block corrected by the pixel value correction unit.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to an image processing device, an image processing method, and an image processing program.

2. Description of the Related Art An image processing apparatus is known that reduces noise by performing median filtering on an image (see, for example, Japanese Laid-Open Patent Publication No. 2002-200013).

JP-A-2000-175081

Degraded pixels having image quality degradation may exist in the vicinity of a pixel of interest in an image (within a pixel block). The pixel value of the degraded pixel may become a value that is far from the pixel value of the target pixel and has a low relevance to the target pixel due to the influence of noise, for example. In the median filtering process performed by the conventional image processing apparatus exemplified in Patent Document 1, when such a deteriorated pixel exists in a pixel block, the pixel value of the target pixel becomes a pixel value with low relevance to it. Instead, it may lead to deterioration of image quality.

In view of the circumstances as described above, an object of one aspect of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program that perform filter processing to improve image quality caused by the presence of deteriorated pixels in a pixel block. is to suppress the deterioration of

An image processing apparatus according to an aspect of the present invention obtains a difference value between a pixel value of a pixel of interest and a pixel value of each of the plurality of peripheral pixels for a pixel block including a pixel of interest and a plurality of peripheral pixels. a difference value acquisition unit; a pixel value correction unit that corrects the pixel values of the peripheral pixels based on the acquired difference value; a filtering unit that determines a value.

An image processing method according to an aspect of the present invention is a method executed by a computer, and for a pixel block including a target pixel and a plurality of peripheral pixels, a pixel value of the target pixel and each pixel of the plurality of peripheral pixels a difference value acquisition step of acquiring a difference value from the value of the pixel block; a pixel value correction step of correcting the pixel values of the peripheral pixels based on the acquired difference value; and a filtering step of determining a median value among the pixel values of the pixels.

An image processing program according to an aspect of the present invention is a program for causing a computer to execute the above image processing method.

According to the above aspect, in the image processing device, the image processing method, and the image processing program that perform filter processing, deterioration of image quality caused by the presence of deteriorated pixels in a pixel block can be suppressed.

1 is an external view showing an example of a digital camera according to one embodiment of the present invention; FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to one embodiment of the present invention; FIG. FIG. 2 is a conceptual diagram showing an example of a current pixel block, which is a pixel block within an image of a current frame; FIG. 4 is a conceptual diagram showing an example of a reference pixel block that was a pixel block at the same position as the current pixel block in an image n frames before the current frame; FIG. 4 is a diagram for explaining correction processing by a pixel value correction unit included in the image processing apparatus; 4 is a flow chart showing an image processing method and an image processing program executed by a computer of the image processing apparatus according to one embodiment of the present invention; FIG. 4 is a diagram showing an example of an output image subjected to median filter processing by the image processing apparatus according to the embodiment of the present invention; FIG. 10 is a diagram showing an example of an output image subjected to median filter processing by a conventional image processing apparatus; FIG. 10 is a diagram showing an example of an output image that has not been subjected to median filtering;

An image processing apparatus 1 according to this embodiment will be described in detail with reference to the drawings. The image processing method and image processing program of this embodiment are realized by causing the computer (circuit component) of the image processing apparatus 1 to execute various processing steps. Also, the image processing apparatus 1 may be called, for example, a Color Process PipeLine. Further, the image processing device 1 may be implemented as a large scale integrated circuit (LSI: Large Scale Integration) or a part thereof.

The image processing apparatus 1 may be installed in fixed imaging devices such as security cameras, surveillance cameras, and in-vehicle cameras, or may be installed in portable imaging devices such as digital cameras and various terminals. Further, the image processing device 1 may be one specialized for performing image processing on images transmitted by wire or wirelessly from a photographing device or other device or images stored in various storage media. That is, the image processing apparatus 1 has a degree of freedom, and various design changes are possible.

FIG. 1 is an external view showing an example of a digital camera (photographing device) 1X. In the digital camera 1X, light from a subject is guided into the camera body 11X through the lens 10X, reflected by the mirror 12X, and forms a real image of the subject on the focusing glass 13X. The object image as this real image is guided to the finder 15X by the pentaprism 14X, and can be observed. A display device 16X is arranged between the focus glass 13X and the pentaprism 14X, and displays various information superimposed on the subject image reflected on the focus glass 13X. A dashed line in FIG. 1 indicates an optical path of light from a subject guided to the finder 15X. In the digital camera 1X, the mirror 12X is raised to open the shutter 17X to guide subject light to the imaging element 18X for photographing.

The digital camera 1X may incorporate the image processing device 1 according to one aspect of the present invention that performs image processing on an image captured by the image sensor 18X, or may process an image captured by the image sensor 18X according to one aspect of the present invention. A communication unit (connection terminal, antenna, etc.) that transmits data to the external image processing apparatus 1 by wire or wirelessly may be provided. That is, the present invention is applicable not only to the digital camera 1X having a photographing function, but also to any device capable of acquiring an image photographed by an external device and applying noise reduction processing using a median filter. In other words, the present invention can also be applied to a system composed of multiple devices.

FIG. 2 is a block diagram showing the configuration of the image processing apparatus 1. As shown in FIG. The image processing apparatus 1 includes a demosaic processing unit 10, a YUV conversion unit 20, a difference value acquisition unit 30, a difference value determination unit 40, a luminance data acquisition unit 50, a brightness determination unit 60, a pixel value correction unit 70, and a subject change determination unit 80. , a filtering unit 90 , an adding unit 100 and a post-processing unit 110 .

An image captured by the imaging device 18X (for example, a RAW image or a Bayer image, hereinafter referred to as an “input image D1”) is input to the demosaic processing unit 10 . The demosaic processing unit 10 performs demosaic processing on the input image D1 to convert it into an RGB image D2. Although illustration is omitted here, a block that performs isolated point removal processing and chromatic aberration correction on the input image D1 may be provided in the preceding stage of the demosaic processing unit 10 .

The YUV conversion unit 20 obtains a YUV image D3 by YUV conversion of the RGB image D2 input from the demosaic processing unit 10 . The YUV image D3 is output to the difference value acquiring section 30 and the pixel value correcting section 70 . Also, the luminance data D11 of the YUV image D3 is output to the luminance data acquisition unit 50, the subject change determination unit 80, and the addition unit 100. FIG.

FIG. 3A is a conceptual diagram showing an example of a current pixel block BC, which is a pixel block in the current frame image ( _YUV image D3). FIG. 3B is a conceptual diagram showing an example of a reference _pixel block _BR which was a pixel block at the same position as the current pixel block BC in an image n (n is a natural number) frames before the current frame. In this embodiment, the reference pixel block _BR is, for example, a pixel block of an image one frame before the current frame. In this embodiment, the filtered YUV image D10 is applied as the image one frame before the current frame. The filtered YUV image D10 is image data after median filtering by the filtering unit 90, and will be described later in detail.

As shown in FIG. 3A, the current pixel block _B 1 _C consists of a target pixel IP _C in the YUV image D3 and eight neighboring pixels PP1 _C ˜ A 3×3 pixel block containing PP8 _C. The difference value acquiring unit 30 sequentially sets each pixel in the YUV image D3 as the pixel of interest IP _C , and for the current pixel block B _C including the set pixel of interest IP _C , obtains the pixel value of the pixel of interest IP _C and a plurality of pixels. obtain (calculate) the difference values DV1 _C to DV8 _C from the respective pixel values of the peripheral pixels PP1 _C to PP8 _C of . Any surrounding pixels included in the current pixel block B _C will be described as “peripheral pixels PP _C ”.

As shown in FIG. 3B, the reference pixel block B _R is adjacent to the target pixel IP _R in the YUV image D10 after filtering one frame before the current frame and in each of the up, down, left, and right diagonal directions of the target pixel IP _R. It is a 3×3 pixel block containing eight peripheral pixels PP1 _R to PP8 _R located. The difference value acquisition unit 30 calculates the pixel value of each pixel of the reference pixel block B _R that was in the same position as the current pixel block B _C in the filtered YUV image D10 and the target pixel IP _C of the current pixel block B _C. Get the difference value of Specifically, the difference value obtaining unit 30 obtains the difference value _DV0R between the target pixel IP _C of the current pixel block B _C and the target pixel IP _R of the reference pixel block B _R , and also obtains the pixel of the target pixel IP _C Difference values DV1 _R to DV8 _R between the value and the pixel values of the peripheral pixels PP1 _R to PP8 _R in the reference pixel block B _R are obtained. Any peripheral pixels included in the reference pixel block B _R are referred to as "peripheral pixels PP _R ".

The difference value acquisition section 30 outputs the difference values DV1 _C to DV8 _C and DV0 _R to DV8 _R to the difference value determination section 40 . For convenience, the data including the difference values DV1 _C to DV8 _C and DV0 _R to DV8 _R are referred to as "difference value data D4".

The difference value determination unit 40 determines whether each of the difference values DV1 _C to DV8 _C and DV0 _R to DV8 _R included in the difference value data D4 input from the difference value acquisition unit 30 exceeds a predetermined threshold value (ε value). determine whether The difference value determination section 40 outputs this determination result (determination data D5) to the pixel value correction section .

In addition to the determination data D5, the pixel value correction unit 70 receives the YUV image D3 from the YUV conversion unit 20 and the filtered YUV image D10 from the filter processing unit 90 .

The pixel value correction unit 70 determines whether the difference value between the target pixel IP _C and the pixel value of the target pixel IP C out of the target pixels (the peripheral pixels PP1 _C to PP8 _C , the target pixel IP _R and the peripheral pixels PP1 _R to PP8 _R ) exceeds the ε value. A pixel is specified by referring to the determination data D5. The pixel value correction unit 70 refers to the determination data D5 among the surrounding pixels PP1 _C to PP8 _C included in the YUV image D3 and the target pixel IP _R and the surrounding pixels PP1 _R to PP8 _R included in the filtered YUV image D10. The pixel value is corrected so that the pixel value difference from the target pixel _IPC corresponds to the ε value (here, the difference value is the same as the ε value). As a result, the pixel values of all the target _pixels in the current pixel block BC and the reference pixel block _BR have a pixel value difference between the target pixel _IPC and the ε value or less. A specific numerical value of the ε value is determined, for example, by the designer of the image processing apparatus 1 based on the performance of the imaging apparatus, his own experience, and the like.

Correction processing by the pixel value correction unit 70 will be described with reference to FIG. A value P ₁ (Y ₁ , U ₁ , V ₁ ) arranged in the YUV space shown in FIG. 4 indicates the pixel of interest IP _C . Also, the value P ₂ (Y ₂ , U ₂ , V ₂ ) indicates one peripheral pixel PP _C among the plurality of peripheral pixels PP1 _C to PP8 _C. As shown in _FIG . ₄ , the distance between the value P1 and the value P2 in the YUV space (in other words, the difference value between the pixel value of the target pixel IP _C and the pixel value of the peripheral pixel PP _C ) is an ε value bigger than Therefore, the pixel value correction unit 70 corrects the value P ₂ to the value P ₃ (Y ₃ , U ₃ , V ₃ ) so that the distance from the value P ₁ in the YUV space is the same as the ε value. . Note that the value _P3 is a value located _{on a} line segment connecting the values P1 and P2 in the YUV space.

For convenience, the current pixel block B _C after correction by the pixel value correction unit 70 (that is, the pixel of interest IP _C and the peripheral pixels PP1 _C to PP8 _C after correction) will be referred to as "corrected pixel block data D8 _C ". Further, the reference pixel block B _R corrected by the pixel value correction unit 70 (that is, the target pixel IP _R after correction and the peripheral pixels PP1 _R to PP8 _R after correction) will be referred to as "corrected pixel block data D8 _R ".

Here, in the median filter, which is useful as an edge preserving filter, if a degraded pixel with image quality degradation exists in a pixel block, the pixel value of the target pixel is replaced with the pixel value of the degraded pixel with low relevance. On the contrary, it sometimes invites deterioration of image quality. As a more detailed example, when degraded pixels exist near edges in an image and noise removal processing is performed using a median filter, the edge is shifted toward the degraded pixels at the same time as the noise is removed (edge shift). sometimes occurred.

Therefore, in the present embodiment, the pixel values of the target pixels in the current pixel block B _C and the reference pixel block B _R are corrected so that the difference in pixel value from the target pixel IP _C is within the ε value. . In other words, a degraded pixel whose pixel value is far from the target pixel IP _C due to the influence of noise despite being close to the target pixel IP _C has a value related to the target pixel IP _C (the value of the target pixel IP _C ). _A value that is close to the pixel value and is constrained to a value such that the difference value of the pixel value from the pixel of interest IPC is the ε value). As a result, in the median filtering process, a pixel value that is far from the target pixel IP _C is not determined as the pixel value of the target pixel IP _C. Therefore, the deterioration of the image quality due to the presence of deteriorated pixels in the pixel block can be suppressed while taking advantage of noise removal and edge preservation by median filtering.

In general, the darker the image, the lower the signal-to-noise ratio. Therefore, the darker the image, the larger the difference value between the pixel values of the pixel of interest and the peripheral pixels. Therefore, the ε value is set according to the brightness of the pixel of interest _IPC .

Specifically, the luminance data acquisition unit 50 acquires the luminance data D6 of the target block from the luminance data D11 input from the YUV conversion unit 20, and outputs the luminance data D6 to the brightness determination unit 60. The target block is a pixel block composed of a plurality of pixel groups including the target pixel IP _C , and is the same as the current pixel block B _C , for example. The luminance data D6 is, for example, the average luminance value of the pixels in the block of interest or the median luminance value of the pixels in the block of interest. Note that the target block is not limited to the same block as the current _pixel block BC. The block of interest may be, for example, a 5×5 or 7×7 pixel block centered on the pixel of interest IP _C (that is, a pixel block different from the current pixel block _BC ).

The brightness determination unit 60 determines brightness of the pixel of interest IP _C based on the brightness data D6 acquired by the brightness data acquisition unit 50 . For example, when the luminance data D6 exceeds a predetermined luminance value, the brightness determination unit 60 determines that the pixel of interest IP _C is bright, and when the luminance data D6 is equal to or less than the predetermined luminance value, determines that the pixel of interest IP _C is dark. do. The brightness determining section 60 outputs this determination result (determination data D7) to the difference value determining section 40. FIG.

The difference value determination unit 40 sets the ε value according to the determination data D7. Specifically, in the case of determination data D7 indicating that the pixel of interest IP _C is bright, the difference value determination unit 40 sets the ε value to the first ε value (first threshold value). In the case of the determination data D7 indicating that the target pixel IP _C is dark, the difference value determining unit 40 sets the ε value to a second ε value (third threshold) that is larger than the first ε value. As a result, the difference value determination unit 40 performs threshold determination using the ε value corresponding to the brightness of the pixel of interest IP _C , that is, threshold determination in consideration of the SN ratio.

In order to reduce the processing load of the image processing apparatus 1, the ε value may be a constant value regardless of the brightness of the pixel of interest _IPC . In this case, the brightness data acquisition section 50 and the brightness determination section 60 can be omitted from the image processing apparatus 1 . Therefore, the configuration of the image processing apparatus 1 can be simplified.

The subject change determination unit 80 receives the luminance data D11 of the image of the current frame (YUV image D3) from the YUV conversion unit 20, and receives the luminance data of the image one frame before the current frame (YUV image D10 after filtering). D9 is input from the filtering section 90 .

The subject change determination unit 80 determines the degree of subject change in the pixel of interest between frames based on the luminance data D9 and D11. Specifically, the subject change determination unit 80 obtains the difference value DV _Y between the pixels of interest in the luminance data D9 and the luminance data D11, and calculates a predetermined evaluation value EV based on the obtained difference value DV _Y. A predetermined threshold determination is performed on the obtained evaluation value EV.

The luminance data _D9 is, for example, the luminance value of the target pixel _IPC in the current pixel block _BC , the average luminance value of the pixels in the current pixel block BC, or the median _luminance value of the pixels in the current pixel block BC. is. The luminance data D11 is, for example, the luminance value of the _{target pixel IP R in} the reference pixel block _BR , the average luminance value of the pixels in the reference pixel block BR, or the median luminance value of the pixels in the reference pixel block _BR . is.

Note that the difference value _DVY may be a difference value between pixel values of the target pixels of the current frame and one frame before instead of the difference value of the luminance data of the pixels of interest of the current frame and one frame before. In this case, the image of the current frame (YUV image D3) and the image of the previous frame (YUV image after filtering D10) are input to the subject change determination unit 80. FIG. The difference value _DVY in this case is, for example, the difference value between the pixel value of the pixel of interest _IPC in the current pixel block _BC and the pixel value of the pixel of interest _IPR in the reference pixel block _BR . The difference value _DVY may be the difference value between the average pixel value in the current pixel block _BC and the average pixel value in the reference _pixel block _BR . It may be a difference value between the value and the median value of the pixel values in the reference pixel block _BR .

The evaluation value EV may be the difference value DV _Y itself, or may be a value obtained by multiplying the difference value DV _Y by a predetermined linear function or non-linear function.

The subject change determination section 80 determines whether or not the evaluation value EV is equal to or less than the threshold TH (second threshold), and outputs this determination result (determination data D12) to the filter processing section 90 .

The filter processing unit 90 receives the corrected pixel block data D8- _C and the corrected pixel block data D8- _R from the pixel value correction unit 70, and also receives the determination data D12 from the subject change determination unit 80. FIG.

The filtering section 90 comprises a two-dimensional median filter 92, a three-dimensional median filter 94 and an output section 96. FIG. The filtering unit 90 performs median filtering on only the pixels of the current pixel block BC according to the determination data _D12 input from the subject change determining unit 80 (in other words, the threshold value determination result for the evaluation value EV). (That is, the median filtering process by the two-dimensional median filter 92, which determines the median value among the pixel values of the nine pixels (target pixel IP _C and corrected peripheral pixels PP1 _C to PP8 _C )), or the current pixel block Median filter processing for both pixels of B _C and reference pixel block B _R (that is, 18 pixels (target pixel IP _C and corrected peripheral pixels PP1 _C to PP8 _C and corrected target pixel IP _R and corrected peripheral pixels PP1 _R to PP8 _R ), which is a median filtering process by a three-dimensional median filter 94 in consideration of the time axis direction).

When the evaluation value EV exceeds the threshold TH, the degree of change in the subject at the pixel of interest between frames is high (in other words, the subject is moving). Therefore, when median filtering is performed by the three-dimensional median filter 94, subject blurring and artifacts occur due to the movement of the subject between frames. Therefore, when the evaluation value EV exceeds the threshold TH, the filtering unit 90 performs median filtering using the two-dimensional median filter 92, which hardly causes subject blurring and artifacts even when the subject moves.

If the evaluation value EV is equal to or less than the threshold TH, the degree of change in the subject at the pixel of interest between frames is low (in other words, the subject does not move). Therefore, even if the median filter processing by the three-dimensional median filter 94 is performed, subject blurring and artifacts are unlikely to occur. Therefore, when the evaluation value EV is equal to or less than the threshold TH, the filter processing unit 90 performs median filtering with a three-dimensional median filter 94 that has a higher noise reduction effect than median filtering with a two-dimensional median filter 92 .

It should be noted that the median filtering process does not use multiplication or division, so the computational load is light. That is, in this embodiment, a high noise reduction effect can be obtained with a light calculation load.

In addition, the post-filter YUV image D10 subjected to median filtering is fed back as the image of the previous frame, and median filtering is performed by the three-dimensional median filter 94. This achieves a high noise reduction effect and reduces flickering. It is also possible to obtain the effect of reducing flickering in hue and brightness caused by factors such as

The output unit 96 outputs the YUV image D10 after filtering to the adding unit 100 and also outputs it to the difference value acquiring unit 30 and the pixel value correcting unit 70 as an image one frame before the current frame. Further, the output unit 96 outputs the luminance data D9 of the filtered YUV image D10 to the subject change determination unit 80 as the luminance data of the image one frame before the current frame.

The addition unit 100 adds the filtered YUV image D10 input from the output unit 96 and the luminance data D11 input from the YUV conversion unit 20 . By adding the luminance data D11 of the original image (that is, the YUV image D3 which is the image before median filtering) to the filtered YUV image D10, a post-texture restored YUV image D13 in which the texture of the original image is restored is obtained. .

The post-processing unit 110 performs post-processing such as gamma correction, edge enhancement processing, and super-resolution processing on the YUV image D13 after texture restoration to produce an output image D14. The post-processing performed by the post-processing unit 110 has a degree of freedom, and various design changes are possible. The output image D14 is displayed, for example, on a display screen of a display device provided at a later stage.

FIG. 5 is a flow chart showing an image processing method and an image processing program executed by a computer (circuit component) of the image processing apparatus 1. FIG.

As shown in FIG. 5, the input image D1 is demosaiced by the demosaicing unit 10 and converted into an RGB image D2 (step S101).

YUV conversion is performed by the YUV conversion unit 20 on the RGB image D2 to obtain a YUV image D3 (step S102).

The difference value acquisition unit 30 obtains difference values DV1 _C to DV8 between the pixel value of the target pixel IP _C and the pixel values of the plurality of peripheral pixels PP1 _C to PP8 _C for the current pixel block B _C in the YUV image D3. _C is obtained, and for each pixel of the reference pixel block B _R in the filtered YUV image D10, difference values DV0 _R to DV8 _R of pixel values from the target pixel IP _C are obtained (step S103).

The difference value determination unit 40 performs threshold determination for the difference value data D4 (difference values DV1 _C to DV8 _C and DV0 _R to DV8 _R ) (step S104).

Of the target pixels (surrounding pixels PP1 _C to PP8 _C , target pixel IP _R and surrounding pixels PP1 _R to PP8 _R ), for pixels whose pixel value difference from the target pixel IP _C exceeds the ε value, the difference value is The pixel value is corrected by the pixel value correction unit 70 so as to be the same value as the ε value (step S105).

The subject change determination unit 80 determines the degree of subject change at the target pixel between frames (step S106).

When the subject change determination unit 80 determines that the degree of subject change in the pixel of interest between frames is high (step S106: HIGH), the filter processing unit 90 selects only the pixels of the current pixel block B to _C as the target. , median filtering by the two-dimensional median filter 92 (step S107).

When the object change determination unit 80 determines that the degree of change in the object in the pixel of interest between frames is low (step S106: LOW), the filter processing unit 90 separates the current pixel block _{BC from the reference pixel block BR .} Both pixels are subjected to median filtering by the three-dimensional median filter 94 (step S108).

The post-filtering YUV image D10 and the brightness data D11 are added by the adder 100 to obtain a post-texture restored YUV image D13 in which the texture of the original image is restored (step S109).

Post-processing by the post-processing unit 110 is performed on the YUV image D13 after texture restoration to obtain an output image D14 (step S110).

FIG. 6A is a diagram showing an example of an output image D14 subjected to median filtering by the image processing apparatus 1. FIG. FIG. 6B is a diagram showing an example of an output image subjected to median filter processing by a conventional image processing apparatus. FIG. 6C is a diagram showing an example of an output image that has not been subjected to median filtering.

In the example of FIG. 6B, as can be seen in comparison with FIG. 6C, the effect of noise removal and edge preservation by median filtering is obtained, while the portion where deteriorated pixels exist (for example, the central portion of the radial striped pattern appearing in the output image) ) causes an edge shift. Specifically, the closer to the center of the striped pattern, the more white and black pixels are mixed in the pixel block, and the median value determined by the median filtering process varies greatly in each pixel block. This causes an edge shift in the central portion of the stripe pattern. On the other hand, in the example of FIG. 6A, the median value determined by the median filtering process is constrained to a value close to the target pixel IP _C (a value at which the maximum difference value of the pixel value from the target pixel IP _C is the ε value). be done. Therefore, even when white and black pixels coexist within a pixel block, the median value determined by median filtering in each pixel block does not vary greatly. Therefore, in the example of FIG. 6A, as can be seen from a comparison with FIGS. 6B and 6C, the effects of noise removal and edge preservation by median filtering are obtained, and image quality such as edge shift is improved even in portions where deteriorated pixels exist. Suppresses deterioration.

That is, according to the image processing apparatus, the image processing method, and the image processing program of the present embodiment, the pixel value of the target pixel, which is far from the target pixel IP _C , is constrained to the pixel value related to the target pixel IP _C. By doing so, it is possible to suppress deterioration in image quality due to the presence of deteriorated pixels in the pixel block while taking advantage of noise removal and edge preservation by median filtering.

In addition, the present invention is not limited to the above-described embodiments, and can be modified in various ways without departing from the gist of the present invention. Also, the functions executed in the above-described embodiments may be combined as appropriate as possible. Various steps are included in the above-described embodiments, and various inventions can be extracted by appropriately combining the disclosed multiple constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiments, if an effect can be obtained, a configuration in which these constituent elements are deleted can be extracted as an invention.

In the above embodiment, the difference value acquiring unit 30 calculates the pixel value of the target pixel IP _C and the pixel values of the target pixels (surrounding pixels PP1 _C to PP8 _C , target pixel IP _R and peripheral pixels PP1 _R to PP8 _R ). Although the difference value is acquired, the configuration of the present invention is not limited to this. The difference value acquiring unit 30 further acquires the average pixel value of the target pixel IP _C and the target pixel IP _R , and acquires the difference value DV _CR between the acquired average pixel value and the pixel value of the target pixel IP _C. may be

In this case, the difference value determination unit 40 performs threshold determination on the difference value DV _CR in addition to the difference values DV1 _C to DV8 _C and DV0 _R to DV8 _R. When the difference value DV _CR exceeds the ε value, the pixel value correction unit 70 sets the average pixel value of the target pixel IP _C and the target pixel IP _R so that the difference value from the target pixel IP _C is the same as the ε value. Corrected so that When the degree of subject change in the pixel of interest between frames is low, the filter processing unit 90 applies the above average pixel value to the pixel values of the 18 _pixels in the current pixel block BC and the reference pixel block _BR . Median filter processing is performed by a three-dimensional median filter 94 to determine the median value from a total of 19 pixel values.

By performing median filtering with the three-dimensional median filter 94 using the average pixel value of the target pixel IP _C and target pixel IP _R , the effect of noise removal and edge preservation by median filtering can be further obtained.

In the above embodiment, the difference value acquisition unit 30 calculates the pixel value of the target pixel IP _C and the pixel values of the target pixels (surrounding pixels PP1 _C to PP8 _C , target pixel IP _R and peripheral pixels PP1 _R to PP8 _R ). Although the difference value is acquired, the configuration of the present invention is not limited to this. The difference value obtaining unit 30 obtains the average pixel value of a pair of pixels (a pair of peripheral pixels PP _C ) arranged at a paired position with the target pixel IP _C interposed between the current pixel block B _C and the target pixel IP _C In addition, for the reference pixel block _{BR, the average pixel value of the pixel pair (pair of peripheral pixels PP R )} at the same position as the above pixel pair and the pixel value of the target pixel IP _C are calculated. may be configured to acquire the difference value of .

In this case, the difference value determination unit 40 determines whether or not the difference value between the average pixel value of each pixel pair and the pixel value of the target pixel IP _C exceeds the ε value. For pixel pairs in which the difference value between the average pixel value and the pixel value of the target pixel IP _C exceeds the ε value, the pixel value correction unit 70 sets the difference value to a value corresponding to the ε value (here, the difference value is ε Correct the average pixel value so that it becomes the same value as the value). The filtering unit 90 has a two-dimensional median filter 92 or a three _- dimensional median filter 94 that determines the median value from the average pixel value of each pixel pair corrected by the pixel value correcting unit 70 and the pixel value of the pixel of interest IPC. Perform median filtering by .

In the gradation part (sky, skin, etc.) in the image, tone jump may occur due to the characteristics of the median filter. As described above, by performing median filter processing using the average pixel value of pixel pairs, occurrence of tone jumps can be suppressed and smooth gradation expression can be achieved.

In the above embodiment, the filter processing unit 90 performs median filtering using the two-dimensional median filter 92 or median filtering using the three-dimensional median filter 94 according to the degree of change in the subject at the target pixel between frames. However, the configuration of the present invention is not limited to this. The filter processing unit 90 is configured to perform median filtering using not only one of the two-dimensional median filter 92 and the three-dimensional median filter 94, but also both, depending on the degree of change in the subject at the pixel of interest between frames. good too.

In this case, the subject change determination unit 80 determines whether the evaluation value EV is (1) below the threshold TH, (2) above the threshold TH and below the threshold TH', or (3) above the threshold TH'. determine whether When the evaluation value EV is (1) equal to or less than the threshold TH, the filtering unit 90 performs median filtering using the three-dimensional median filter 94 . Further, when the evaluation value EV exceeds (3) the threshold TH′, the filter processing unit 90 performs median filtering by the two-dimensional median filter 92 . Further, when the evaluation value EV (2) exceeds the threshold TH and is equal to or less than the threshold TH′, the filter processing unit 90 performs the pixel value after the median filtering by the two-dimensional median filter 92 and The processed pixel value is synthesized at a predetermined ratio (for example, 50%:50%), and the synthesized pixel value is obtained as the pixel value of the pixel of interest after median filtering. As a result, it is possible to obtain a higher noise reduction effect (effect of the three-dimensional median filter 94) while suppressing subject blurring and artifacts (effect of the two-dimensional median filter 92).

The "predetermined ratio" may be a constant value regardless of the evaluation value EV, or may be a variable value that varies according to the evaluation value EV. In the latter case, the "predetermined ratio" is such that the closer the evaluation value EV is to the threshold TH, the higher the ratio of the pixel values subjected to median filtering by the three-dimensional median filter 94, and the closer the evaluation value EV is to the threshold TH', The proportion of pixel values in the median filtering process by the two-dimensional median filter 92 is high.

In the above embodiment, noise is removed from the image by median filtering, but the configuration of the present invention is not limited to this. For example, instead of median filtering, trimming average processing may be used to remove noise from an image. In the trimmed average process, m pixels with large pixel values and m pixels with small pixel values are subtracted from all pixels included in an n×n (n is a positive odd number) pixel block (n ² −2m) pixels are output. Here, if m is n ² /2, the number of remaining pixels is 1. Therefore, the median value can be determined from the pixel values of the pixels in the pixel block after the correction by the trimmed average process as well as the median filter process.

The invention described in the scope of claims at the time of filing of the present application will be additionally described below.
[Appendix 1]
a difference value obtaining unit that obtains a difference value between a pixel value of the target pixel and a pixel value of each of the plurality of peripheral pixels for a pixel block that includes the target pixel and a plurality of peripheral pixels;
a pixel value correction unit that corrects the pixel values of the peripheral pixels based on the obtained difference value;
a filter processing unit that determines a median value among pixel values of pixels in the pixel block corrected by the pixel value correction unit;
comprising
Image processing device.
[Appendix 2]
further comprising a difference value determining unit that determines whether the difference value obtained by the difference value obtaining unit exceeds a first threshold for each of the plurality of peripheral pixels;
The pixel value correction unit corrects the pixel values of peripheral pixels in which the difference value exceeds the first threshold such that the difference value corresponds to the first threshold.
The image processing device according to appendix 1.
[Appendix 3]
The difference value obtaining unit obtains a difference value between a pixel value of the pixel of interest and a pixel value of each of the plurality of peripheral pixels for a current pixel block, which is a pixel block in an image of the current frame, and For each pixel of a reference pixel block that was located at the same position as the current pixel block in an image n (n is a natural number) frames before the current frame, the difference in pixel value from the target pixel of the current pixel block. get the value and
The pixel value correcting unit adjusts the pixel value of a pixel whose difference value from a target pixel in the current pixel block exceeds the first threshold so that the difference value corresponds to the first threshold. correct the value,
The filter processing unit performs filter processing for determining a median value among pixel values of pixels of the current pixel block and the reference pixel block after correction by the pixel value correction unit.
The image processing device according to appendix 2.
[Appendix 4]
The difference value obtaining unit also obtains a difference value between an average pixel value of target pixels in the current pixel block and the reference pixel block and a pixel value of the target pixel in the current pixel block,
The pixel value correction unit, when a difference value between the average pixel value and the pixel value of the pixel of interest in the current pixel block exceeds the first threshold value, also corrects the average pixel value of the pixel of interest in the current pixel block. Correcting so that the difference value from the pixel value becomes a value corresponding to the first threshold,
The filter processing unit performs filter processing for determining a median value from pixel values of pixels in the current pixel block and the reference pixel block after correction by the pixel value correction unit and the average pixel value.
The image processing device according to appendix 3.
[Appendix 5]
a subject change determination unit that determines a degree of subject change between frames based on luminance data of the current pixel block and the reference pixel block;
According to the determination result of the subject change determination unit, the filter processing unit performs filtering processing only for pixels in the current pixel block or for pixels in both the current pixel block and the reference pixel block. to filter,
The image processing apparatus according to appendix 3 or appendix 4.
[Appendix 6]
The subject change determination unit obtains a difference value between luminance data of the current pixel block and the reference pixel block, obtains a predetermined evaluation value based on the obtained difference value, and obtains a predetermined evaluation value for the evaluation value. Perform threshold judgment,
The filter processing unit performs filtering on only pixels in the current pixel block or on pixels in both the current pixel block and the reference pixel block, depending on the result of threshold determination for the evaluation value. to filter,
The image processing device according to appendix 5.
[Appendix 7]
the luminance data of the current pixel block is a luminance value of a pixel of interest in the current pixel block, an average luminance value of pixels in the current pixel block, or a median luminance value of pixels in the current pixel block;
The luminance data of the reference pixel block is a luminance value of a pixel of interest in the reference pixel block, an average luminance value of pixels in the reference pixel block, or a median luminance value of pixels in the reference pixel block.
The image processing device according to appendix 6.
[Appendix 8]
The filter processing unit performs filtering on pixels in both the current pixel block and the reference pixel block when the evaluation value is equal to or less than the second threshold, and the evaluation value exceeds the second threshold. if it exceeds, filter only the pixels of the current pixel block;
The image processing apparatus according to appendix 6 or appendix 7.
[Appendix 9]
a luminance data obtaining unit that obtains luminance data of a block of interest made up of a plurality of pixel groups including the pixel of interest;
a brightness determination unit that determines the brightness of the pixel of interest based on the acquired brightness data;
When the brightness determination unit determines that the pixel of interest is dark, the difference value determination unit performs threshold determination on the difference value using a third threshold that is larger than the first threshold.
The image processing apparatus according to any one of appendices 2 to 8.
[Appendix 10]
The difference value obtaining unit arranges the difference value between the pixel value of the pixel of interest and the pixel value of each of the plurality of peripheral pixels in the pixel block at a paired position with the pixel of interest interposed therebetween. obtaining a difference value between the average pixel value of the pixel pair obtained and the pixel value of the target pixel;
The difference value determination unit determines whether the difference value exceeds the first threshold for each of the pixel pairs,
The pixel value correcting unit corrects the average pixel value for pixel pairs in which the difference value exceeds the first threshold such that the difference value becomes a value corresponding to the first threshold,
The filter processing unit performs filter processing for determining a median value from the average pixel value of each pixel pair after correction by the pixel value correction unit and the pixel value of the target pixel.
The image processing apparatus according to any one of appendices 2 to 9.
[Appendix 11]
The filter processing by the filter processing unit is median filter processing or trim average processing for determining a median value among pixel values of pixels of each pixel block after correction by the pixel value correction unit.
The image processing apparatus according to any one of appendices 1 to 10.
[Appendix 12]
A computer-implemented image processing method comprising:
a difference value obtaining step of obtaining, for a pixel block including a pixel of interest and a plurality of peripheral pixels, a difference value between a pixel value of the pixel of interest and a pixel value of each of the plurality of peripheral pixels;
a pixel value correction step of correcting the pixel values of the peripheral pixels based on the obtained difference value;
a filter processing step of determining a median value among the pixel values of pixels in the pixel block corrected by the pixel value correcting step;
including,
Image processing method.
[Appendix 13]
An image processing program for causing a computer to execute the image processing method according to appendix 12.

1 : Image processing device 10 : Demosaicing unit 20 : YUV conversion unit 30 : Difference value acquisition unit 40 : Difference value determination unit 50 : Luminance data acquisition unit 60 : Brightness determination unit 70 : Pixel value correction unit 80 : Subject change determination unit 90: filter processing unit 92: two-dimensional median filter 94: three-dimensional median filter 96: output unit 100: addition unit 110: post-processing unit

Claims (13)

a difference value obtaining unit that obtains a difference value between a pixel value of the target pixel and a pixel value of each of the plurality of peripheral pixels for a pixel block that includes the target pixel and a plurality of peripheral pixels;
a pixel value correction unit that corrects the pixel values of the peripheral pixels based on the obtained difference value;
a filter processing unit that determines a median value among pixel values of pixels in the pixel block corrected by the pixel value correction unit;
comprising
Image processing device. further comprising a difference value determining unit that determines whether the difference value obtained by the difference value obtaining unit exceeds a first threshold for each of the plurality of peripheral pixels;
The pixel value correction unit corrects the pixel values of peripheral pixels in which the difference value exceeds the first threshold such that the difference value corresponds to the first threshold.
The image processing apparatus according to claim 1. The difference value obtaining unit obtains a difference value between a pixel value of the pixel of interest and a pixel value of each of the plurality of peripheral pixels for a current pixel block, which is a pixel block in an image of the current frame, and For each pixel of a reference pixel block that was located at the same position as the current pixel block in an image n (n is a natural number) frames before the current frame, the difference in pixel value from the target pixel of the current pixel block. get the value and
The pixel value correcting unit adjusts the pixel value of a pixel whose difference value from a target pixel in the current pixel block exceeds the first threshold so that the difference value corresponds to the first threshold. correct the value,
The filter processing unit performs filter processing for determining a median value among pixel values of pixels of the current pixel block and the reference pixel block after correction by the pixel value correction unit.
The image processing apparatus according to claim 2. The difference value obtaining unit also obtains a difference value between an average pixel value of target pixels in the current pixel block and the reference pixel block and a pixel value of the target pixel in the current pixel block,
The pixel value correction unit, when a difference value between the average pixel value and the pixel value of the pixel of interest in the current pixel block exceeds the first threshold value, also corrects the average pixel value of the pixel of interest in the current pixel block. Correcting so that the difference value from the pixel value becomes a value corresponding to the first threshold,
The filter processing unit performs filter processing for determining a median value from pixel values of pixels in the current pixel block and the reference pixel block after correction by the pixel value correction unit and the average pixel value.
The image processing apparatus according to claim 3. a subject change determination unit that determines a degree of subject change between frames based on luminance data of the current pixel block and the reference pixel block;
According to the determination result of the subject change determination unit, the filter processing unit performs filtering processing only for pixels in the current pixel block or for pixels in both the current pixel block and the reference pixel block. to filter,
The image processing apparatus according to claim 3 or 4. The subject change determination unit obtains a difference value between luminance data of the current pixel block and the reference pixel block, obtains a predetermined evaluation value based on the obtained difference value, and obtains a predetermined evaluation value for the evaluation value. Perform threshold judgment,
The filter processing unit performs filtering on only pixels in the current pixel block or on pixels in both the current pixel block and the reference pixel block, depending on the result of threshold determination for the evaluation value. to filter,
The image processing apparatus according to claim 5. the luminance data of the current pixel block is a luminance value of a pixel of interest in the current pixel block, an average luminance value of pixels in the current pixel block, or a median luminance value of pixels in the current pixel block;
The luminance data of the reference pixel block is a luminance value of a pixel of interest in the reference pixel block, an average luminance value of pixels in the reference pixel block, or a median luminance value of pixels in the reference pixel block.
The image processing apparatus according to claim 6. The filter processing unit performs filtering on pixels in both the current pixel block and the reference pixel block when the evaluation value is equal to or less than the second threshold, and the evaluation value exceeds the second threshold. if it exceeds, filter only the pixels of the current pixel block;
The image processing apparatus according to claim 6 or 7. a luminance data obtaining unit that obtains luminance data of a block of interest made up of a plurality of pixel groups including the pixel of interest;
a brightness determination unit that determines the brightness of the pixel of interest based on the acquired brightness data;
When the brightness determination unit determines that the pixel of interest is dark, the difference value determination unit performs threshold determination on the difference value using a third threshold that is larger than the first threshold.
The image processing apparatus according to any one of claims 2 to 8. The difference value obtaining unit arranges the difference value between the pixel value of the pixel of interest and the pixel value of each of the plurality of peripheral pixels in the pixel block at a paired position with the pixel of interest interposed therebetween. obtaining a difference value between the average pixel value of the pixel pair obtained and the pixel value of the target pixel;
The difference value determination unit determines whether the difference value exceeds the first threshold for each of the pixel pairs,
The pixel value correcting unit corrects the average pixel value for pixel pairs in which the difference value exceeds the first threshold such that the difference value becomes a value corresponding to the first threshold,
The filter processing unit performs filter processing for determining a median value from the average pixel value of each pixel pair after correction by the pixel value correction unit and the pixel value of the target pixel.
The image processing apparatus according to any one of claims 2 to 9. The filter processing by the filter processing unit is median filter processing or trim average processing for determining a median value among pixel values of pixels of each pixel block after correction by the pixel value correction unit.
The image processing apparatus according to any one of claims 1 to 10. A computer-implemented image processing method comprising:
a difference value obtaining step of obtaining, for a pixel block including a pixel of interest and a plurality of peripheral pixels, a difference value between a pixel value of the pixel of interest and a pixel value of each of the plurality of peripheral pixels;
a pixel value correction step of correcting the pixel values of the peripheral pixels based on the obtained difference value;
a filter processing step of determining a median value among the pixel values of pixels in the pixel block corrected by the pixel value correcting step;
including,
Image processing method. An image processing program for causing a computer to execute the image processing method according to claim 12.

Images