JP2017041763A - Image processing system and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system capable of acquiring an excellent correction image while maintaining a restoration effect in accordance with optical characteristics, an imaging apparatus, a program, a recording medium, and an image processing method system.SOLUTION: The image processing system includes: first acquiring means for acquiring photography conditions of a photography image; second acquiring means for acquiring an image restoration filter on the basis of the photography conditions and a specific frequency gain corresponding to the image restoration filter; and processing means for summing up at a predetermined rate images corrected by using the photography image and the image restoration filter on the basis of information on noise characteristics and a gain out of the photography conditions.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus and an image processing method.

  Due to the influence of diffraction, aberration, etc. generated in the imaging optical system, light generated from one point of the subject does not converge to one point but has a very small spread. A distribution having such a small spread is called a point spread function (PSF). Due to the influence of the imaging optical system, the captured image is formed by convolving the PSF with the subject image, and the image is blurred and the resolution is deteriorated.

  In recent years, it has become common to hold captured images as electronic data, and a technique for correcting image degradation due to an optical system by image processing (hereinafter referred to as an image restoration technique) has been proposed. However, when recovering an image by image processing, there is a problem that the amount of correction increases and noise data is amplified simultaneously with the recovery processing, particularly when deterioration is large. For this reason, depending on the amount of noise in the original image, performing the recovery process may result in undesirable results.

  Patent Document 1 discloses a method of correcting a PTF (Phase Transfer Function) component of optical deterioration information by image restoration processing and correcting an MTF (Modulation Transfer Function) component by sharpness processing. Further, in Patent Document 2, in order to keep the quality of the image corrected by the image restoration processing constant, a method for setting the filter gain at the time of image restoration to be constant, or a quality determination according to the value of the restoration gain. A method for changing the reference value is disclosed.

Japanese Patent No. 5546229 JP 2009-033561 A

  However, in the method of Patent Document 1, the phase component (PTF) is improved, but the amplitude component (MTF) depends on the sharpness processing. In particular, in the case of an image having a lot of noise, the edge detection accuracy is lowered, and a desired recovery effect cannot be obtained. Also, with the method of Patent Document 2, it is not possible to obtain an optimum result with respect to an image taken in various shooting states such as a digital camera.

  In view of such a problem, the present invention provides an image processing device, an imaging device, a program, a storage medium, and an image processing method capable of maintaining a recovery effect and obtaining a good corrected image according to optical characteristics. To do.

  An image processing apparatus according to an aspect of the present invention includes a first acquisition unit that acquires a shooting condition of a captured image, an image restoration filter based on the shooting condition, and a gain at a specific frequency corresponding to the image restoration filter. And a process of adding the captured image and the image corrected by using the image restoration filter at a predetermined ratio based on the information on noise characteristics in the photographing condition and the gain. And means.

  According to another aspect of the present invention, there is provided an image processing method comprising: obtaining a photographing condition of a photographed image; an image restoration filter based on the photographing condition; and a gain at a specific frequency corresponding to the image restoration filter. And a step of adding the photographed image and the image corrected by using the image restoration filter at a predetermined ratio based on information on noise characteristics in the photographing condition and the gain. It is characterized by that.

  According to the present invention, it is possible to provide an image processing device, an imaging device, a program, a storage medium, and an image processing method capable of maintaining a recovery effect and obtaining a good corrected image according to optical characteristics.

It is a block diagram of an imaging device provided with an image processing device concerning an embodiment of the present invention. It is explanatory drawing of an image restoration filter. It is explanatory drawing of an image restoration filter. It is explanatory drawing of a point spread function. It is explanatory drawing of the amplitude component and phase component of an optical transfer function. It is a flowchart which shows an image restoration process (1st, 2nd embodiment). It is a figure which shows the relationship between ISO sensitivity and a maximum gain (1st Embodiment). It is explanatory drawing of the maximum gain (1st Embodiment). It is explanatory drawing of the weighting for every frequency and a target frequency (2nd Embodiment). 12 is a flowchart illustrating image restoration processing (third embodiment). It is explanatory drawing of a maximum gain (3rd Embodiment). It is a figure which shows the relationship between a maximum gain and a weighted addition rate (3rd Embodiment).

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted. First, definitions of terms and image restoration processing (image processing method) described in the present embodiment will be described. The image restoration processing described here is used as appropriate in each embodiment described later.
[Input image]
The input image is a digital image (photographed image) obtained by receiving light with an image pickup device via an image pickup optical system, and is deteriorated by an optical transfer function OTF due to aberration of the image pickup optical system including a lens and various optical filters. doing. The imaging optical system can be configured using not only a lens but also a mirror (reflection surface) having a curvature.

  The color component of the input image has information on RGB color components, for example. As the color component, other commonly used color spaces such as brightness, hue, and saturation expressed in LCH, luminance expressed in YCbCr, and color difference signals can be selected and used. XYZ, Lab, Yuv, and JCh can be used as other color spaces. Further, a color temperature may be used.

The input image and the output image can be accompanied by shooting conditions such as the focal length of the lens, aperture value, shooting distance, and various correction information for correcting this image. When the image is transferred from the imaging apparatus to another image processing apparatus and correction processing is performed, it is preferable to add information regarding shooting conditions and correction to the captured image. As another delivery method of information regarding imaging conditions and correction, the imaging apparatus and the image processing apparatus may be directly or indirectly connected to be delivered.
[Image recovery processing]
Next, an outline of the image restoration process will be described. When the captured image (degraded image) is g (x, y), the original image is f (x, y), and the point spread function PSF that is a Fourier pair of the optical transfer function OTF is h (x, y). The following formula (1) is established.

g (x, y) = h (x, y) * f (x, y) (1)
Here, * is convolution (convolution integration, sum of products), and (x, y) are coordinates on the captured image.

  Moreover, if Formula (1) is Fourier-transformed and converted into a display format on the frequency plane, Formula (2) represented by the product for each frequency is obtained.

G (u, v) = H (u, v) · F (u, v) (2)
Here, H is an optical transfer function OTF obtained by Fourier transforming the point spread function PSF (h), and G and F are obtained by Fourier transforming the degraded image g and the original image f, respectively. Function. (U, v) is a coordinate on a two-dimensional frequency plane, that is, a frequency.

  In order to obtain the original image f from the photographed degraded image g, both sides may be divided by the optical transfer function H as shown in the following equation (3).

G (u, v) / H (u, v) = F (u, v) (3)
The original image f (x, y) is obtained as a restored image by performing inverse Fourier transform on F (u, v), that is, G (u, v) / H (u, v), and returning it to the actual surface. .

When R is the result of inverse Fourier transform of H− ¹ , the original image f (x, y) is similarly obtained by performing convolution processing on the actual image as in the following equation (4). Can be obtained.

g (x, y) * R (x, y) = f (x, y) (4)
Here, R (x, y) is called an image restoration filter. When the image is a two-dimensional image, generally, the image restoration filter R is also a two-dimensional filter having taps (cells) corresponding to each pixel of the image. In general, the greater the number of taps (number of cells) of the image recovery filter R, the higher the recovery accuracy. Therefore, the number of taps that can be realized is set according to the required image quality, image processing capability, aberration characteristics, and the like. Since the image restoration filter R needs to reflect at least aberration characteristics, it is different from a conventional edge enhancement filter (high-pass filter) of about 3 taps each in horizontal and vertical directions. Since the image restoration filter R is set based on the optical transfer function OTF, it is possible to correct both the amplitude component and the deterioration of the phase component with high accuracy.

  Since an actual image includes a noise component, using the image restoration filter R created by taking the reciprocal of the optical transfer function OTF as described above, the noise component is greatly amplified along with the restoration of the deteriorated image. End up. This is because the MTF is raised so that the MTF (amplitude component) of the optical system is returned to 1 over the entire frequency in a state where the amplitude of noise is added to the amplitude component of the image. The MTF, which is amplitude degradation due to the optical system, returns to 1, but at the same time, the noise power spectrum also rises. As a result, the noise is amplified according to the degree to which the MTF is raised (recovery gain).

  Therefore, when noise is included, a good image cannot be obtained as a viewing image. This is expressed by the following formulas (5-1) and (5-2).

G (u, v) = H (u, v) .F (u, v) + N (u, v) (5-1)
G (u, v) / H (u, v) = F (u, v) + N (u, v) / H (u, v) (5-2)
Here, N is a noise component.

  For an image including a noise component, there is a method of controlling the degree of recovery according to the intensity ratio SNR between the image signal and the noise signal, for example, as in the Wiener filter expressed by the following formula (6).

Here, M (u, v) is the frequency characteristic of the Wiener filter, | H (u, v) | is the absolute value (MTF) of the optical transfer function OTF, and C is a constant. In this method, for each frequency, the smaller the MTF, the smaller the recovery gain (degree of recovery), and the larger the MTF, the larger the recovery gain. In general, since the MTF of the imaging optical system is high on the low frequency side and low on the high frequency side, this method substantially reduces the recovery gain on the high frequency side of the image.

  Next, the image restoration filter will be described with reference to FIGS. The number of taps of the image restoration filter is determined according to the aberration characteristics of the imaging optical system and the required restoration accuracy. The image restoration filter in FIG. 2 is an 11 × 11 tap two-dimensional filter as an example. In FIG. 2, values (coefficients) in each tap are omitted, but a cross section of this image restoration filter is shown in FIG. The distribution of the values (coefficient values) of each tap of the image restoration filter has a function of returning the signal value (PSF) spatially spread due to aberration to the original one point ideally.

  Each tap of the image restoration filter is subjected to convolution processing (convolution integration, product sum) in the image restoration processing corresponding to each pixel of the image. In the convolution process, in order to improve the signal value of a predetermined pixel, the pixel is matched with the center of the image restoration filter. Then, the product of the signal value of the image and the coefficient value of the filter is taken for each corresponding pixel of the image and the image restoration filter, and the sum is replaced with the signal value of the central pixel.

  Next, with reference to FIGS. 4 and 5, the characteristics of the image restoration in the real space and the frequency space will be described. 4A and 4B are explanatory diagrams of the point spread function PSF. FIG. 4A shows the point spread function PSF before image restoration, and FIG. 4B shows the point spread function PSF after image restoration. FIG. 5 is an explanatory diagram of the amplitude component MTF (FIG. 5A) and the phase component PTF (FIG. 5B) of the optical transfer function OTF. In FIG. 5A, the broken line (A) indicates the MTF before image recovery, and the alternate long and short dash line (B) indicates the MTF after image recovery. Also, the broken line (A) in FIG. 5B indicates the PTF before image recovery, and the alternate long and short dash line (B) indicates the PTF after image recovery. As shown in FIG. 4A, the point spread function PSF before image restoration has an asymmetric spread, and the phase component PTF has a non-linear value with respect to the frequency due to this asymmetry. Since the image restoration process amplifies the amplitude component MTF and corrects the phase component PTF to be zero, the point spread function PSF after the image restoration has a symmetrical and sharp shape.

As described above, the image restoration filter can be obtained by performing inverse Fourier transform on a function designed based on the inverse function of the optical transfer function OTF of the imaging optical system. The image restoration filter used in the present embodiment can be appropriately changed. For example, the above-described Wiener filter can be used. When using the Wiener filter, it is possible to create an image restoration filter in real space that is actually convolved with the image by performing inverse Fourier transform on Equation (6). In addition, the optical transfer function OTF changes according to the image height (image position) of the imaging optical system even in one shooting state. For this reason, the image restoration filter is changed and used according to the image height.
[Imaging device]
With reference to FIG. 1, an imaging apparatus 200 including an image processing apparatus according to an embodiment of the present invention will be described. FIG. 1 is a configuration diagram of the imaging apparatus 200. An image processing program for performing image restoration processing of a captured image is installed in the imaging device 200, and this image restoration processing is executed by an image processing unit (image processing device) 204 inside the imaging device 200.

  The imaging apparatus 200 includes an imaging optical system (lens) 201 and an imaging apparatus main body (camera main body). The imaging optical system 201 includes a diaphragm 201a and a focus lens 201b, and is configured integrally with an imaging apparatus main body (camera main body). However, the present invention is not limited to this, and can also be applied to an imaging apparatus in which the imaging optical system 201 is replaceably attached to the imaging apparatus body.

  The image sensor 202 is composed of a CCD or CMOS sensor, and generates a captured image by photoelectrically converting a subject image (imaging light) obtained via the imaging optical system 201. That is, the subject image is converted into an analog signal (electric signal) by photoelectric conversion by the image sensor 202. The analog signal is converted into a digital signal by the A / D converter 203, and the digital signal is input to the image processing unit 204.

  The image processing unit 204 performs predetermined processing on the digital signal and also performs image restoration processing. As shown in FIG. 1, the image processing unit 204 includes an imaging condition acquisition unit (first acquisition unit) 204a, an optical transfer function acquisition unit (second acquisition unit), and a processing unit (processing unit). The imaging condition acquisition unit (acquisition means) 204 a acquires the imaging conditions of the imaging apparatus from the state detection unit 207. The shooting conditions include an aperture, a shooting distance, a focal length of the zoom lens, and the like. The state detection unit 207 may acquire the imaging condition directly from the system controller 210 or may acquire it from the imaging optical system control unit 206.

  The optical transfer function OTF or coefficient data necessary for generating the optical transfer function OTF is held in the storage unit (storage means) 208. The storage unit 208 is configured by a ROM, for example. The output image processed by the image processing unit 204 is stored in the image recording medium 209 in a predetermined format. On the display unit 205 constituted by a liquid crystal monitor or an organic EL display, an image obtained by performing a predetermined display process on the image subjected to the image restoration process is displayed. However, the image displayed on the display unit 205 is not limited to this, and an image subjected to simple processing for high-speed display may be displayed on the display unit 205.

  The system controller performs a series of controls in the imaging apparatus 200. The imaging optical system 201 is mechanically driven by the imaging optical system control unit 206 based on an instruction from the system controller 210. The imaging optical system control unit 206 controls the aperture diameter of the aperture 201a as the F number imaging state setting. In addition, the imaging optical system control unit 206 controls the position of the focus lens 201b by an unillustrated autofocus (AF) mechanism or manual manual focus mechanism in order to perform focus adjustment according to the subject distance. Note that functions such as aperture diameter control and manual focus of the aperture 201a may not be executed according to the specifications of the imaging apparatus 200.

  An optical element such as a low-pass filter or an infrared cut filter may be arranged in the imaging optical system 201. However, when an element that affects the characteristics of the optical transfer function OTF such as a low-pass filter is used, an image restoration filter is created. It may be necessary to consider at the point of time. The infrared cut filter also affects each of the RGB channel PSFs, particularly the R channel PSF, which is the integral value of the spectral wavelength point spread function (PSF), and therefore needs to be considered at the time of creating the image restoration filter. There is a case.

The image processing unit 204 is configured by an ASIC, and the imaging optical system control unit 206, the state detection unit 207, and the system controller 210 are each configured by a CPU or MPU. One or more of the image processing unit 204, the imaging optical system control unit 206, the state detection unit 207, and the system controller 210 may be configured to be shared by the same CPU or MPU.
[First embodiment]
With reference to FIG. 6, the image restoration process of this embodiment will be described. FIG. 6 is a flowchart showing the image restoration process. The flowchart in FIG. 6 is executed based on a command from the image processing unit 204.

  In step S11, a captured image is acquired. The captured image is stored in the storage unit 208. Further, an image stored in the image recording medium 209 may be acquired as a captured image.

  In step S12, the imaging condition acquisition unit 204a acquires the imaging conditions. The imaging conditions are a focal length, an aperture value, an imaging distance, and the like of the imaging optical system 201. In addition, noise characteristics (for example, ISO sensitivity) at the time of shooting are also acquired simultaneously. In the case of an imaging device in which a lens is replaceably attached to the camera body, the shooting conditions further include a lens ID and a camera ID. The shooting conditions may be acquired directly from the imaging device or may be acquired from information attached to the image.

  In step S13, the optical transfer function acquisition unit 204b acquires an optical transfer function OTF suitable for the imaging condition acquired in step S12. The optical transfer function OTF is selected from a plurality of optical transfer functions OTF held in advance. Further, a function for generating the optical transfer function OTF and a coefficient group used for generating the function may be stored in advance, and an optical transfer function OTF suitable for imaging conditions may be newly generated. Furthermore, an optical transfer function OTF suitable for the photographing conditions may be generated by interpolation processing from the optical transfer function OTF held in advance. In this case, it is possible to reduce the amount of data stored in the image restoration filter. As the interpolation processing, for example, bilinear interpolation (linear interpolation), bicubic interpolation, or the like is used, but is not limited thereto.

  In step S14, the maximum gain applicable at the time of shooting is acquired from the characteristic table of FIG. 7 stored in the storage unit 208 according to the ISO sensitivity among the shooting conditions acquired in step S12. FIG. 7 is a diagram illustrating an example of the relationship between ISO sensitivity and maximum gain. In FIG. 7, the horizontal axis represents ISO sensitivity, and the vertical axis represents maximum gain. In the characteristic table of FIG. 7, a maximum gain Gmax allowable in the system, a minimum gain Gmin necessary for producing an effect, and ISO sensitivities a and b for switching these values are set, and the ISO sensitivity therebetween is linear. Is interpolated.

  In the image restoration process, it is necessary to calculate the restoration filter characteristic using an expression such as a Wiener filter expressed by Expression (6). Usually, the recovery filter characteristic is calculated by determining the constant C in the equation (6). However, when the imaging apparatus 200 is used, there is a possibility that the S / N ratio of the output image changes depending on the readout method of the imaging element 202. A typical factor that changes the S / N ratio is the ISO sensitivity setting. ISO sensitivity setting means that, for example, when a dark subject is photographed at a predetermined aperture value or shutter speed, an image with a desired brightness is obtained by applying gain to the image signal by the image sensor 202 or the A / D converter 203. This setting is possible. When the gain increases, the noise signal superimposed on the captured image is amplified, and the output image becomes an image with a noticeable noise. In addition to the ISO sensitivity setting, the S / N ratio may change if the driving method changes such as long-time shooting or moving image shooting. In this embodiment, the case of ISO sensitivity setting will be described for simplicity. When the constant C in Expression (6) determined under a predetermined condition is used even when the ISO sensitivity is changed, the constant C is applied to an image having a worse S / N ratio than the condition assumed by the image restoration filter. There is a case. At this time, the image restoration filter amplifies noise, which is not preferable for a captured image. Therefore, it is preferable to define the maximum gain that can be set by the image restoration filter according to the ISO sensitivity.

  In this embodiment, the ISO sensitivity and maximum gain characteristic table is stored. However, the maximum gain is calculated by estimating the noise component of the image from the OB (Optical Black) region of the captured image. Also good. In that case, the storage unit 208 may hold a table with the S / N of the captured image on the horizontal axis and the maximum gain on the vertical axis. Further, the maximum gain is acquired by the same method in the case of long-second shooting or moving image shooting.

  In step S15, the processing unit 204c generates an image restoration filter having a gain characteristic corresponding to the ISO sensitivity, using the optical transfer function OTF of the imaging optical system 201. The generated image restoration filter is an image restoration filter obtained in consideration of the characteristics of the optical transfer function OTF from the viewpoint of noise characteristics of the image sensor 202.

  Here, a method for generating an image restoration filter will be described with reference to FIG. FIG. 8A is a diagram showing MTF characteristics of different optical characteristics (A) and (B). In FIG. 8A, the horizontal axis represents the spatial frequency, and the vertical axis represents the amplitude characteristic (MTF). FIG. 8B is a diagram illustrating gain characteristics when an image restoration filter is generated by the Wiener filter represented by Expression (6). In FIG. 8B, the horizontal axis represents the spatial frequency and the vertical axis represents the gain. In the equation (6), a constant C is incorporated so as to reduce the gain on the high frequency side, but the way of decreasing the gain differs depending on the original optical characteristics as shown in FIG. In FIG. 8B, in the case of the optical characteristic (A), the maximum gain is Gmax (A) at the frequency fa, and in the case of the optical characteristic (B), the maximum gain is Gmax (B) at the frequency fb. In this case, the gain is reduced on the high frequency side, but the maximum gain is affected by the original optical characteristics, so it is difficult to manage the influence of noise amplification of the final image.

  Therefore, in the present embodiment, as shown in FIG. 8C, the maximum gain is defined as a predetermined value so as not to be affected by the original optical characteristics. In FIG. 8C, the maximum gain is Gmax for both the optical characteristics (A) and (B), and it is defined that no gain greater than Gmax is applied in the entire frequency band. In order to realize such characteristics, a function F (α) that defines the maximum gain shown in Expression (7) is added instead of the constant C in Expression (6).

α is a variable for defining the maximum gain. The expression represented by Expression (7) is an example for defining the maximum gain, and may be defined by other expressions or means.

  In this embodiment, it is possible to control the influence on noise by generating an image restoration filter that defines a maximum gain. That is, when the noise characteristics of the imaging apparatus 200 are known in advance, an optimal image restoration filter that does not depend on the original optical characteristics is generated by determining α in Expression (7) with a value corresponding to the noise characteristics. be able to.

  In step S <b> 16, the image processing unit 204 performs image restoration processing of the captured image using the image restoration filter generated in step S <b> 15. In other words, the image restoration process of the photographed image is performed by convolving the image restoration filter with the photographed image.

In step S17, a recovered image (output image) is output based on the result of the image recovery process in step S16.
[Second Embodiment]
In the first embodiment, it has been described that filter generation is performed by defining a maximum gain according to imaging conditions. In FIG. 8C, the maximum gain is reached at the frequency fb ′ in the optical characteristic (A), and the maximum gain is reached at the frequency fa ′ in the optical characteristic (B). Since the noise generated in the image sensor 202 is mainly a random component, the frequency near the Nyquist frequency is dominant. Although the method described in the first embodiment can manage the maximum gain in the entire frequency band, it is more effective to generate a filter that is managed in detail for a specific frequency of the target noise. Is effective.

  The image restoration process of the present embodiment is executed along the flow shown in FIG. In this embodiment, in step S15, an image restoration filter is generated by managing the frequency of the target noise in more detail than in the first embodiment. Since other steps are the same as those in the first embodiment, the description thereof is omitted. As in the first embodiment, the flowchart of FIG. 6 is executed based on a command from the image processing unit 204.

  FIG. 9 is an explanatory diagram of weighting and target frequency for each frequency. FIG. 9A is a diagram illustrating the MTF characteristic of a predetermined optical characteristic. FIG. 9B is a diagram illustrating the gain characteristics of the image restoration filter when the characteristics of FIG. 9A are defined by the maximum gain Gmax. The image restoration filters (A) to (C) are generated by changing the frequency to be weighted. The image restoration filter (A) places the highest importance on the low frequency, the image restoration filter (B) puts the importance on the low frequency next, and the image restoration filter (C) puts the highest priority on the high frequency. Such an image restoration filter may be set in advance or may be automatically generated by using dynamic parameters using Expression (8).

  In Expression (8), Gmax is set to α in Expression (7), and a function G (u, v) that gives weighting for each frequency is added. Note that other equations and methods may be used as long as the same can be realized without using the above equations.

  FIG. 9C is a diagram showing a change in MTF characteristics when image restoration processing is performed by each image restoration filter of FIG. 9B. In contrast to the original characteristics, the image restoration filter (A) is lifted at the lowest frequency, and the image restoration filter (C) is lifted at the highest frequency.

When the target frequency F_target is determined, the gain relating to the frequency F_target can be acquired from the gain characteristic of the image restoration filter generated by changing the function G (u, v). In FIG. 9B, the gains applied to the frequency F_target of the image restoration filters A, B, and C are Ga, Gb, and Gc, respectively. Thus, by preliminarily specifying the maximum gain applied to the frequency F_target based on the noise characteristics of the image sensor 202, it is possible to generate an image restoration filter that can control noise reduction in more detail with high accuracy. In that case, if the weighting is extremely low or high, there is a possibility that harmful effects such as ringing may occur. Therefore, the function G (u, v) needs to be defined in an optimal range in advance. Further, when the gain characteristic is extremely lowered (particularly less than 1), the image becomes sleepy due to the LPF effect. Therefore, it is preferable to define the minimum gain as well as the maximum gain applied to the frequency F_target.
[Third embodiment]
In the first and second embodiments, the method of generating an image restoration filter by performing inverse Fourier transform on the acquired optical transfer function OTF has been described. Ideally, it is preferable to perform processing in the frequency domain as in the first and second embodiments. However, since inverse Fourier transform is required, if the calculation speed of the imaging apparatus 200 and the product cost are not met, It may be difficult to perform the inverse Fourier transform process. In this case, it is conceivable to perform an inverse Fourier transform externally in advance, store the calculated spatial filter in the storage unit 208, extract an image restoration filter that meets the imaging conditions, and perform an image restoration process. . In that case, it is necessary to calculate in advance all the spatial filters required as noise countermeasures, but the amount of storage in the storage unit 208 becomes enormous because it is necessary for the combination of the imaging conditions and the maximum gain.

  In this embodiment, when an image restoration filter is acquired from the storage unit 208, noise countermeasures are taken while minimizing the capacity stored in the storage unit 208. With reference to FIG. 10, the image restoration process of this embodiment is demonstrated. FIG. 11 is a flowchart showing image restoration processing according to the present embodiment. The flowchart in FIG. 10 is executed based on a command from the image processing unit 204.

  Steps S111 and S112 are the same as steps S11 and S12 in FIG.

  In step 113, an image restoration filter corresponding to the photographing condition acquired from the storage unit 208 in step S112 is acquired. In the first embodiment, the optical transfer function OTF is acquired. However, in the present embodiment, as described above, an image restoration filter obtained as a result of inverse Fourier transform of the optical transfer function OTF is acquired. At this time, a nearby image restoration filter may be used as an image restoration filter for shooting conditions that are not stored, or may be generated by interpolation from surrounding image restoration filters.

  In step S114, the maximum gain in the image restoration filter acquired in step S113 is acquired from the storage unit 208. The maximum gain will be described with reference to FIG. FIG. 11 is a diagram illustrating amplitude characteristics of different optical characteristics (A) and (B), and FIG. 11 (b) is a diagram illustrating maximum gains of the optical characteristics (A) and (B). In the optical characteristic (A), the maximum gain is Ga, and in the optical characteristic (B), the maximum gain is Gb. When the image restoration filter having the optical characteristics (A) and (B) is generated outside, the gain value of the target frequency (F_target) set in advance is linked with the image restoration filter and stored in the storage unit 208. Keep it. By doing this, it becomes possible to confirm later the gain characteristics in the frequency space that cannot be determined only from the image restoration filter. Since one piece of information per optical characteristic is sufficient, the size is negligible compared to the size of the image restoration filter, and the influence on the storage unit 208 is extremely small. If no image restoration filter is stored, a neighboring value may be used as the maximum gain, or it may be generated by interpolation from the surrounding maximum gain.

  In step S115, the weighted addition rate is determined from the imaging condition acquired in step S112 and the maximum gain acquired in step S114. With reference to FIG. 12, an example of a method for determining the weighted addition rate will be described. FIG. 12 is a diagram showing a weighted addition rate for maximum gain information at a predetermined ISO sensitivity. In FIG. 12, when the maximum gain is smaller than the maximum gain a, the weighted addition rate is 0%, that is, the image after correction using the image restoration filter becomes the final output as it is. On the other hand, when the maximum gain is greater than the maximum gain b, the weighted addition rate is 80%, that is, the weighted average is performed at a ratio of 80% for the uncorrected image and 20% for the corrected image. By doing this, although it is simple, in a region where the gain is high, it is possible to reduce noise deterioration by using a large number of original signals before correction. It is necessary to have this table for each ISO sensitivity that can be set. However, the maximum gain a under typical conditions and the slope between the maximum gains a and b are held, and the remaining ISO sensitivity table is appropriately calculated by numerical calculation. It may be generated. As described above, the case where the processing cannot be performed in the frequency space can be realized simply and with the memory consumption being minimized. In this embodiment, an example of ISO sensitivity has been described. However, similar to the first and second embodiments, the noise amount of an image may be estimated from the image and the same may be performed.

  In addition, this invention is implement | achieved also by performing the following processes. Processing for supplying software (program) for realizing the functions of each embodiment to a system or apparatus via a network or various recording media, and for the computer (or CPU, MPU, etc.) of the system or apparatus to read and execute the program It is.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

204 Image processing unit (image processing apparatus)
204a Imaging condition acquisition unit (first acquisition means)
204b Optical transfer function acquisition unit (second acquisition means)
204c Processing unit (processing means)

Claims (7)

First acquisition means for acquiring shooting conditions of a shot image;
Second acquisition means for acquiring an image restoration filter based on the imaging condition and a gain of a specific frequency corresponding to the image restoration filter;
Processing means for adding the photographed image and the image corrected by using the image restoration filter at a predetermined ratio based on information on noise characteristics in the photographing condition and the gain. Image processing device.   The image processing apparatus according to claim 1, further comprising a storage unit that stores the image restoration filter and the gain.   The image processing apparatus according to claim 1, wherein the information related to the noise characteristic is ISO sensitivity.   An image pickup apparatus comprising the image processing apparatus according to claim 1.   The program for making a computer function as each means of the image processing apparatus of any one of Claim 1 to 3.   A computer-readable recording medium on which the program according to claim 5 is recorded. Obtaining the shooting conditions of the shot image;
Acquiring an image restoration filter based on the imaging condition, and a gain of a specific frequency corresponding to the image restoration filter;
A step of adding the photographed image and the image corrected using the image restoration filter at a predetermined ratio based on information on noise characteristics in the photographing condition and the gain. Image processing method.