JP2004282686A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus in which even a comparatively large foreign object can be appropriately corrected. <P>SOLUTION: In the imaging apparatus, a white image resulting from calibration photographing is divided into a plurality of small areas BK, and a dust present area BL including a portion PL wherein luminance is lowered by the influence of dust entering an image pickup part, is detected. An inverted image (image represented with the spatial distribution of gains) Gb is generated from an inverse of a source image Ga corresponding to the dust present area BL. The inverted image Gb and the source image Ga are multiplied to obtain an image Gc in which the influence of the dust is removed. Namely, a dust corrected portion table corresponding to the inverted image Gb is generated and the dust corrected portion table and the photographed image are multiplied to exclude the influence of the dust (foreign object) from the photographed image. As a result, even the comparatively large foreign object can be appropriately corrected. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an imaging device having an imaging unit including an imaging element and an imaging optical system.
[0002]
[Prior art]
In a conventional digital camera, dust, dust, and dust (these are collectively referred to as “foreign matter”) adhere to an optical lens, an optical LPF (Low Pass Filter), and the like, and this is reflected in a photographed image and thus becomes a photographed image. May have black spots on a part of it.
[0003]
Causes of dust entering the imaging optical system include inadequate dust removal at the manufacturing stage in the factory, wear of mechanical members such as shutters and apertures inside the imaging optical system, and lenses. For example, there is an interchangeable single-lens reflex type digital camera that is sealed when a lens is replaced.
[0004]
In this regard, Patent Literature 1 and Patent Literature 2 disclose techniques for making foreign matter less noticeable by interpolation based on adjacent pixels.
[0005]
[Patent Document 1]
JP 2000-212314 A
[Patent Document 2]
JP 2002-209147 A
[0006]
[Problems to be solved by the invention]
However, the techniques of Patent Literature 1 and Patent Literature 2 are effective methods when the dust is at most 1 to 2 pixels, and thus it is difficult to apply the technique to dust having more than ten pixels.
[0007]
On the other hand, dust and the like often adhere to the optical LPF at the time of lens replacement. In this case, since there is a constant turn-in of light, information on the subject image is not completely lost due to the influence of the dust and the like.
[0008]
The present invention has been made in view of the above problems, and has as its object to provide an imaging apparatus capable of appropriately performing foreign matter correction even on a relatively large foreign matter.
[0009]
[Means for Solving the Problems]
In order to solve the above-mentioned problem, the invention according to claim 1 is an imaging apparatus having an imaging unit including an imaging device and an imaging optical system that forms a subject image on the imaging device, wherein (a) the imaging device includes: Image processing means for performing image processing on an image signal obtained by the element; and (b) storage means for storing, for a foreign substance present in the image pickup unit, foreign substance information indicating an influence of the foreign substance on the image signal. The image processing means includes: (a-1) foreign matter correction means for performing foreign matter correction on the image signal by digital processing involving multiplication processing with a data value obtained based on the foreign matter information.
[0010]
According to a second aspect of the present invention, in the imaging device according to the first aspect of the present invention, the image processing means includes: (a-2) a multiplier performing a multiplication process; and (a-3) multiplication by the multiplier. Shading correction means for performing shading correction on the image signal by digital processing accompanying the processing, wherein the foreign matter correction is performed using the multiplier or another multiplier of the same type as the multiplier. Is
[0011]
According to a third aspect of the present invention, in the imaging apparatus according to the first aspect of the present invention, the foreign matter correction unit includes a multiplier that performs the multiplication process and a DMA that transfers the foreign object information from the storage unit to the multiplier. And a controller.
[0012]
Further, the invention according to claim 4 is the imaging apparatus according to any one of claims 1 to 3, wherein the imaging apparatus has a calibration mode for performing calibration related to the foreign matter correction before or after photographing, The foreign substance information is generated based on information acquired by the calibration.
[0013]
According to a fifth aspect of the present invention, in the imaging apparatus according to the fourth aspect of the present invention, the foreign matter information is information of a captured image acquired by the imaging device in the calibration, or a reciprocal of a pixel level of the captured image. It is generated based on the information of
[0014]
According to a sixth aspect of the present invention, in the image pickup apparatus according to any one of the first to fifth aspects, the image processing means includes: (a-4) a plurality of blocks each of the image captured by the image sensor; And calculating means for calculating an exposure evaluation value for each block, and the foreign matter information is generated in units of the blocks.
[0015]
According to a seventh aspect of the present invention, in the imaging apparatus according to the fourth or fifth aspect, in the calibration, an achromatic chart is photographed in the calibration.
[0016]
According to an eighth aspect of the present invention, in the imaging apparatus according to the fourth or fifth aspect, in the calibration, foreign object information in a plurality of states in the imaging optical system is acquired.
[0017]
According to a ninth aspect of the present invention, in the imaging apparatus according to the fourth or fifth aspect, in the calibration, the exposure condition is changed while maintaining the state of the imaging optical system immediately before or immediately after shooting. Then, the foreign substance information is obtained.
[0018]
According to a tenth aspect of the present invention, in the imaging device according to any one of the first to ninth aspects, the image processing means can change a degree of noise processing on the image signal. The image processing apparatus further includes a noise processing unit, wherein the noise processing unit determines a degree of noise processing of a foreign substance image portion in which the influence of the foreign substance appears in the captured image acquired by the imaging device, with the captured image excluding the foreign substance image part. There is a noise reduction processing means for performing noise reduction processing differently.
[0019]
Further, according to an eleventh aspect of the present invention, in the imaging apparatus according to the tenth aspect, the noise reduction processing means performs the noise reduction processing at a degree corresponding to a degree of foreign matter correction of the foreign matter image portion by the foreign matter correcting means. Means for performing the following.
[0020]
According to a twelfth aspect of the present invention, in the imaging device according to the tenth or eleventh aspect, the noise reduction processing means performs the noise reduction processing at a degree corresponding to a degree of a luminance change in the foreign matter image portion. Have means to do so.
[0021]
According to a thirteenth aspect of the present invention, there is provided the image pickup apparatus according to the twelfth aspect, wherein (c) a luminance change detecting means for detecting the degree of the luminance change based on a photographed image acquired by the image sensor at the time of actual photographing. Is further provided.
[0022]
BEST MODE FOR CARRYING OUT THE INVENTION
<First embodiment>
<Main components of imaging device>
FIG. 1 is a diagram illustrating functional blocks of an imaging device 1 according to the first embodiment of the present invention.
[0023]
The imaging device 1 is configured as a digital camera, and includes an imaging sensor 11, a signal processing unit 2 connected to the imaging sensor 11, an image processing unit 3 connected to the signal processing unit 2, and a camera control unit 40. ing.
[0024]
The imaging sensor 11 is configured as an area imaging sensor having a single-plate pixel array in which R (red), G (green), and B (blue) primary color transmission filters are arranged in a checkered pattern, and is an all-pixel readout type. Image sensor. In the image sensor 11, when the accumulation of the electric charge is completed, the photoelectrically converted signal is shifted to a light-shielded transfer path, read out therefrom via a buffer, and formed into an image by an imaging optical system including the lens group 13. An image signal relating to the subject image is output. An optical LPF 12 is provided in front of the image sensor 11. Note that the imaging unit of the imaging device 1 includes an imaging sensor 11, an optical LPF 12, and a lens group 13.
[0025]
The signal processing unit 2 has a CDS 21, an AGC 22, and an A / D conversion unit 23.
[0026]
The image signal output from the image sensor 11 is sampled by the CDS 21 to remove noise from the image sensor 11, and then subjected to sensitivity correction by the AGC 22.
[0027]
The A / D converter 23 is configured by a 14-bit AD converter, and digitizes the analog signal normalized by the AGC 22. From the digitized image signal, an image file subjected to predetermined image processing by the image processing unit 3 is generated.
[0028]
The image processing unit 3 has a CPU and a memory, and includes a digital processing unit 3a, an image compression unit 37, a video encoder 38, and a memory card driver 39.
[0029]
The digital processing unit 3a includes a color balance evaluation integrator 30, a pixel interpolation unit 31, a resolution conversion unit 32, and a white balance control unit 33. Further, the digital processing unit 3a includes a shading correction unit 34, a dust correction unit 35, and a gamma correction unit 36.
[0030]
The image data input to the image processing unit 3 is written in an image memory 41 composed of, for example, an SDRAM in synchronization with the reading of the image sensor 11. Thereafter, the image data stored in the image memory 41 is accessed, and various processes are performed by the digital processing unit 3a.
[0031]
With respect to the image data on the image memory 41, pixel integration data for each small area is calculated for each color (RGB) by the color balance evaluation integrator 30. The small area is a divided area obtained by dividing one screen into many. By performing the integration of the pixel values in such a small area, an effect equivalent to performing the averaging process can be obtained. Using the calculation result of the color balance evaluation integrator 30, gain data of each RGB pixel is obtained.
[0032]
Then, the gain of each of the RGB pixels is independently corrected by the white balance control unit 33, and the RGB white balance is corrected. In this white balance correction, a portion that is originally considered to be white from the photographing subject is estimated from luminance, saturation data, and the like, and the average, G / R ratio, and G / B ratio of R, G, and B of the portion are obtained. Correction control is performed using the G / R ratio and the G / B ratio as R and B correction gains.
[0033]
The image data on which the white balance correction has been performed is subjected to so-called shading correction, in which a different gain is applied to each pixel position by the shading correction unit 34. The non-volatile memory 42 stores a shading correction table in which gain data of each pixel, which is derived from the optical performance of the lens group 13 in advance, is described. Then, at the time of shading correction, the shading correction section 34 multiplies the image data by the shading correction table transferred from the nonvolatile memory 42 to the image memory 41 and performs correction. This shading correction will be described later in detail.
[0034]
After the shading correction, the dust correction unit 35 performs dust correction (foreign matter correction). The non-volatile memory 42 stores a dust correction partial table derived in advance by calibration (described later). At the time of dust correction, similarly to the shading correction, the dust correction data transferred from the nonvolatile memory 42 to the image memory 41 and the image data are multiplied to perform the correction. This dust correction will be described later in detail.
[0035]
The image data subjected to the shading correction and the dust correction are subjected to masking of each of the R, G, and B pixels with the respective filter patterns by the pixel interpolator 31, and then, for G having pixels up to the high band, the surrounding four pixels by a median (intermediate value) filter. Is replaced with the average value of the intermediate two values. Average interpolation is performed on R and B.
[0036]
The RGB data in the interpolated image data is subjected to nonlinear conversion suitable for each output device in the gamma correction unit 36, and then Y, RY, and BY data are calculated from RGB by a matrix operation, and the image data is stored in the image memory. 41.
[0037]
The Y, RY, and BY data are reduced or thinned out horizontally and vertically, and the resolution conversion unit 32 converts the resolution into the number of pixels of the recorded image. In the monitor display as well, thinning is performed by the resolution converter 32, and a low-resolution image to be displayed on the LCD monitor 43 and the electronic viewfinder 44 is created.
[0038]
At the time of preview (live view), a low-resolution image of 640 * 240 pixels read from the image memory 41 is encoded into NTSC / PAL by the video encoder 38, and is converted into a field image by the electronic viewfinder 44 or the LCD monitor 43. Perform playback. Further, at the time of image recording, after the image of the set recording resolution is subjected to compression processing by the image compression unit 37, the compressed image is recorded on the memory card 9 by the memory card driver 39.
[0039]
The camera control unit 40 includes a CPU and a memory, and processes an operation input performed by a photographer on a camera operation switch 49 having a shutter button and the like. A calibration mode (described later) can be set by a photographer's operation input to the camera operation switch 49.
[0040]
During live view display in which the subject is displayed in a moving image manner on the LCD monitor 43 or the like before the main photographing, the optical aperture value of the shutter 45 is fixed at the open state by the aperture driver 46. Regarding the charge accumulation time of the image sensor 11, exposure control data is calculated by the camera control unit 40 based on light amount data measured in a photometry area selected from images acquired by the image sensor 11. Then, the feedback control is performed by the timing generator sensor driver 47 so that the exposure time of the image sensor 11 becomes appropriate by using a program chart set in advance based on the exposure control data calculated by the camera control unit 40.
[0041]
Further, in the image pickup apparatus 1, at the time of actual photographing, the exposure amount to the image pickup sensor 11 is controlled by the aperture driver 46 and the timing generator sensor driver 47 according to a preset program chart based on the light amount data measured as described above.
[0042]
<About the principle of dust correction>
In an interchangeable lens type single-lens reflex digital camera, dust enters the imaging optical system when the lens is replaced, and if the dust adheres to the optical lens or the optical LPF, the dust is projected on the imaging surface, and the dust appears on the captured image. The shadow of is reflected. As a result, a phenomenon occurs in which a specific portion of a captured image is always dark.
[0043]
Also, in a digital camera, the dust is generated when the mechanical members (aperture blades, mechanical shutters, etc.) inside the imaging optical system are worn out, and when dust or dirt enters during manufacturing at a factory. As in the case, a phenomenon occurs in which a part of the captured image becomes dark.
[0044]
In order to remove dust and the like existing in the image pickup unit of the image pickup apparatus 1, it is necessary to disassemble the camera once, clean all image pickup optical system members, and reassemble the camera in a normal lens fixed type digital camera. It becomes. On the other hand, even with a lens-interchangeable digital camera, special equipment is required to remove dust.
[0045]
Therefore, by applying the characteristic of shading correction, that is, "a different gain can be applied at a pixel position", dust, dust, and the like adhering to an imaging optical system (a lens, an optical LPF (Low Pass Filter), etc.) Dust correction for digitally correcting (removing the shadow of a foreign substance) a captured image in which a part of the image has become dark due to the influence of dust or the like (foreign substance) is performed. In the following, in order to simplify the description, dust will be described as a representative of foreign matter.
[0046]
FIG. 2 is a diagram for explaining the principle of dust correction.
[0047]
For example, as shown in FIG. 2A, the image is divided into a plurality of small areas BK in a matrix. The small area BK has a size of, for example, 40 (pixels) horizontally × 32 (pixels) vertically. In consideration of the analysis using the RAW image data, it is preferable that both the horizontal and vertical pixels have an even number of pixels. Then, based on the divided small area, a dust existence area BL (parallel hatched portion) in which a part PL whose luminance is reduced due to the influence of dust exists is detected.
[0048]
Here, as shown in FIG. 2B, the original image Ga corresponding to the dust existing area BL is multiplied by the correction image Gb generated by taking the reciprocal of the pixel level of each part of the original image Ga. This makes it possible to obtain an image Gc from which the influence of dust has been removed. Since the magnitude relationship between the pixel levels of the original image Ga is inverted, this correction image Gb is hereinafter referred to as an “inverted image Gb”. Each color component is represented by N bits, and is different from an image defined as a complement (2N−x) of N bits for the pixel level x of the original image Ga, and is a “gain inverted image” represented by a gain. is there. As described above, since the inverted image Gb is expressed in gain, a dust correction partial table is generated based on the information of the inverted image Gb, and the correction table is multiplied by a separately shot image to obtain a shot image. It is possible to erase trash from. The generation of the dust correction partial table will be described below.
[0049]
FIG. 3 is a diagram for explaining generation of the dust correction partial table.
[0050]
First, as shown in FIG. 3A, a dust presence area BL (thick frame portion) is detected from image data (RAW data) divided into small areas BK. This image data is obtained by performing calibration for photographing an achromatic chart, for example, a white chart (white image).
[0051]
Here, the luminance distribution at the position of the virtual line LI in the dust presence area BL is considered as shown in FIG. The horizontal axis of FIG. 3B indicates the horizontal position along the virtual line LI shown in FIG. 3A, and the vertical axis indicates the luminance value.
[0052]
Next, dust correction data (thick line portion) shown in FIG. 3C is generated in order to perform the brightness correction on the dust existing area BL. The horizontal axis of FIG. 3C indicates a horizontal position along the virtual line LI shown in FIG. 3A, and the vertical axis indicates a gain value. A method of generating the dust correction data will be described below.
[0053]
From all the small areas BL existing around the dust existence area BL, the average value of the pixel values for each of the RGB colors is obtained. The average value is divided by all the pixels in the dust presence area BL to obtain the gain of each pixel. In this case, if the gain value is equal to or less than a fixed value, for example, the equal-size line Ke, the equal-size is set. When the gain value exceeds the upper limit line Ku, clipping is performed at the upper limit line Ku. Thereby, an extremely large gain value can be eliminated.
[0054]
In FIG. 3C, dust correction data relating to a pixel column on the virtual line LI is shown. Is generated). The generated dust correction data is stored in the nonvolatile memory 42 as a dust correction partial table. This dust correction partial table functions as storage means for storing foreign matter information indicating the effect of dust (foreign matter) on image data (image signal).
[0055]
<About shading correction and dust correction>
FIG. 4 is a diagram for explaining shading correction and dust correction.
[0056]
The image memory 41 is divided into an image data storage area M1, a shading correction table storage area M2, a dust correction entire table M3, and a corrected image data storage area M4.
[0057]
Image data (RAW data) acquired by the image sensor 11 is stored in the image data storage area M1.
[0058]
The shading correction table storage area M2 stores a shading correction table configured as gain data. By matching the arrangement (array) of the gain data with the arrangement of the image data, that is, by associating both pixels with each other, each pixel data can be multiplied by a predetermined gain. That is, in the case of a shading correction table in which the higher the gain in the captured image, the higher the gain, the higher the gain near the image, that is, the higher the shading correction.
[0059]
The dust correction entire table storage area M3 stores a dust correction table. Since this dust correction entire table is transferred using a DMA (direct memory access) channel, data obtained by integrating all the dust correction partial tables constituted by a gain curve (thick line) Gn1 as shown in FIG. It is a table. That is, after filling the dust correction entire table having the data amount corresponding to the photographed image with the same size data (data having a gain value of 1), the dust correction entire table storage area M3 stores the address corresponding to the dust existing area BL in the dust correction entire table storage area M3. By writing the dust correction partial table, a dust correction entire table is generated. If there are a plurality of dust detection areas, each dust correction partial table is written to a position (address) corresponding to these areas.
[0060]
First, in the shading correction, the image data and the shading correction data are transferred from the image data storage area M1 and the shading correction table storage area M2 to the shading correction unit 34 using separate DMA channels (DMA controllers) D1 and D2. Then, the transferred image data and the shading correction data are digitally multiplied by a multiplier 34 a in the shading correction unit 34 to perform shading correction.
[0061]
Next, in the dust correction, the dust correction data is transferred to the dust correction unit 35 from the dust correction entire table storage area M3 by the DMA channel D3. Then, the transferred dust correction data (that is, the data value obtained based on the foreign matter information) and the image data after shading correction output from the shading correction unit 34 are output by a multiplier 35a of the same type as the multiplier 34a. The dust correction is performed by multiplying digitally.
[0062]
Finally, the image data subjected to shading correction and dust correction output from the dust correction unit 35 is stored in the corrected image data storage area M4 by the DMA channel D4.
[0063]
As described above, in the imaging apparatus 1, the position and density of dust obtained by performing the calibration work before or after shooting are stored in the camera as dust correction data, and the dust existing area in the shot image obtained at the time of actual shooting is stored. By digitally increasing the gain of BL, the shadow of dust is removed.
[0064]
<Operation of imaging apparatus 1>
Garbage has the following properties.
[0065]
(1) The place moves. It doesn't move in a short time, but stays in the same place for a long time.
[0066]
{Circle around (2)} The number increases due to lens replacement or mechanical member wear.
[0067]
As described above, since dust has a property that changes with time, it is preferable to warn the photographer to periodically perform calibration relating to dust correction. The warning operation regarding the calibration will be described below.
[0068]
FIG. 5 is a flowchart illustrating a calibration warning operation in the imaging apparatus 1. This operation is executed by the camera control unit 40.
[0069]
After the imaging device 1 is started, a timer (not shown) built in the imaging device 1 starts measuring time (step 1).
[0070]
In step S2, it is determined whether it is the timing for calibration. This timing is set to occur, for example, when a certain time has elapsed from the start of time measurement. Here, if it is the timing of the calibration, the process proceeds to step S3; otherwise, step S2 is repeated.
[0071]
In step S3, a warning is displayed on the LCD monitor 43 indicating that calibration is to be performed.
[0072]
In step S4, it is determined whether the calibration has been completed. If the calibration has been completed, the process proceeds to step S5. If the calibration has not been completed, the process returns to step S3.
[0073]
In step S5, it is determined whether the calibration has been completed normally. Here, when the calibration is successful, that is, when the calibration ends normally, the process proceeds to step S6, and when the calibration fails, the process returns to step S3.
[0074]
In step S6, the timer started in step S1 is reset.
[0075]
The above warning operation of the calibration allows the photographer to appropriately execute the calibration.
[0076]
Actually, the position of dust slightly changes depending on the lens, and depending on the aperture, the focal length, and the focal position, in relation to the light rays. For this reason, a warning of calibration may be issued immediately before and immediately after shooting.
[0077]
Hereinafter, an operation in a calibration mode for performing calibration in the imaging apparatus 1 will be described.
[0078]
FIG. 6 is a flowchart showing the operation in the calibration mode.
[0079]
When the photographer operates the camera operation switch 49 to set the calibration mode, first, a warning that the photographer will photograph a white image (eg, a white chart) is displayed on the LCD monitor 43 to warn the photographer.
[0080]
Then, zoom drive and aperture drive are performed (step S11). In the case of a power zoom lens, the imaging conditions (the conditions determined from the optical characteristics of the lens such as the focal length and the aperture) are automatically set so that the light drop in the peripheral portion is not erroneously detected as dust. After setting, the lens and the diaphragm are driven based on the imaging conditions. In the case of manual zooming, zoom driving is instructed, and in the case of a single focus lens, shooting at that focal length is instructed.
[0081]
Next, the photographer shoots a white image, and determines whether or not the shooting has been completed (step S12). Here, for example, it is determined whether the shutter button has been pressed by the photographer. If the shooting has been completed, the process proceeds to step S13. If the shooting has not been completed, step S12 is repeated.
[0082]
In step S13, it is determined whether the captured image is sufficiently bright. In the photographed white image data, pixel integration data for each small area (block) is calculated for each color component (RGB) by a color balance evaluation integrator 30, and it is determined whether the image is bright based on the exposure evaluation value. I do.
[0083]
Here, if it is sufficiently bright, the process proceeds to step S14. On the other hand, when the whole image is dark, the process is temporarily terminated as a calibration error, and a warning that calibration photographing is performed again is warned.
[0084]
In step S14, it is determined whether or not the RGB luminance difference is equal to or smaller than a predetermined threshold. This threshold is determined based on the noise level, and is determined including a margin. If the RGB luminance difference is equal to or smaller than the predetermined threshold, the process proceeds to step S15. On the other hand, if the value exceeds the predetermined threshold, the process is temporarily terminated as a calibration error, and a warning is issued that calibration photography is to be performed again.
[0085]
Through the operations in steps S13 and S14, it can be determined whether or not the captured image is an image for which a dust correction partial table can be created. That is, it is possible to determine whether or not an image satisfying the condition that the RGB evaluation values of the respective small areas are equal to or more than a certain value and are substantially equal in the dust detection target area at the center of the image.
[0086]
Next, when the captured image is determined to be a white image (gray image), analysis is performed for each small area (step S15).
[0087]
Here, for all the small areas, for example, the average value of the pixel integrated data for the G pixel is obtained. If there is a dark small area whose luminance is lower than the average value by a certain value or more, this small area is set to the dust existing area. Detected as However, since there is a possibility that the peripheral light amount of the image is reduced by the optical lens, the detection area is limited to only a portion (for example, a central portion of the image) where the light amount is small in consideration of the optical performance.
[0088]
In step S16, the dust correction unit 35 creates the dust correction partial table described above.
[0089]
In step S17, the dust correction partial table created in step S16 is stored in the non-volatile memory.
[0090]
In step S18, a normal end flag of the calibration is set. With this normal flag, the determination operation of step S5 shown in the flowchart of FIG. 5 can be performed.
[0091]
By the above operation of the imaging apparatus 1, a dust correction partial table is generated based on the image data acquired by the calibration shooting, and the dust correction is performed based on the dust correction partial table. .
[0092]
That is, a digital camera that has been processed as a defective product because a foreign substance is reflected in the image can be shipped as a non-defective product. Further, even in a case where dust enters in a lens exchange type single-lens reflex digital camera at the time of lens exchange and it is conventionally necessary to perform dust removal work using special equipment, this work can be omitted. Furthermore, even if a digital camera is used for a long time and dust generated due to wear of mechanical equipment inside the imaging optical system adheres to the optical system, the camera itself can perform dust correction, so that a good photographed image can be obtained. It becomes.
[0093]
Further, similarly to the shading correction, foreign matter correction is performed by digital processing using a multiplier and a DMA controller, so that highly accurate correction can be performed with a simple configuration.
[0094]
In the above-described imaging apparatus 1, only one DMA channel D3 (see FIG. 4) is used to transfer the dust correction data to the dust correction unit 35. However, even if a plurality of DMA channels are used. good.
[0095]
FIG. 7 is a diagram for explaining dust correction using a plurality of DMA channels.
[0096]
The image memory 41 has an image data storage area M1 having the same configuration as that of FIG. 4, a shading correction table storage area M2, and a corrected image data storage area M4.
[0097]
The image memory 41 is different from the dust correction entire table storage area M3 (FIG. 4) in that four table storage areas, namely, a dust correction partial table 1 storage area M31, a dust correction partial table 2 storage area M32, and a dust correction It has a partial table 3 storage area M33 and a dust correction partial table 4 storage area M34.
[0098]
Each of the dust correction partial table storage areas M31 to M34 stores a dust correction partial table corresponding to the dust presence area BL having a gain value as shown in FIG. However, since the number of dust corrections is limited due to DMA channel restrictions and the like, it is preferable to select the dust correction target in consideration of the position, size, and the like.
[0099]
Then, the dust correction data is transferred from the dust correction partial table storage areas M31 to M34 to the dust correction unit 35 by the DMA channels D31 to D34. Then, the transferred dust correction data and the image data after shading correction output from the shading correction unit 34 are multiplied by the multiplier 35a in the dust correction unit 35 as in the case of FIG. Performs dust correction. In this case, in each of the DMA channels D31 to D34, the start timing and the end timing of the data transfer and the amount of dust correction data are calculated based on the data in the dust presence area BL, and the DMA setting is performed.
[0100]
As described above, even by the dust correction using a plurality of DMA channels, appropriate foreign matter correction can be performed even for a relatively large foreign matter.
[0101]
<Second embodiment>
<Main components of imaging device>
FIG. 8 is a diagram illustrating functional blocks of an imaging device 5 according to the second embodiment of the present invention.
[0102]
The imaging device 5 of the second embodiment has a configuration similar to that of the imaging device 1 of the first embodiment, but differs mainly in that a noise reduction processor 51 is added. The noise reduction processor 51 will be described below.
[0103]
FIG. 9 is a diagram for explaining the noise reduction processor 51.
[0104]
The Y, RY, and BY data calculated by the matrix operation after the nonlinear conversion by the gamma correction unit 36 is input to the noise reduction processor 51, and a minute noise component included in the image data is input. Is performed. Then, the Y, RY, and BY data whose noise has been reduced by the noise reduction processor 51 are stored in the image memory 41 and input to the resolution conversion unit 32, after which the same processing as in the first embodiment is performed. Is
[0105]
The noise reduction processor 51 includes an LPF (Low Pass Filter) unit 511, an HPF (High Pass Filter) unit, a coring processing unit 513, a multiplier 514, and an adder 515, and converts the image data. On the other hand, the level (degree) of the noise reduction processing can be changed.
[0106]
The LPF unit 511 is configured as, for example, a 5 × 5 digital filter, and can dynamically change the noise reduction amount by changing its coefficient or order. For example, a plurality of filter coefficients used in the LPF unit 511 are previously stored in a noise reduction table 41a in an image memory 41 composed of an SDRAM, and a data table in which filter patterns used in each pixel are regularly arranged by a DMA channel D5. By sending the filter coefficient to the LPF unit 511, the filter coefficient can be switched for each pixel.
[0107]
The HPF unit 512 is configured as, for example, a 5 × 5 digital filter.
[0108]
The coring processing unit 513 has input / output characteristics as shown in FIG. That is, the input of the threshold value (−Th) 〜0 to the threshold value Th is fixed to “0” and output, and the input value equal to or less than the threshold value (−Th) outputs (input value + Th) and is equal to or more than the threshold value Th. By inputting (input value -Th), noise is reduced. The noise reduction amount can be dynamically changed by changing the threshold value Th. Specifically, a plurality of thresholds used in the coring processing unit 513 are stored in advance in the noise reduction table 41a in the image memory 41, and a data table in which the thresholds used for each pixel are arranged in order by the DMA channel D5 is stored. By sending the threshold value to the LFP unit 511, the threshold value can be switched for each pixel.
[0109]
The multiplier 514 is configured as a part for suppressing a noise component included in a high frequency component of the Y signal (luminance signal), and dynamically changes a noise reduction amount by controlling a gain of the Y signal. Is possible. For example, a plurality of gain data used in the multiplier 514 is stored in advance in the noise reduction table 41a in the image memory 41, and a data table in which the gain data used in each pixel is regularly arranged by the DMA channel D5 is stored in the multiplier 514. By transmitting, the gain of the high frequency component of the Y data can be switched for each pixel.
[0110]
The adder 515 adds the Y data processed by the LPF unit 511 and the Y data output from the multiplier 514. The Y data added by the adder 515 is stored in the Y data storage unit 41b in the image memory 41 by the DMA channel D6.
[0111]
With the above-described noise reduction processor 51, the noise reduction amount of a specific portion (specific pixel) in a captured image can be made stronger or weaker than that of a peripheral portion (peripheral pixel).
[0112]
On the other hand, in the imaging device 5, as described in the first embodiment, dust correction is performed by multiplying the gain by the dust existing area BL (FIG. 3A) detected by the calibration. For BL, noise is also locally increased. Therefore, although dust correction can be appropriately performed, dust erasure traces due to dust correction are conspicuous, and a partially unsightly image may be generated.
[0113]
Therefore, in such a case, the noise reduction amount for the dust presence area BL is set to be stronger by the noise reduction processor 51 than in the image parts other than the dust detection area BL. Thus, effective noise reduction can be performed on the dust presence area BL.
[0114]
However, when a high-frequency image is present in the dust presence area BL, the image in the dust detection area BL is reduced by increasing the noise reduction amount of the noise reduction processor 51, so that the resolution is partially low. An unsightly image will be generated. Specifically, as shown in FIG. 11A, when a high-frequency component (illustrated by four lines) HG exists in the dust detection area BL including the portion PM where the luminance is reduced due to the influence of dust, the above-described dust correction is performed. By performing the high-frequency noise reduction processing, the high-frequency component HG is blurred as in the dust presence area BL shown in FIG.
[0115]
Therefore, in the imaging device 5 of the second embodiment, based on the information on the dust presence area BL detected by the calibration, what kind of image exists in the portion of the captured image corresponding to the dust presence area BL, An evaluation value for the component is calculated. Then, by performing a noise reduction process based on the evaluation value, a natural image subjected to dust correction is generated. The operation of the imaging device 5 will be described in detail below.
[0116]
<Operation of imaging device 5>
First, the imaging device 5 obtains an evaluation value for detecting a frequency component at a dust detection point. As the evaluation value, for example, an output value of a Sobel Filter can be used, which will be described below.
[0117]
FIG. 12 is a diagram for explaining calculation of an evaluation value using a Sobel filter.
[0118]
The non-volatile memory 42 of the imaging device 5 stores four Sobel filters F1 to F4 (Sobel filter group FG) each configured by a matrix to which vertically symmetric, left-right symmetric, and oblique left-right symmetric coefficients are assigned. The output values of the Sobel filters F1 to F4 are summed by the adder AD and output as a line segment evaluation value. With such a configuration, since the output value from the filter having a matrix that matches the direction of the line segment among the four Sobel filters F1 to F4 increases, the line segment evaluation that is the sum of the output values of the Sobel filters F1 to F4 becomes large. If the magnitude of the value is determined, detection of a line segment, that is, detection of a high-frequency component can be performed. That is, the Sobel filter group FG and the adder AD function as a luminance change detecting unit that detects a degree of a luminance change in the image signal based on the image signal acquired by the image sensor 11 at the time of the main photographing.
[0119]
Here, since the dust detection portion obtained by the calibration is stored in the nonvolatile memory 42, the sobel filter F1 is used for the portion of the photographed image corresponding to the dust presence area BL (FIG. 3A). By obtaining the outputs of F4 to F4, it can be determined whether or not there is a high-frequency image at the dust correction location.
[0120]
Next, the imaging device 5 performs a noise reduction process based on the evaluation value calculated by the Sobel filter group FG. A series of operations of the imaging device 5 will be described below.
[0121]
First, after performing calibration in the same manner as in the first embodiment, the subject is photographed by a known photographing sequence. Then, after storing (buffering) the captured RAW image data in the image data storage area M1 in the image memory 41, the pixel interpolation unit 31 performs pixel interpolation.
[0122]
Next, a line segment evaluation value (evaluation value) is calculated for the image data on which the pixel interpolation has been performed by the above-described Sobel filter group FG, and line segment detection is performed. Specifically, based on the information on the dust presence area BL stored in the nonvolatile memory 42, a line segment evaluation value is calculated by the Sobel filter group FG and the adder AD for the captured image portion corresponding to the dust position. This calculation is preferably performed within a range that is affected by dust.
[0123]
Then, the average evaluation value per unit area is calculated by dividing the calculated line segment evaluation value by the area of the dust presence area BL to be subjected to dust correction. If the average evaluation value is larger than a predetermined reference value α, it is determined that a high-frequency component exists, and a setting for reducing the noise reduction amount so as to suppress the noise reduction processing in the noise reduction processor 51 is performed. Do. On the other hand, if the average evaluation value is equal to or less than the reference value α, it is determined that the image is a clear image having no high-frequency component, and the noise reduction amount is set so that the noise reduction processing in the noise reduction processor 51 is sufficiently performed. Set to rise. In these cases, a process of switching the data table (parameter) for noise reduction correction stored in the noise reduction table 41a of FIG. 9 according to the average evaluation value is performed. That is, the degree (level) of noise processing of the foreign matter image portion in the dust presence area BL in which the influence of dust (foreign matter) appears in the photographed image acquired by the imaging sensor 11 is made different from the photographed image excluding the foreign matter image portion, That is, the noise reduction processing is performed at a degree corresponding to the degree of the luminance change of the foreign matter image portion.
[0124]
After setting the noise reduction amount for the dust-existing area BL as described above, the RAW image data is again converted to pixel interpolation, gamma correction, Y, RY, and BY data as in the first embodiment. After performing the YCC image conversion and the dust correction, the noise reduction processor 51 performs an appropriate noise reduction process according to the high-frequency component to generate a dust corrected YCC image.
[0125]
By the operation of the imaging device 5 described above, the same effects as those of the first embodiment can be obtained, and the dust correction can be performed because the noise reduction process is performed by determining whether a high-frequency component exists in the dust correction portion. High-quality images can be generated. That is, when the dust correction is performed as in the first embodiment, an image in which noise is extremely increased only in the image portion subjected to the dust correction may be created. However, in the second embodiment, the dust correction is performed. However, even if the noise of the correction mark is not conspicuous and the high frequency component is present at the correction location, a natural image can be obtained without the high frequency component being dulled by the noise reduction processing.
[0126]
It is not essential to use a Sobel filter to detect a high-frequency component, and an AF evaluation contrast value that is an output value of a contrast calculator used for an AF evaluation value may be used.
[0127]
FIG. 13 is a diagram for explaining an evaluation value based on the AF evaluation contrast value.
[0128]
In the horizontal contrast calculator, the difference (| G−G |) between the luminance value of the target pixel itself and the luminance value of the same color pixel that is separated by a certain pixel in the horizontal direction is calculated for the integration area Et having a certain size. A contrast value is calculated by multiplying. Since this contrast value has a characteristic that it increases as the in-focus state approaches in the AF operation, it can be used as an AF evaluation value. When no feature point such as a high-frequency component exists in the integration region Et, the contrast value becomes a small value.
[0129]
By processing the captured image with the contrast calculator, it is possible to determine whether or not a feature point exists at a specific portion in the image. Therefore, if the contrast value is calculated for the dust-existing area BL, it can be determined whether or not this dust-existing area BL has a feature point (high-frequency component).
[0130]
That is, since the information of the dust presence area BL obtained by the calibration is stored in the nonvolatile memory 42, the AF evaluation contrast value is calculated for the portion of the captured image corresponding to the dust presence area BL. It can be determined whether there is a feature point in the image portion of the dust correction area BL.
[0131]
Before the actual photographing, since the position / range to be subjected to dust correction has been detected, the contrast is set before the RAW image is buffered after the integration area Et (FIG. 13) of the contrast value for AF evaluation is set. The arithmetic unit may calculate the AF evaluation contrast value. In the case of the Sobel filter, the use after pixel interpolation is restricted. However, in the case of the contrast value for AF evaluation, it can be used without any trouble for a RAW image. If the calculation of the contrast value has been completed, the subsequent processing is the same as in the case of the Sobel filter.
[0132]
Further, in the imaging device 5, the evaluation value of the noise reduction processing may be calculated based on the live view image immediately before the main imaging, not limited to the image acquired at the time of the main imaging. In this case, since the live view image immediately before photographing and the photographed main image rarely coincide completely, it is preferable to set an image determination area having a large area including the dust presence area BL.
[0133]
In a captured image, when a high-frequency component exists in a portion where dust is present, the dust itself generally becomes inconspicuous, but on the other hand, in a portion where there is no high-frequency component and there is no dust, the dust is conspicuous. There is nature.
[0134]
Therefore, the state of the image in the dust presence area BL, that is, the presence or absence of a high-frequency component is detected. If no high-frequency component is detected, dust correction is performed to perform normal-level noise reduction processing. Conversely, when a high frequency component is detected, as shown by a gain curve (bold line) Gn2 in FIG. 14 corresponding to FIG. The gain itself is kept low to reduce the increase in noise, and a normal level noise reduction process is performed.
[0135]
By performing the above-described processing, a dust-corrected image without a sense of incongruity can be generated without providing a circuit capable of performing different noise reduction processing for each part of the image as in the noise reduction processor 51 shown in FIG. That is, the configuration of the noise reduction processing in the imaging device 5 can be simplified.
[0136]
<Modification>
Regarding the calibration in each of the above embodiments, it is not essential to issue a warning at regular intervals by the timekeeping function of the imaging device, and a warning may be issued by detecting lens replacement. Further, a shock sensor may be provided to issue a warning when an output equal to or more than a predetermined value is detected.
[0137]
In each of the above embodiments, it is not essential to perform calibration by photographing a white chart (achromatic image), and calibration may be performed by inserting a diffusion plate.
[0138]
In the calibration, immediately before or immediately after shooting, the state of the imaging optical system is maintained, that is, shooting is performed while changing the exposure condition without changing the focal length and aperture of the lens, and dust correction data is acquired. Is also good.
[0139]
In each of the above embodiments, it is not essential to use different multipliers for shading correction and dust correction, and a multiplier may be used for shading correction and dust correction.
[0140]
◎ Regarding the dust correction partial table in each of the above embodiments, it is not essential to generate the dust correction partial table based on the inverted image generated from the reciprocal of the captured image data acquired at the time of calibration, but based on the captured image acquired at the time of calibration. May be generated. In this case, at the time of dust correction, the reciprocal of the correction data read from the dust correction partial table is calculated, and the reciprocal is multiplied by the image at the time of actual shooting.
[0141]
At the time of calibration, a plurality of dust correction data may be acquired by changing a plurality of states in the imaging optical system, for example, changing the focal length of the lens, the aperture, and the like. This makes it possible to perform foreign matter correction corresponding to various states of the imaging optical system.
[0142]
In the second embodiment, it is not essential to perform the noise reduction processing only on the Y signal, and the noise reduction processing may be performed on the RY signal and the BY signal. In addition, the noise reduction processing may be performed using a circuit in which some of the plurality of noise reduction portions included in the noise reduction processor 51 illustrated in FIG. 9 are omitted.
[0143]
In calculating the evaluation value in the second embodiment, it is not essential to use a Sobel filter, and a line segment detection filter such as a Prewitt filter may be used.
[0144]
In the second embodiment, for example, when the average correction gain of the dust presence area BL (FIG. 3) is increased, the level of the noise reduction processing is increased, and when the average correction gain is reduced, the noise is reduced. The level of the reduction processing may be reduced. That is, the noise reduction processing is performed at a degree corresponding to the degree of foreign matter correction of the image portion corresponding to the dust presence area BL. Thereby, appropriate noise reduction processing can be performed, and image quality can be improved.
[0145]
【The invention's effect】
As described above, according to the first to fifth aspects of the present invention, the image signal is multiplied by the data processing based on the foreign substance information indicating the influence of the foreign substance on the image signal. Since foreign matter correction is performed, appropriate foreign matter correction can be performed even for a relatively large foreign matter.
[0146]
In particular, in the invention of claim 2, foreign matter correction is performed using a multiplier used for shading correction or another multiplier of the same type as that, so that the configuration can be simplified and cost increase can be suppressed.
[0147]
According to the third aspect of the present invention, since the foreign matter correction means has a multiplier for performing multiplication processing and a DMA controller for transferring foreign matter information from the storage means to the multiplier, the configuration can be simplified and cost increase can be suppressed. .
[0148]
According to the fourth aspect of the present invention, the foreign substance information is generated based on information obtained by a calibration performed before or after the photographing, so that the foreign substance information can be appropriately generated.
[0149]
According to the fifth aspect of the present invention, since the foreign substance information is generated based on information of a photographed image acquired by the imaging device in the calibration or information of a reciprocal of a pixel level of the photographed image, the foreign substance information is easily generated. it can.
[0150]
In the invention of claim 6, since the foreign substance information is generated in units of blocks for calculating the exposure evaluation value, the memory can be saved and the configuration is simplified.
[0151]
According to the seventh aspect of the present invention, since the achromatic chart is photographed in the calibration, the foreign substance information can be appropriately acquired.
[0152]
According to the invention of claim 8, since the calibration acquires the foreign substance information in a plurality of states in the imaging optical system, foreign substance correction corresponding to various states of the imaging optical system can be performed.
[0153]
According to the ninth aspect of the invention, in the calibration, the foreign matter information is acquired by changing the exposure condition while maintaining the state of the imaging optical system immediately before or immediately after the photographing, so that optimal foreign matter information can be acquired.
[0154]
According to the tenth aspect of the present invention, the noise reduction processing is performed by making the degree of noise processing of a foreign image portion in which an influence of a foreign object appears in the captured image obtained by the image sensor different from that of the captured image excluding the foreign image portion. Therefore, appropriate noise correction processing can be performed on an image on which foreign matter correction has been performed, and image quality can be improved.
[0155]
According to the eleventh aspect of the present invention, since the noise reduction processing is performed to a degree corresponding to the degree of foreign matter correction of the foreign matter image portion by the foreign matter correction means, more appropriate noise reduction processing can be performed, and image quality can be further improved.
[0156]
In the twelfth aspect of the present invention, since the noise reduction processing is performed at a degree corresponding to the degree of luminance change in the foreign matter image portion, more appropriate noise reduction processing can be performed, and image quality can be further improved.
[0157]
According to the invention of claim 13, since the degree of the luminance change is detected based on the photographed image acquired by the image sensor at the time of the actual photographing, the degree of the luminance change can be detected with high accuracy.
[Brief description of the drawings]
FIG. 1 is a diagram showing functional blocks of an imaging device 1 according to a first embodiment of the present invention.
FIG. 2 is a diagram for explaining the principle of dust correction.
FIG. 3 is a diagram for explaining generation of a dust correction partial table.
FIG. 4 is a diagram for explaining shading correction and dust correction.
FIG. 5 is a flowchart showing a calibration warning operation in the imaging apparatus 1.
FIG. 6 is a flowchart illustrating an operation in a calibration mode.
FIG. 7 is a diagram for explaining dust correction using a plurality of DMA channels.
FIG. 8 is a diagram showing functional blocks of an imaging device 5 according to a second embodiment of the present invention.
FIG. 9 is a diagram for explaining a noise reduction processor 51.
FIG. 10 is a diagram for explaining input / output characteristics of a coring processing unit 513.
FIG. 11 is a diagram for describing noise reduction processing when a high-frequency image exists in the dust presence area BL.
FIG. 12 is a diagram for explaining calculation of an evaluation value using a Sobel filter.
FIG. 13 is a diagram for explaining an evaluation value based on an AF evaluation contrast value.
FIG. 14 is a diagram for explaining a gain curve Gn2 for dust correction.
[Explanation of symbols]
1, 5 imaging device
3 Image processing unit
11 Image sensor
12 Optical LPF
13 lens groups
34 Shading correction unit
34a, 35a Multiplier
35 Dust correction unit
40 Camera control unit
41 Image memory
41a Noise reduction table
42 Non-volatile memory
43 LCD monitor
51 Noise reduction processor
511 LPF section
512 HPF section
513 Coring processing unit
514 Multiplier
BL garbage area
D1-6, D31-34 DMA channel
F1-F4 Sobel filter
Gb inverted image
HG high frequency component

Claims (13)

An imaging device having an imaging unit including an imaging element and an imaging optical system that forms a subject image on the imaging element,
(A) image processing means for performing image processing on an image signal acquired by the image sensor;
(B) storage means for storing, for a foreign substance present in the imaging unit, foreign substance information indicating an influence of the foreign substance on the image signal;
With
The image processing means,
(A-1) foreign matter correction means for performing foreign matter correction on the image signal by digital processing involving multiplication processing with a data value obtained based on the foreign matter information;
An imaging device comprising: The imaging device according to claim 1,
The image processing means,
(A-2) a multiplier for performing a multiplication process;
(A-3) shading correction means for performing shading correction on the image signal by digital processing involving multiplication processing in the multiplier;
Has,
The imaging apparatus according to claim 1, wherein the foreign matter correction is performed using the multiplier or another multiplier of the same type as the multiplier. The imaging device according to claim 1,
The foreign matter correction means,
A multiplier for performing the multiplication process;
A DMA controller that transfers the foreign object information from the storage unit to the multiplier;
An imaging device comprising: The imaging device according to any one of claims 1 to 3,
Having a calibration mode for performing calibration related to the foreign matter correction before or after shooting,
The imaging apparatus according to claim 1, wherein the foreign substance information is generated based on information acquired by the calibration. The imaging device according to claim 4,
The imaging apparatus according to claim 1, wherein the foreign substance information is generated based on information on a captured image acquired by the imaging device in the calibration or information on a reciprocal of a pixel level of the captured image. The imaging device according to any one of claims 1 to 5,
The image processing means,
(A-4) calculating means for dividing the captured image obtained by the image sensor into a plurality of blocks, and calculating an exposure evaluation value for each block;
Has,
The imaging apparatus according to claim 1, wherein the foreign object information is generated in the block unit. In the imaging device according to claim 4 or 5,
In the calibration, an achromatic chart is photographed. In the imaging device according to claim 4 or 5,
The imaging apparatus according to claim 1, wherein the calibration acquires foreign substance information in a plurality of states in the imaging optical system. In the imaging device according to claim 4 or 5,
In the calibration, immediately before or immediately after photographing, the foreign object information is acquired by changing an exposure condition while maintaining a state of the imaging optical system. The imaging device according to any one of claims 1 to 9,
The image processing means,
(A-5) noise processing means capable of changing the degree of noise processing on the image signal;
Further having
The noise processing means,
Noise reduction processing means for performing a noise reduction process by making the degree of noise processing of a foreign image portion in which the influence of the foreign substance appears in the captured image obtained by the imaging element different from that of the captured image excluding the foreign image section,
An imaging device comprising: The imaging device according to claim 10,
The noise reduction processing means,
Means for performing noise reduction processing at a degree corresponding to the degree of foreign matter correction of the foreign matter image portion by the foreign matter correction means,
An imaging device comprising: In the imaging device according to claim 10 or 11,
The noise reduction processing means,
Means for performing noise reduction processing at a degree corresponding to the degree of luminance change in the foreign matter image portion,
An imaging device comprising: The imaging device according to claim 12,
(C) a luminance change detecting means for detecting a degree of the luminance change based on a photographed image acquired by the image sensor at the time of actual photographing;
An imaging device, further comprising:

Images