JP2011066827A - Image processing apparatus, image processing method and program - Google Patents

Abstract

An image in which a region other than a main subject is blurred is appropriately generated.
An image capturing apparatus includes an image acquisition unit that acquires at least two images with different depths of field captured at substantially the same angle of view, and each pixel corresponding to at least two images. A difference generation unit 6a that generates a difference, and a blurred image generation unit that generates a blurred image by performing a blurring process on any one of at least two images based on the generated difference between pixels. 6b.
[Selection] Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  Conventionally, focus bracket shooting is performed at the in-focus position of the main subject and the focal positions before and after that, and the blur amount of each pixel of the reference image is calculated from a plurality of captured images, and the reference image is subjected to blur processing. There is known an imaging apparatus that generates a blur-emphasized image (see, for example, Patent Document 1).

JP 2008-271241 A

By the way, in the case of the above-mentioned patent document 1, a change in the position of the focus lens in focus bracket shooting causes a shift in the balance between the main subject and its background, resulting in coordinates where there are no corresponding pixels among a plurality of captured images. There is a problem such as.
Here, as a shooting condition where the focus lens position change does not affect the in-focus state of the subject, it may be set to increase the depth of field, but focus bracket shooting was performed for images with a deep depth of field. However, the difference in the amount of background blur becomes small, and it becomes difficult to extract differences between a plurality of captured images.

  Accordingly, an object of the present invention is to provide an image processing apparatus, an image processing method, and a program capable of appropriately generating an image in which an area other than the main subject is blurred.

In order to solve the above problem, an image processing apparatus according to claim 1 is provided.
An acquisition unit that acquires at least two images with different depths of field captured at substantially the same angle of view, and a difference that generates a difference between corresponding pixels between the at least two images acquired by the acquisition unit A generation unit that generates a blurred image by performing a blurring process on any one of the at least two images based on the difference between the pixels generated by the difference generation unit; It is characterized by providing these.

The invention according to claim 2 is the image processing apparatus according to claim 1,
The image generating means performs the blurring process on an out-of-focus region of an image having a shallower depth of field among the at least two images.

The invention according to claim 3 is the image processing apparatus according to claim 2,
The image processing unit further includes an adjusting unit that adjusts a blurring degree of the blurring process based on a difference between the pixels generated by the difference generating unit, and the image generating unit is based on the blurring degree adjusted by the adjusting unit. The blurring process is performed.

The invention according to claim 4 is the image processing apparatus according to claim 3,
The apparatus further comprises information acquisition means for acquiring frequency component information of each pixel of the at least two images acquired by the acquisition means, and the difference generation means is between the at least two images acquired by the information acquisition means. A difference between frequency component information of each corresponding pixel is generated.

The invention according to claim 5 is the image processing apparatus according to claim 4,
The adjustment unit includes a setting unit that sets a low-pass filter based on the difference between the frequency component information of the pixels generated by the difference generation unit so that the cutoff frequency becomes smaller as the difference increases. The image generation means performs the blurring process using the low-pass filter set by the setting means.

The invention according to claim 6 is the image processing apparatus according to claim 3,
The acquisition unit acquires three or more images in which the depth of field changes in a predetermined direction, and the difference generation unit acquires the three or more images acquired by the acquisition unit. A difference between the pixels corresponding to each other in the order of the images is generated, and the adjustment unit is configured to generate the difference between the three or more images acquired by the acquisition unit. The blurring degree of the blurring process is adjusted based on a change in pixel difference.

The invention according to claim 7 is the image processing apparatus according to any one of claims 1 to 6,
The at least two images having different depths of field are images picked up with substantially the same brightness and different aperture values.

The invention according to claim 8 is the image processing apparatus according to any one of claims 1 to 7,
The image processing apparatus further comprises alignment means for aligning the at least two images, and the adjustment means generates a difference between corresponding pixels between the at least two images aligned by the alignment means. It is a feature.

The image processing method of the invention according to claim 9
An image processing method using an image processing apparatus, wherein a step of acquiring at least two images with different depths of field captured at substantially the same angle of view corresponds to the acquired at least two images Generating a difference between the pixels, and generating a blurred image by performing a blurring process on any one of the at least two images based on the generated difference between the pixels. , Is executed.

The program of the invention according to claim 10 is:
An acquisition means for acquiring at least two images with different depths of field, which are imaged at substantially the same angle of view, and each pixel corresponding between the at least two images acquired by the acquisition means. A difference generation means for generating a difference between the pixels, and based on the difference between the pixels generated by the difference generation means, blurring processing is performed on any one of the at least two images to obtain a blurred image. It is characterized by functioning as image generation means for generating.

  According to the present invention, it is possible to appropriately generate a blurred image by blurring an area other than the main subject.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. 3 is a flowchart illustrating an example of an operation related to a blurred image generation process performed by the imaging apparatus of FIG. 1. It is a figure which shows typically an example of the image for demonstrating the blurred image production | generation process of FIG. It is a figure which shows typically an example of the image for demonstrating the blurred image production | generation process of FIG. It is a block diagram which shows schematic structure of the imaging device of the modification 1. 6 is a flowchart illustrating an example of an operation related to a blurred image generation process by the imaging apparatus of FIG. 5.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.

The imaging apparatus 100 according to the present embodiment generates a difference between corresponding pixels between at least two images P1 and P2 having different depths of field captured at substantially the same angle of view, and calculates the difference between the pixels. Based on this, the blurred image B is generated by performing the blurring process on any one of the images P1 and P2 (for example, the first image P1).
Specifically, as illustrated in FIG. 1, the imaging apparatus 100 includes an imaging unit 1, an imaging control unit 2, an image data generation unit 3, a memory 4, an alignment unit 5, and an image processing unit 6. A display control unit 7, a display unit 8, a recording medium 9, an operation input unit 10, and a central control unit 11.

  The imaging unit 1 continuously images a subject as an imaging unit. Specifically, the imaging unit 1 includes a lens unit 1a, a diaphragm 1b, and an electronic imaging unit 1c.

The lens unit 1a includes a plurality of lenses and includes a zoom lens, a focus lens, and the like.
Although not shown, the lens unit 1a includes a zoom drive unit that moves the zoom lens in the optical axis direction and a focus drive unit that moves the focus lens in the optical axis direction when imaging a subject. May be.

The diaphragm 1b adjusts the amount of light passing through the lens unit 1a.
In addition, although not shown, the diaphragm 1b may include a diaphragm driving unit that increases or decreases the diameter of the diaphragm 1b in accordance with a predetermined control signal from the exposure control unit 2a.

  The electronic imaging unit 1c is composed of, for example, an image sensor such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), and converts an optical image that has passed through various lenses of the lens unit 1a into a two-dimensional image signal. To do.

  Although not shown, the imaging control unit 2 includes a timing generator, a driver, and the like. Then, the imaging control unit 2 scans and drives the electronic imaging unit 1c with a timing generator and a driver, and converts the optical image into a two-dimensional image signal with the electronic imaging unit 1c every predetermined period, and the electronic imaging unit 1c. The image frame is read out from the imaging area for each screen and is output to the image data generation unit 3.

Further, the imaging control unit 2 includes an exposure control unit 2a that controls execution of AE (automatic exposure processing).
The exposure control unit 2a sequentially sets exposure conditions when the imaging unit 1 and the imaging control unit 2 capture an object. That is, the exposure control unit 2a adjusts the shutter speed (exposure time) and signal amplification factor (ISO sensitivity) of the electronic imaging unit 1c, the aperture value of the aperture 1b, and the like based on a predetermined program diagram. The control signal is output to the electronic imaging unit 1c and the diaphragm 1b. Specifically, the exposure control unit 2a determines the aperture value (AV value) and the exposure time so that the brightness (BV value) of the images captured by the imaging unit 1 and the imaging control unit 2 becomes a predetermined value. (TV value) and ISO sensitivity (SV value) are determined according to the following general formula of APEX value.
Image brightness (BV value) = Aperture value (AV value) + Exposure time (TV value)-ISO sensitivity (SV value)

In addition, when the electronic imaging unit 1c continuously captures the main subject S at a predetermined imaging frame rate in the aperture bracket imaging mode, the exposure control unit 2a has two continuous shot images P1 in which the imaging order is continuous. , P2 is controlled so that the brightness (BV value) is substantially equal and the aperture value (AV value) is different from each other so that the depths of field of P2 are different from each other. Specifically, the exposure control unit 2a sets the aperture value (AV value) to the most open side so that the depth of field of the first image P1 (see FIG. 3A) is the smallest, As the continuous shooting of the second and subsequent images (see FIG. 3B) proceeds, the aperture value (AV value) is gradually reduced so that the depth of field gradually increases. At this time, the exposure control unit 2a determines the TV value (exposure time) according to the change of the aperture value (AV value) so that the brightness (BV value) of the continuously shot images is fixed to a predetermined value. Is changed (decreased) (longer) or the SV value (ISO sensitivity) is increased (increased).
As described above, the imaging control unit 2 drives and controls the electronic imaging unit 1c and the aperture stop 1b by the exposure control unit 2a, and continuously includes at least two images having substantially the same angle of view as the imaging unit 1 and having different depths of field. To capture.

The image data generation unit 3 appropriately adjusts the gain for each of the RGB color components with respect to the analog value signal of the image data transferred from the electronic imaging unit 1c, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to a memory 4 used as a buffer memory via a DMA controller (not shown).

  The memory 4 is composed of, for example, a DRAM or the like, and temporarily stores data processed by the image processing unit 6, the central control unit 11, and the like.

The alignment unit 5 includes an image acquisition unit 5 a that acquires images continuously captured by the imaging unit 1.
The image acquisition unit 5a is a continuous-shot image captured by the imaging unit 1 and generated by the image data generation unit 3 in the aperture bracket imaging mode, that is, two continuous-shot images with different depths of field (for example, one The luminance signal Y of the first image P1, the second image P2, etc.) is acquired from the memory 4.
Although the image acquisition unit 5a acquires two continuous shot images having different depths of field, the image acquisition unit 5a is not limited to this, and may be, for example, three or more.
Here, the image acquisition unit 5a constitutes acquisition means for acquiring at least two images with different depths of field that are successively captured.

Further, the alignment unit 5 includes a feature amount calculation unit 5b that performs a feature extraction process for extracting feature points from the continuous shot image.
For example, the feature amount calculation unit 5b divides the luminance signal Y of the first image P1 into a predetermined plurality of areas (for example, horizontal × vertical: 4 × 3, etc.), and then each area is divided into a predetermined plurality (for example, , Horizontal x vertical: 3 x 3 etc.) (for example, a square of 16 x 16 pixels). And the feature-value calculating part 5b extracts the block (feature point) with a high feature with many high frequency components from several blocks for every area as a template. Here, the feature extraction process is a process of selecting a feature having a high characteristic convenient for tracking from a large number of candidate blocks.
Note that the number of divided areas and blocks, and the number of pixels in each block are examples, and are not limited to these.

In addition, the alignment unit 5 includes a block matching unit 5c that performs block matching processing for alignment of continuous shot images.
For example, the block matching unit 5c performs alignment with the first image P1 of the continuous shot images P1 and P2 as a reference image and the second image P2 as a target image. Specifically, the block matching unit 5c determines where in the second image P2 the template (for example, a square of 16 × 16 pixels) of the first image P1 extracted by the feature extraction processing corresponds. That is, a position (corresponding region) where the template pixel value optimally matches is searched for in the second image P2. Then, an optimal evaluation value between the first image P1 and the second image P2 having the best evaluation value (for example, difference sum of squares (SSD), difference sum of absolute values (SAD), etc.) of pixel values is obtained. The offset is calculated as the motion vector of the template.

Then, the alignment unit 5 aligns the first image P1 and the second image P2 based on the feature points extracted from the first image P1. That is, the alignment unit 5 performs coordinate conversion of each pixel of the second image P2 (target image) with respect to the first image P1 (reference image) based on the feature points extracted from the first image P1. An equation (projection transformation matrix) is calculated, and the second image P2 is coordinate-transformed according to the coordinate transformation equation to align with the first image P1.
Specifically, the alignment unit 5 calculates the motion vectors of a plurality of templates calculated by the block matching unit 5c by majority, and the motion determined to be statistically greater than or equal to a predetermined percentage (for example, 50%). Using the vector as the entire motion vector, the projection transformation matrix of the second image P2 is calculated using the feature point correspondences related to the motion vector. Then, the alignment unit 5 performs coordinate conversion on the YUV data of the second image P2 according to the projective transformation matrix and performs alignment with the YUV data of the first image P1.
Here, the alignment unit 5 constitutes an alignment unit that aligns at least two images P1 and P2 having different depths of field.

  The above-described alignment method is an example and is not limited to this. That is, for example, when calculating the entire motion vector, the reliability of the entire motion vector may be improved by performing processing for determining the validity of the tracking result (corresponding to the feature point) of each template. .

The image processing unit 6 includes a difference generation unit 6a that generates a difference between pixel values of corresponding pixels between continuous shot images.
Specifically, the difference generation unit 6a corresponds between the YUV data of the second image P2 and the YUV data of the first image P1 in which each pixel is coordinate-converted by the alignment unit 5 according to the projective transformation matrix. A blur data map M is generated by calculating a difference between pixel values of the pixels to be processed. That is, since the YUV data of these two images P1 and P2 are images that are continuously shot with the brightness (BV value) being matched and the aperture value (AV value) being changed, the aperture 1b Differences other than the blur amount derived from a change in the diameter of the image (difference in depth of field) are very small values. That is, when the first image P1 is captured while focusing on the main subject S, and the second and subsequent images are set so that the depth of field is deeper, continuous shooting images P1, P2 are taken. The difference of the main subject S portion at is very small. Therefore, except for the main subject S portion, an area composed of pixels in which the difference between the pixel values of the corresponding pixels between the YUV data of the two continuous shot images P1 and P2 is smaller than the predetermined threshold is set to be equal to or higher than the predetermined threshold. A blur data map M (FIG. 4A) in which a pixel area is defined as a blur area (non-focus area N1) in the first image P1, and a difference between pixel values of each pixel in the non-focus area N1 is recorded. ))).
Here, the difference generation unit 6a constitutes difference generation means for generating difference information of pixel values of corresponding pixels between at least two images P1 and P2 having different depths of field.

Further, the image processing unit 6 performs a blurring process on the non-focused region N1 of the first image P1 based on the difference between the pixel values of the respective pixels generated by the difference generation unit 6a, thereby blurring the image B. Is provided with a blurred image generation unit 6b.
Specifically, the blurred image generation unit 6b includes a blur level adjustment unit 6c that adjusts the blur level of the blur processing for the non-focused region N1 of the first image P1. The blurring degree adjustment unit 6c is, for example, input based on a predetermined operation of the operation input unit 10 by the user, or a predetermined coefficient set by default is generated by the difference generation unit 6a. By multiplying the difference between the pixel values of each pixel of M, the degree of blurring (blur amount) for each pixel in the blurring process is adjusted to generate an additional blur data map.
Then, the blurred image generation unit 6b, based on the blurring degree for each pixel of the additional blur data map generated by the blurring degree adjustment unit 6c, the first image P1, that is, the two continuous shot images P1, A blurred image B is generated by performing a blurring process on the non-focused region N1 of the image P1 having a shallower depth of field of P2. Specifically, based on the YUV data of the first image P1, the blurred image generation unit 6b calculates the pixel value of each pixel in the out-of-focus area N1 of the image P1 for the corresponding pixel in the additional blur data map. Correction is made according to the degree of blurring, and YUV data of the blurred image B in which new blur is added to the area N2 (non-focused area N1) other than the main subject S is generated.
In addition, as a method of blurring processing of the region N2 other than the main subject S of the first image P1, for example, an average obtained by adding a predetermined weight to the pixel values of the peripheral pixels of the processing target pixel Is a process of setting the pixel value of the pixel to be processed. At this time, the size of blurring (blurring) of the blurred image B is adjusted by changing the range to be averaged (the number of neighboring points for calculating the average) according to the blurring degree of the processing target pixel.

  As described above, the blurred image generation unit 6b applies to any one of at least two continuous shot images P1 and P2 based on the difference between the pixel values of the pixels generated by the difference generation unit 6a. Thus, an image generating means for generating a blurred image B by performing a blurring process is configured. Moreover, the blurring degree adjusting unit 6c constitutes an adjusting unit that adjusts the blurring degree of the blurring process based on the difference between the pixel values of each pixel generated by the difference generating unit 6a.

The display control unit 7 performs control for reading display image data temporarily stored in the memory 4 and displaying the read image data on the display unit 8.
Specifically, the display control unit 7 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder reads the luminance signal Y and the color difference signals Cb and Cr read from the memory 4 and stored in the VRAM (not shown) under the control of the central control unit 11 via the VRAM controller. Are periodically read out, and a video signal is generated based on these data and output to the display unit 8.

  The display unit 8 is, for example, a liquid crystal display device, and displays an image captured by the electronic imaging unit 1c on the display screen based on a video signal from the display control unit 7. Specifically, the display unit 8 sequentially updates a plurality of image frames generated by imaging the subject by the imaging unit 1 and the imaging control unit 2 at a predetermined display frame rate in the still image capturing mode and the moving image capturing mode. While displaying the live view image. The display unit 8 displays an image (rec view image) recorded as a still image or an image being recorded as a moving image.

  The recording medium 9 is composed of, for example, a non-volatile memory (flash memory) or the like, and recording still image data encoded in a predetermined compression format by an encoding unit (not shown) of the image processing unit 6 or a plurality of recording media 9 The moving image data composed of image frames is stored.

  The operation input unit 10 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 10 includes a shutter button related to an imaging instruction of a subject, a selection determination button related to an instruction to select an imaging mode and a function, a zoom button related to an instruction to adjust the zoom amount, and the like (all illustrated). Abbreviation), a predetermined operation signal is output to the central control unit 11 in accordance with the operation of these buttons.

  The central control unit 11 controls each unit of the imaging device 100. Specifically, the central control unit 11 includes a CPU (not shown) and performs various control operations in accordance with various processing programs (not shown) for the imaging apparatus 100.

Next, the blurred image generation processing by the imaging apparatus 100 will be described with reference to FIGS.
FIG. 2 is a flowchart illustrating an example of an operation related to the blurred image generation process. FIGS. 3A to 4B are diagrams schematically illustrating an example of an image related to a blurred image generation process.

The blurred image generation process is executed when an instruction to select an aperture bracket imaging mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button of the operation input unit 10 by the user. It is processing.
As shown in FIG. 2, first, the display control unit 7 displays a live view image on the display screen of the display unit 8 based on a plurality of image frames generated by imaging the subject by the imaging unit 1 (step S1). .

Next, the exposure control unit 2a of the imaging control unit 2 sets the aperture value (AV value) to the most open side so that the depth of field becomes the shallowest, and also sets the exposure time (TV value) and ISO sensitivity (SV). Value) is determined in accordance with the general formula of the APEX value, and then a predetermined control signal is output to the electronic imaging unit 1c and the aperture 1b (step S2). Thereby, the aperture stop 1b is in the most open state.
Subsequently, the central control unit 11 determines whether or not a continuous shooting instruction is input based on a predetermined operation of the shutter button of the operation input unit 10 by the user (step S3). Here, if it is determined that a continuous shooting instruction has been input (step S3; YES), the imaging control unit 2 focuses the lens unit 1a on the main subject S, and the first image P1 (FIG. 3 (FIG. 3)). The optical image of (a) is captured by the electronic imaging unit 1c, and the image data generation unit 3 generates YUV data of the first image P1 transferred from the electronic imaging unit 1c (step S4). The YUV data of the first image P1 is temporarily stored in the memory 4.

Next, the exposure control unit 2a of the imaging control unit 2 does not change the brightness of the first image P1, and the aperture value so that the depth of field is deeper than that of the first image P1. (AV value) is reduced by a predetermined amount (for example, one step), exposure time (TV value) and ISO sensitivity (SV value) are determined according to the general formula of APEX value, and then a predetermined control signal is sent to electronic imaging unit 1c and Output to the diaphragm 1b (step S5). As a result, the diaphragm 1b is in a state of being squeezed by a predetermined amount.
Subsequently, the imaging control unit 2 uses the electronic imaging unit 1c for an optical image of the second image P2 (see FIG. 3B) having a depth of field deeper than the first image P1 under a predetermined imaging condition. The image data generation unit 3 generates YUV data of the second image P2 transferred from the electronic imaging unit 1c (step S6). As a result, the first image P1 and the second image P2 are continuously captured at substantially the same angle of view. The YUV data of the second image P2 is temporarily stored in the memory 4.

Next, the alignment unit 5 predetermines a projective transformation matrix for projectively transforming the YUV data of the second image P2 with reference to the YUV data of the first image P1 temporarily stored in the memory 4. The image conversion model (for example, a similarity conversion model or a joint conversion model) is used (step S7).
Specifically, the image acquisition unit 5a of the alignment unit 5 acquires the luminance signal Y of the first image P1 from the memory 4 out of the two continuous shot images P1 and P2. Next, the feature amount calculation unit 5b of the alignment unit 5 selects a predetermined number (or a predetermined number or more) of high feature blocks (feature points) based on the luminance signal of the first image P1. The contents of the block are extracted as a template. Then, the block matching unit 5c of the alignment unit 5 searches the second image P2 for a position where the pixel value of the template extracted by the feature extraction process optimally matches, and the degree of difference between the pixel values The optimal offset between the first image P1 and the second image P2 having the best evaluation value is calculated as a motion vector of the template. Then, the alignment unit 5 statistically calculates the entire motion vector based on the motion vectors of the plurality of templates calculated by the block matching unit 5c, and uses the feature point correspondence related to the motion vector to obtain the second image. The projection transformation matrix of the image P2 is calculated.

  Next, the alignment unit 5 performs projective transformation on the second image P2 based on the calculated projective transformation example, so that the YUV data of the first image P1 and the YUV of the second image P2 are converted. A process of aligning the data is performed (step S8).

  Then, the difference generation unit 6a of the image processing unit 6 is a pixel of each pixel corresponding between the YUV data of the second image P2 and the YUV data of the first image P1 whose coordinates are converted by the positioning unit 5. A value difference is calculated (step S9). Subsequently, the difference generation unit 6a specifies a pixel in which the difference between the pixel values of each pixel is equal to or greater than a predetermined threshold, and sets an area including pixels in which the difference is equal to or greater than the predetermined threshold in the first image P1. As a blur area (non-focus area N1), a blur data map M (see FIG. 4A) in which a difference between pixel values of each pixel in the non-focus area N1 is recorded is generated (step S10).

Next, the blurred image generation unit 6b of the image processing unit 6 multiplies each pixel difference in the blur data map M by a predetermined coefficient, thereby each region N2 other than the main subject S in the first image P1. An additional blur data map is generated by adjusting the blurring degree (blur amount) for each pixel (step S11).
Subsequently, the blurred image generation unit 6b performs a blurring process on the region N2 other than the main subject S of the first image P1 according to the degree of blurring for each pixel of the additional blur data map, thereby blurring the image. B YUV data is generated (step S12).

Thereafter, the central control unit 11 stores the YUV data of the blurred image B generated by the blurred image generating unit 6b in a predetermined storage area of the recording medium 9 (step S13).
Thus, the blurred image generation process is completed.

As described above, according to the imaging apparatus 100 of the present embodiment, each pixel corresponding between two continuous shot images P1 and P2 having different depths of field that are continuously captured at substantially the same angle of view. A difference between pixel values is generated, and a blurring process is performed on the first image P1 having a shallower depth of field based on the difference between the pixel values of the respective pixels. Even if the image sensor mounted on 1c is small, the aperture 1b cannot be opened largely because the amount of light in the peripheral portion of the image is reduced, or an imaging apparatus having no zoom lens is installed, it is not the main subject S. The image B in which the area N2 of the first image P1 is blurred more greatly than the blur of the non-focused area N1 of the first image P1 can be appropriately generated.
That is, in the aperture bracket imaging, two continuous shot images P1 and P2 having different depths of field are obtained by capturing images with substantially the same brightness and different aperture values. Based on the difference between the pixel values of the corresponding pixels between the continuous shot images P1 and P2, the out-of-focus region mainly including other than the main subject S in the first image P1 having the shallower depth of field. By specifying N1 and performing blurring processing on the out-of-focus area N1, it is possible to appropriately generate the image B in which the area N2 other than the main subject S is blurred.
Further, since the two continuous shot images P1 and P2 are different only in the aperture value, there is a deviation in the balance between the main subject S and its background as in the case of generating a blurred image using conventional focus bracket photography. Thus, there is no problem that coordinates in which corresponding pixels do not exist between a plurality of captured images are generated.

  Further, the type of main subject S is adjusted by adjusting the blurring degree of the blurring process for the first image P1 based on the difference in pixel value of each corresponding pixel between the two continuous shot images P1 and P2. A predetermined amount of blur can be given to the area N2 other than the main subject S according to the user's preference and the like, and a more interesting blurred image B can be generated.

  Further, the two continuous shot images P1 and P2 are aligned, and a difference between the pixel values of corresponding pixels between the continuous shot images P1 and P2 is generated. It is possible to appropriately generate the difference between the pixel values of the corresponding pixels between the two continuous shot images P1 and P2, without generating coordinates where the corresponding pixels do not exist between the images.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
Below, the modification of the imaging device 100 is demonstrated.

<Modification 1>
The imaging apparatus 100 according to the first modification uses a low-pass filter based on the difference between the frequency characteristics of the corresponding pixels between the two continuous shot images P1 and P2 so that the cutoff frequency becomes smaller as the difference becomes larger. Set and perform blurring using the low-pass filter.
Note that the imaging apparatus 100 according to the first modification is substantially the same except for the blurring process using the low-pass filter, and the detailed description thereof is omitted.

FIG. 5 is a block diagram illustrating a schematic configuration of the imaging apparatus 100 according to the first modification.
As shown in FIG. 5, the image processing unit 6 includes a frequency characteristic acquisition unit 6d that acquires the frequency characteristics of each pixel of the two continuous shot images P1 and P2.
The frequency characteristic acquisition unit 6d calculates and acquires the frequency characteristic (frequency component information) of each pixel for each of at least two continuous shot images P1 and P2 as information acquisition means.
Then, the difference generation unit 6a of the image processing unit 6 calculates the difference between the frequency characteristics of the corresponding pixels between the two continuous shot images P1 and P2 acquired by the frequency characteristic acquisition unit 6d. Then, the difference generation unit 6a excludes, as the main subject S portion, an area composed of pixels in which the frequency characteristic difference between the pixels corresponding to the YUV data of the two continuous shot images P1 and P2 is smaller than a predetermined threshold value. A blur data map that records a difference in frequency characteristics of each pixel in the non-focused region N1 with a region composed of pixels that are equal to or greater than a predetermined threshold as a blur region (non-focused region N1) in the first image P1. M is generated.

The blur adjustment unit 6c of the blurred image generation unit 6b sets a low-pass filter having a predetermined cutoff frequency for each pixel based on the difference in frequency characteristics of each pixel of the blur data map M generated by the difference generation unit 6a. To adjust the blurring degree of the blurring process. Specifically, the blurring degree adjustment unit 6c sets a low-pass filter for each pixel so that the cutoff frequency becomes smaller as the difference between the frequency characteristics of the processing target pixels becomes larger.
Then, the blurred image generation unit 6b uses the predetermined low-pass filter set by the blurring degree adjustment unit 6c to each pixel of the non-focused region N1 (region N2 other than the main subject S) of the first image P1. Is subjected to a filtering process to perform a blurring process for removing high frequency components higher than the cutoff frequency.
Here, based on the difference between the frequency characteristics of the pixels of the two continuous shot images P1 and P2 generated by the difference generation unit 6a, the blurring degree adjustment unit 6c has a lower cutoff frequency as the difference increases. Thus, the setting means for setting the low-pass filter is configured.

Next, a blurred image generation process performed by the imaging apparatus 100 according to the first modification will be described with reference to FIG.
FIG. 6 is a flowchart illustrating an example of an operation related to the blurred image generation process.
The blurred image generation processing described below is substantially the same as the blurred image generation processing of the above embodiment except for the blur processing using a low-pass filter, and detailed description thereof is omitted.

As shown in FIG. 6, first, the display control unit 7 displays a live view image on the display screen of the display unit 8 in substantially the same manner as in the above embodiment (step S1).
Next, in substantially the same manner as in the above embodiment, the exposure control unit 2a sets the aperture value (AV value) to the most open side so that the depth of field is the shallowest, as well as the exposure time (TV value) and ISO. Sensitivity (SV value) is determined according to the general formula of APEX value (step S2).

  Subsequently, in substantially the same manner as in the above embodiment, the central control unit 11 determines whether or not a continuous shooting imaging instruction has been input (step S3). If it is determined that a continuous shooting instruction has been input (step S3; YES), the imaging control unit 2 uses the optical image of the first image P1 (see FIG. 3A) as a predetermined imaging condition. The image is picked up by the electronic image pickup unit 1c (step S4).

Next, in substantially the same manner as in the above embodiment, the exposure control unit 2a of the imaging control unit 2 does not change the brightness of the first image P1, and the field of view is greater than that of the first image P1. The aperture value (AV value) is reduced by a predetermined amount (for example, one step) so that the depth becomes deeper, and the exposure time (TV value) and ISO sensitivity (SV value) are determined according to the general formula of the APEX value (step S5). .
Subsequently, in substantially the same manner as in the above-described embodiment, the imaging control unit 2 predetermines the optical image of the second image P2 (see FIG. 3B) having a deeper depth of field than the first image P1. The image is picked up by the electronic image pickup unit 1c under the image pickup conditions (step S6).

  Next, as in the above embodiment, the alignment unit 5 performs projective conversion on the YUV data of the second image P2 with reference to the YUV data of the first image P1 temporarily stored in the memory 4. A projective transformation matrix is calculated using a predetermined image transformation model (step S7). Thereafter, in substantially the same manner as in the above-described embodiment, the alignment unit 5 performs projective transformation on the second image P2 based on the calculated projective transformation example, so that the YUV data of the first image P1 and the second image P2 are transformed. A process of aligning the YUV data of the first image P2 is performed (step S8).

Next, the frequency characteristic acquisition unit 6d of the image processing unit 6 calculates and acquires the frequency characteristic of each pixel for each of the two continuous shot images P1 and P2 (step S21).
After that, the difference generation unit 6a of the image processing unit 6 performs the frequency of each pixel corresponding between the YUV data of the second image P2 and the YUV data of the first image P1 whose coordinates are converted by the alignment unit 5. A difference between the characteristics is calculated (step S22). Subsequently, the difference generation unit 6a identifies pixels in which the difference in frequency characteristics of each pixel is equal to or greater than a predetermined threshold, and sets an area composed of pixels in which the difference is equal to or greater than the predetermined threshold in the first image P1. As a blur area (non-focus area N1), a blur data map M (see FIG. 4A) in which a difference in frequency characteristics of each pixel in the non-focus area N1 is recorded is generated (step S23).

Next, the blurring degree adjustment unit 6c of the blurred image generation unit 6b performs blurring processing by setting a low-pass filter having a predetermined cutoff frequency for each pixel according to the difference in frequency characteristics of each pixel in the blur data map M. The degree of blur is adjusted (step S24).
Then, the blurred image generation unit 6b uses the predetermined low-pass filter set by the blurring degree adjustment unit 6c to each pixel of the non-focused region N1 (region N2 other than the main subject S) of the first image P1. By performing a filtering process on the image, a blurring process for removing high frequency components higher than the cutoff frequency is performed to generate YUV data of the blurred image B (step S25).

Thereafter, in substantially the same manner as in the above embodiment, the central control unit 11 stores the YUV data of the blurred image B generated by the blurred image generation unit 6b in a predetermined storage area of the recording medium 9 (step S13).
Thus, the blurred image generation process is completed.

  Therefore, according to the imaging apparatus 100 of the first modification, based on the difference in the frequency characteristics of the corresponding pixels between the two continuous shot images P1 and P2, the cutoff frequency becomes smaller as the difference becomes larger. By setting the low-pass filter in this manner and performing blurring processing on the out-of-focus area N1 of the first image P1, blurring processing considering the frequency characteristics of each pixel of the continuous shot images P1 and P2 is performed. Therefore, it is possible to appropriately generate the image B in which the region N2 other than the main subject S is blurred.

In the above embodiment, the first image P1 is exemplified as the target image for the blurring process. However, the present invention is not limited to this, and the second image P2, that is, the two images P1 is used. , P2 may be an image having a deeper depth of field. That is, based on the difference between the pixel values of the pixels of the continuous shot images P1 and P2, the unfocused region N1 mainly including the main subject S in the second image P2 may be identified and blurred. .
For example, the second image P2 and the blur data map M are saved, and the non-focused region N1 mainly including the main subject S in the second image P2 is specified and subjected to the blurring process. From a state with a depth of field as deep as that of the first image P2, to a state with a depth of field as shallow as that of the first image P1, and further to a blurred state beyond that, an arbitrary degree Background blur can be generated after the imaging is completed.

In addition, although the number of images continuously captured in aperture bracket imaging is two, the number is not limited to this, and may be three or more, for example.
At this time, the difference generation unit 6a generates a difference between pixel values of corresponding pixels between images having consecutive sequential shooting sequences among three or more continuous shooting images having different depths of field. Thus, the blurring degree adjustment unit 6c may adjust the blurring degree of the blurring process based on the change in the difference in pixel value between corresponding pixels for three or more continuous shot images.
That is, since the depth of field gradually increases as the number of continuous shots increases in the continuous shot image, the area where the blur is small in the first image P1 with the shallowest depth of field is the field of view. You can focus on the second and third images by increasing the depth slightly. On the other hand, an area with large blur in the first image P1 cannot be focused unless the number of continuous shots is increased to increase the depth of field.
And even if it is a region (out-of-focus region N1) that is blurred in the first image P1, once it is in focus, the depth of field becomes deeper as the number of continuous shots increases, Since the focus will not be lost in the future, the change in the difference in the pixel value of the corresponding pixel between the images in which the continuous shooting order is continuous becomes small. That is, the number of continuous shots required until the change in the difference in pixel value becomes smaller is smaller in the area where blur is smaller in the first image P1, and is larger in the area where blur is larger in the first image P1. Become more.

Therefore, the blur size in the non-focused area N1 of the first image P1 can be specified based on the change in the pixel value difference between the pixels in accordance with the increase in the number of continuous shots. By adjusting the degree of blurring in the blurring process according to the size of the image, the out-of-focus area N1 (area N2 other than the main subject S) is considered in consideration of the blur of the out-of-focus area N1 in the first image P1. Can be appropriately generated.
Note that the adjustment of the blurring degree of the blurring process is not performed based on the change in the pixel value difference of each pixel corresponding between three or more continuous shot images, but instead of each corresponding pixel between the continuous shot images. You may perform based on the change of a pixel value.

  In the above-described embodiment, the exposure control unit 2a opens the aperture value (AV value) the most so that the depth of field of the first image P1 (see FIG. 3A) is the smallest. The aperture value (AV value) is gradually reduced so that the depth of field gradually increases as the continuous shooting of the second and subsequent images (see FIG. 3B) progresses. However, it is not limited to this. For example, the exposure control unit 2a sets the aperture value (AV value) so that the depth of field of the first image P1 is the deepest, and the field of view is increased as continuous shooting of the second and subsequent images proceeds. The aperture value (AV value) may be gradually opened so that the depth becomes gradually shallower.

Further, in capturing the two continuous shot images P1 and P2, the brightness of the images is made substantially equal and the aperture values are made different from each other, but the depths of field of these images P1 and P2 are made different. Therefore, at least the aperture value may be changed, and the brightness does not necessarily have to be substantially equal.
That is, if the aperture value is changed in the direction of aperture reduction and the brightness of the continuous shot images P1 and P2 is fixed to a predetermined value, it is necessary to lengthen the exposure time or increase the ISO sensitivity. If the time is increased, camera shake tends to occur, and if the ISO sensitivity is increased, the amount of noise may increase. Therefore, for example, in a sufficiently bright environment as in the case of imaging in fine weather, the exposure time and ISO sensitivity may not be changed even if the aperture value is changed in the direction in which the aperture value is reduced.

  Furthermore, in the above embodiment, the distance from the imaging device 100 to the main subject S may be measured, and the blurring degree may be adjusted according to the distance. That is, in a state where the main subject S is in focus, the distance from the position in the optical axis direction of the focus lens to the main subject S is calculated using a predetermined conversion means (for example, a conversion program or a conversion table), and the subject distance Get information. Then, the region N2 other than the main subject S is more naturally blurred by adjusting the degree of blurring of the region N2 other than the main subject S (the out-of-focus region N1) to be larger as the subject distance is shorter. The blurred image B can be generated appropriately.

  In the above embodiment, the two continuous shot images P1 and P2 are aligned. However, whether or not the images P1 and P2 are aligned is arbitrarily changed as appropriate. be able to. That is, for example, when there is almost no blurring in the two continuous shot images P1 and P2 as in the case where the imaging apparatus 100 is imaged in a fixed position using a tripod or the like, these continuous shot images are not necessarily provided. There is no need to align P1 and P2.

  Furthermore, in the above-described embodiment, the two continuous shot images P1 and P2 are targeted for processing. However, the present invention is not limited to this, and images with different depths of field captured at least at substantially the same angle of view. If so, the first image P1 and the third and subsequent images may be processed.

In addition, the configuration of the imaging apparatus 100 is merely an example illustrated in the above embodiment, and is not limited thereto.
Further, the continuous shot images P1 and P2 are picked up by an image pickup device different from the image pickup device 100, the image data of the processing target image transferred from the image pickup device is acquired, and the blurred image B is generated. An image processing apparatus (imaging apparatus 100) having a configuration may be used.

In addition, in the above-described embodiment, functions as an acquisition unit, a difference generation unit, and an image generation unit include an image acquisition unit 5a of the alignment unit 5, a difference generation unit 6a of the image processing unit 6, and a blurred image generation unit. The configuration realized by driving 6b is not limited to this, but may be realized by executing a predetermined program or the like by the CPU of the central control unit 11.
That is, a program including an acquisition processing routine, a difference generation processing routine, and an image generation processing routine is stored in a program memory (not shown) that stores the program. Then, the CPU of the central control unit 11 may function as an acquisition unit that acquires at least two images P1 and P2 with different depths of field, which are imaged at substantially the same angle of view, by an acquisition process routine. Further, the CPU of the central control unit 11 may function as a difference generation unit that generates a difference between corresponding pixels between at least two images P1 and P2 acquired by the acquisition unit by the difference generation processing routine. good. In addition, the CPU of the central control unit 11 performs the blurring process on any one of the at least two images P1 and P2 based on the difference of each pixel generated by the difference generation unit by the image generation processing routine. May be made to function as image generation means for generating the blurred image B.

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Imaging part 2 Imaging control part 5 Positioning part 5a Image acquisition part 6 Image processing part 6a Difference generation part 6b Blur image generation part 6c Blur degree adjustment part 6d Frequency characteristic acquisition part 11 Central control part

Claims (10)

Obtaining means for obtaining at least two images with different depths of field, which are imaged at substantially the same angle of view;
A difference generating unit that generates a difference between corresponding pixels between the at least two images acquired by the acquiring unit;
Based on the difference between the pixels generated by the difference generation means, an image generation means for performing a blurring process on any one of the at least two images to generate a blurred image;
An image processing apparatus comprising: The image generating means includes
The image processing apparatus according to claim 1, wherein the blurring process is performed on an out-of-focus region of an image having a shallower depth of field among the at least two images. An adjustment unit for adjusting a blurring degree of the blurring process based on the difference between the pixels generated by the difference generation unit;
The image generating means includes
The image processing apparatus according to claim 2, wherein the blurring process is performed based on the blurring degree adjusted by the adjusting unit. Further comprising information acquisition means for acquiring frequency component information of each pixel of the at least two images acquired by the acquisition means;
The difference generation means includes
The image processing apparatus according to claim 3, wherein a difference in frequency component information of each corresponding pixel between the at least two images acquired by the information acquisition unit is generated. The adjusting means includes
Based on the difference between the frequency component information of each pixel generated by the difference generation means, the setting means for setting the low-pass filter so that the cutoff frequency becomes smaller as the difference becomes larger,
The image generating means includes
The image processing apparatus according to claim 4, wherein the blurring process is performed using the low-pass filter set by the setting unit. The acquisition means acquires three or more images in which the depth of field changes in a predetermined direction,
The difference generation unit generates a difference between pixels corresponding to each other between images in which the acquisition order is continuous among the three or more images acquired by the acquisition unit,
The adjusting means includes
The blurring degree of the blurring process is adjusted based on a change in the difference between the pixels generated by the difference generating unit in the three or more images acquired by the acquiring unit. The image processing apparatus according to 3.   The at least two images having different depths of field are images picked up with substantially the same brightness and different aperture values. The image processing apparatus described. An alignment means for aligning the at least two images;
The difference generation means includes
The image processing apparatus according to claim 1, wherein a difference of each corresponding pixel is generated between at least two images aligned by the alignment unit. An image processing method using an image processing apparatus,
Obtaining at least two images with different depths of field captured at substantially equal angles of view;
Generating corresponding pixel differences between the acquired at least two images;
Generating a blurred image by performing a blurring process on any one of the at least two images based on the generated difference between the pixels;
An image processing method characterized in that The computer of the image processing device
Acquisition means for acquiring at least two images with different depths of field, which are imaged at substantially the same angle of view;
Difference generation means for generating a difference between corresponding pixels between the at least two images acquired by the acquisition means;
Based on the difference between the pixels generated by the difference generation unit, an image generation unit that generates a blurred image by performing a blurring process on any one of the at least two images;
A program characterized by functioning as