JP4753856B2 - Motion detection apparatus and method, and imaging apparatus - Google Patents

Description

  The present invention relates to a motion detection device and a motion detection method for detecting motion on an image, and an imaging device using the motion detection method. The present invention particularly relates to a technique for performing motion detection from image data without using a camera shake detection sensor.

  There has already been proposed a technique for generating camera shake information representing a camera shake state during still image shooting using a camera shake detection sensor such as a gyro sensor and generating a restored image without camera shake using an image restoration filter based on the camera shake information. (See Patent Document 1 below). Such camera shake correction is called restoration-type camera shake correction, but it requires a camera shake detection sensor to detect camera shake, and thus increases the cost of the imaging apparatus.

  In order to detect camera shake without using a camera shake detection sensor, it is also conceivable to capture first and second reference images before and after the detection target image and detect camera shake from motion information between the images.

  This detection method will be described. FIG. 14 shows a shooting scene of an imaging apparatus using a CCD (Charge Coupled Devices) image sensor. In FIG. 14, reference numeral 300 represents a locus of camera shake applied to the imaging apparatus in the vicinity of the imaging timing of the detection target image. Due to this camera shake, the subject that falls within the shooting area changes every moment.

  In FIG. 14, a rectangular area denoted by reference numeral 301 represents a shooting area at the timing when exposure of the first reference image is started, and a rectangular area denoted by reference numeral 302 starts exposure of the detection target image. The rectangular area denoted by the reference numeral 303 represents the photographing area at the timing, and the rectangular area denoted by the reference numeral 304 represents the photographing area at the timing when the exposure of the second reference image is started. The imaging region at the timing when the exposure of the frame image is started.

  A curve 310 in FIG. 15 represents a temporal change in the vertical position of the imaging region. When a CCD is used, the exposure period of each pixel in the same image is the same. Therefore, when motion detection is performed based on image comparison between adjacent frames, an average motion between adjacent frames is detected. become. As shown in FIG. 16, the detected motion is detected as motion at the same time, and motion at two sampling points is detected from three consecutive images (that is, motion at two sampling times). Will be detected).

  In the example of the curve 310, the vertical position difference between the point 311 and the point 312 is detected as a vertical movement between the first reference image and the detection target image (zero in FIG. 15), and the point 312 and the point 313 are Is detected as a vertical movement between the detection target image and the second reference image. The camera shake trajectory during the exposure period of the detection target image includes a motion vector between the first reference image and the detection target image corresponding to the vector from the point 311 to the point 312 and a detection target corresponding to the vector from the point 312 to the point 313. It is estimated based on the motion vector between the image and the second reference image.

  As a result, the true camera shake trajectory during the exposure period of the detection target image is represented by reference numeral 321 in FIG. 17, whereas the actually estimated camera shake trajectory is represented by reference numeral 322. . This is due to the fact that there is only two pieces of motion information for estimating the hand movement trajectory while the hand movement is complex.

JP 2006-129236 A JP 2006-243974 A

  If the camera shake period is sufficiently longer than the frame period, or if camera shake of constant speed (or constant acceleration) is applied in the same direction, there is little problem even if the above method is used, but the camera shake period is the frame period. If the movement of the camera shake is complicated, the camera shake trajectory during the exposure period cannot be estimated accurately as described above. In order to estimate the camera shake trajectory with high accuracy, it is necessary to detect motion with high accuracy for the detection target image.

  Accordingly, an object of the present invention is to provide a motion detection device and a motion detection method that can accurately detect motion without using a gyro sensor or the like. Another object of the present invention is to provide an imaging device using the motion detection device.

  In order to achieve the above object, a motion detection apparatus according to the present invention performs motion detection on a detection target image obtained by shooting based on an output of an image pickup device that performs shooting while varying exposure timing between different horizontal lines. A motion detection apparatus, wherein each image obtained by photographing before and after the detection target image is used as a reference image, and the detection target image and each reference image are divided into regions in the vertical direction, and the detection target image and each reference image A region motion detecting means for detecting a motion of an image in each divided region as a region motion is provided, and motion detection for the detection target image is performed based on each region motion.

  As a result, the number of motion detection samplings can be increased as compared with the prior art, and as a result, highly accurate motion detection can be performed.

  Specifically, for example, the motion detection device performs motion detection on the detection target image by regarding each region motion as movement at different times with respect to the entire detection target image.

  In addition, for example, the motion detection apparatus may further include a correction unit that corrects each region motion by performing high-frequency emphasis processing in the time direction on data representing each region motion arranged in time series.

  Thereby, further improvement in accuracy of motion detection can be expected.

  Further, for example, the characteristics of the high frequency emphasis processing are changed according to the number of area divisions in the vertical direction with respect to the detection target image and each reference image.

  Further, for example, each region motion arranged in time series represents a blur locus included in the detection target image.

  In order to achieve the above object, the imaging apparatus according to the present invention includes a blur trajectory estimation unit that estimates a blur trajectory included in the detection target image based on a motion detection result by the motion detection device, and the estimated Restoring means for generating a restored image with reduced blurring in the detection target image based on a blur trajectory.

  In addition, in order to achieve the above object, the motion detection method according to the present invention is based on the output of an image sensor that performs imaging while varying the exposure timing between different horizontal lines, and motion detection for a detection target image obtained by the imaging. A motion detection method for performing detection, wherein each image obtained by photographing before and after the detection target image is used as a reference image, the detection target image and each reference image are divided into regions in the vertical direction, and the detection target image and each It is characterized in that a motion of an image in each divided region between reference images is detected as a region motion, and motion detection is performed on the detection target image based on each region motion.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the motion detection apparatus and motion detection method which can perform a motion detection accurately, without using a gyro sensor etc.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle.

  FIG. 1 is an overall block diagram of an imaging apparatus 1 according to an embodiment of the present invention. The imaging device 1 is a digital video camera capable of shooting moving images and still images. However, the imaging apparatus 1 may be a digital still camera that can capture only a still image.

  The imaging apparatus 1 includes an imaging unit 11, an AFE (Analog Front End) 12, a video signal processing unit 13, a microphone 14, an audio signal processing unit 15, a compression processing unit 16, and an SDRAM as an example of an internal memory. (Synchronous Dynamic Random Access Memory) 17, memory card (storage means) 18, decompression processing unit 19, video output circuit 20, audio output circuit 21, TG (timing generator) 22, CPU (Central Processing Unit) ) 23, a bus 24, a bus 25, an operation unit 26, a display unit 27, and a speaker 28. The operation unit 26 includes a recording button 26a, a shutter button 26b, an operation key 26c, and the like. Each part in the imaging apparatus 1 exchanges signals (data) between the parts via the bus 24 or 25.

  First, basic functions of the imaging device 1 and each part constituting the imaging device 1 will be described.

  The TG 22 generates a timing control signal for controlling the timing of each operation in the entire imaging apparatus 1, and provides the generated timing control signal to each unit in the imaging apparatus 1. Specifically, the timing control signal is given to the imaging unit 11, the video signal processing unit 13, the audio signal processing unit 15, the compression processing unit 16, the expansion processing unit 19, and the CPU 23. The timing control signal includes a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync.

  The CPU 23 comprehensively controls the operation of each unit in the imaging apparatus 1. The operation unit 26 receives an operation by a user. The operation content given to the operation unit 26 is transmitted to the CPU 23. The SDRAM 17 functions as a frame memory. Each unit in the imaging apparatus 1 temporarily records various data (digital signals) in the SDRAM 17 during signal processing as necessary.

  The memory card 18 is an external recording medium, for example, an SD (Secure Digital) memory card. In this embodiment, the memory card 18 is illustrated as an external recording medium. However, the external recording medium is composed of one or a plurality of randomly accessible recording media (semiconductor memory, memory card, optical disk, magnetic disk, etc.). can do.

  FIG. 2 is an internal configuration diagram of the imaging unit 11 of FIG. By using a color filter or the like for the imaging unit 11, the imaging device 1 is configured to generate a color image by shooting.

  The imaging unit 11 includes an optical system 35, a diaphragm 32, an imaging element 33, and a driver 34. The optical system 35 includes a plurality of lenses including the zoom lens 30 and the focus lens 31. The zoom lens 30 and the focus lens 31 are movable in the optical axis direction. The driver 34 controls the movement of the zoom lens 30 and the focus lens 31 based on the control signal from the CPU 23, and controls the zoom magnification and focal length of the optical system 35. Further, the driver 34 controls the opening degree (size of the opening) of the diaphragm 32 based on a control signal from the CPU 23.

  Incident light from the subject enters the image sensor 33 through the lenses and the diaphragm 32 constituting the optical system 35. Each lens constituting the optical system 35 forms an optical image of the subject on the image sensor 33. The TG 22 generates a drive pulse for driving the image sensor 33 in synchronization with the timing control signal, and applies the drive pulse to the image sensor 33.

  The image sensor 33 is, for example, an XY address scanning type CMOS (Complementary Metal Oxide Semiconductor) image sensor. A CMOS image sensor is formed by forming a plurality of pixels arranged two-dimensionally in a matrix, a vertical scanning circuit, a horizontal scanning circuit, a pixel signal output circuit, and the like on a semiconductor substrate capable of forming a CMOS. In the imaging device 33, an imaging surface is formed by a plurality of pixels arranged two-dimensionally, and the imaging surface has a plurality of horizontal lines and a plurality of vertical lines.

  The image sensor 33 has an electronic shutter function, and each pixel is exposed by a so-called rolling shutter. In the rolling shutter, the timing (time) at which each pixel on the imaging surface is exposed differs in the vertical direction for each horizontal line. That is, the exposure timing differs between different horizontal lines on the imaging surface.

  The image sensor 33 photoelectrically converts an optical image incident through the optical system 35 and the diaphragm 32, and sequentially outputs an electrical signal obtained by the photoelectric conversion to the AFE 12 at the subsequent stage. More specifically, in each shooting, each pixel on the imaging surface stores a signal charge having a charge amount corresponding to the exposure time. Each pixel sequentially outputs an electrical signal having a magnitude proportional to the amount of stored signal charge to the subsequent AFE 12. When the optical images incident on the optical system 35 are the same and the aperture amount of the diaphragm 32 is the same, the magnitude (intensity) of the electrical signal from the image sensor 33 (each pixel) increases in proportion to the exposure time. To do.

  The rolling shutter will be described with reference to FIG. As shown in FIG. 3A, the image pickup surface of the image pickup element 33 is formed of first to nth horizontal lines (n is an integer of 2 or more), and the first, second,. ) And the n-th horizontal line in the vertical direction.

FIG. 3B shows the exposure timing of each horizontal line when a certain image is taken. T _Si represents the exposure start timing of each pixel belonging to the i-th horizontal line, and T _Ei represents the exposure end timing of each pixel belonging to the i-th horizontal line (where i is 1 to n) Each integer between). The length of the exposure period of each pixel is the same. That is, the time length between the timings T _{S1 and} T _E1, the time length between the timings T _{S2 and} T _E2 ,..., And the time length between the timings T _{Sn and} T _En are the same. However, with respect to the exposure start timing (and exposure end timing) of the i-th horizontal line, that of the (i + 1) -th horizontal line is delayed by a predetermined time (where i is a value between 1 and (n-1)). Take an integer).

  Returning to the description of each part in FIG. The AFE 12 amplifies the analog signal output from the imaging unit 11 (image sensor 33), and converts the amplified analog signal into a digital signal. The AFE 12 sequentially outputs the digital signals to the video signal processing unit 13.

  The video signal processing unit 13 generates a video signal representing an image (hereinafter also referred to as “captured image” or “frame image”) captured by the imaging unit 11 based on the output signal of the AFE 12. The video signal is composed of a luminance signal Y representing the luminance of the photographed image and color difference signals U and V representing the color of the photographed image. The video signal generated by the video signal processing unit 13 is sent to the compression processing unit 16 and the video output circuit 20.

  The video signal processing unit 13 detects an AF evaluation value according to the contrast amount in the focus detection area in the captured image and an AE evaluation value according to the brightness of the captured image, and transmits them to the CPU 23. The CPU 23 adjusts the position of the focus lens 31 via the driver 34 in FIG. 2 according to the AF evaluation value, thereby forming an optical image of the subject on the image sensor 33. Further, the CPU 23 adjusts the opening degree of the diaphragm 32 (and the amplification degree of signal amplification in the AFE 12 as necessary) via the driver 34 of FIG. 2 according to the AE evaluation value, so that the received light amount (brightness of the image). Control).

  The microphone 14 converts the sound (sound) given from the outside into an analog electric signal and outputs it. The audio signal processing unit 15 converts an electrical signal (audio analog signal) output from the microphone 14 into a digital signal. The digital signal obtained by this conversion is sent to the compression processing unit 16 as an audio signal representing the audio input to the microphone 14.

  The compression processing unit 16 compresses the video signal from the video signal processing unit 13 using a predetermined compression method. At the time of moving image or still image shooting, the compressed video signal is sent to the memory card 18 and recorded on the memory card 18. The compression processing unit 16 compresses the audio signal from the audio signal processing unit 15 using a predetermined compression method. At the time of moving image shooting, the video signal from the video signal processing unit 13 and the audio signal from the audio signal processing unit 15 are compressed while being temporally related to each other by the compression processing unit 16, and after compression, they are stored in the memory card 18. To be recorded.

  The operation mode of the imaging device 1 includes a shooting mode in which a moving image or a still image can be shot, and a playback mode in which the moving image or the still image stored in the memory card 18 is played back and displayed on the display unit 27. Transition between the modes is performed according to the operation on the operation key 26c. The start or end of moving image shooting is performed according to the operation on the recording button 26a. In addition, still image shooting is performed according to the operation on the shutter button 26b.

  When the user performs a predetermined operation on the operation key 26 c in the reproduction mode, a compressed video signal representing a moving image or a still image recorded on the memory card 18 is sent to the expansion processing unit 19. The decompression processing unit 19 decompresses the received video signal and sends it to the video output circuit 20. In the shooting mode, the generation of the video signal by the video signal processing 13 is normally performed regardless of whether or not a moving image or a still image is shot and recorded, and the video signal is output from the video output circuit 20. Sent to.

  The video output circuit 20 converts a given digital video signal into a video signal (for example, an analog video signal) in a format that can be displayed on the display unit 27 and outputs the video signal. The display unit 27 is a display device such as a liquid crystal display, and displays an image corresponding to the video signal output from the video output circuit 20.

  When a moving image is reproduced in the reproduction mode, a compressed audio signal corresponding to the moving image recorded on the memory card 18 is also sent to the expansion processing unit 19. The decompression processing unit 19 decompresses the received audio signal and sends it to the audio output circuit 21. The audio output circuit 21 converts a given digital audio signal into an audio signal in a format that can be output by the speaker 28 (for example, an analog audio signal) and outputs the audio signal to the speaker 28. The speaker 28 outputs the sound signal from the sound output circuit 21 to the outside as sound (sound).

  Camera shake may be applied to the imaging apparatus 1 during the exposure period of each captured image. The imaging device 1 has a function of detecting a blur locus on an image derived from camera shake based on comparison between adjacent images. Furthermore, based on this detection result, a function is provided for restoring an image obtained by removing the blur from the captured image including the blur. Hereinafter, these functions will be described in detail.

  In the shooting mode, shooting is sequentially performed in a predetermined frame period (for example, 1/60 seconds), and one frame image (shot image) is obtained in each frame. Now, a certain frame image is regarded as an image to be subjected to camera shake detection, and is referred to as a “detection target image”. Since the detection target image is also a target for image restoration based on the camera shake detection result, it can also be called a restoration target image. The frame image in the frame immediately before the frame from which the detection target image is obtained is referred to as “first reference image”, and the frame image in the frame after the frame from which the detection target image is obtained is referred to as “second reference image”. "

  Although a moving image is formed by a plurality of frame images arranged in time series, any frame image forming the moving image can be a detection target image. A single still image is only one of frame images that are sequentially photographed at a predetermined frame period. Accordingly, a still image to be recorded on the memory card 18 in response to the pressing of the shutter button 26b can also be a detection target image. In the following description, attention is paid to a certain detection target image.

  In the description of this embodiment, the words “camera shake” and “camera shake” are used. However, when attention is paid only to camera shake, camera shake and camera shake can be considered synonymous. When attention is paid only to camera shake, the camera shake is derived from camera shake.

[Explanation of the principle of motion detection method]
First, the principle of the motion detection method of this embodiment will be described. When exposure is performed with the rolling shutter, the start time (and end time) of the exposure period is not the same time in the vertical direction in the image, and the exposure period is delayed as it goes from the top to the bottom in the vertical direction. Therefore, when motion detection is performed based on comparison between adjacent frame images, if motion detection is performed at different positions in the vertical direction, sampling points (sampling times) for motion detection differ. That is, between the motion detected for a certain vertical position and the motion detected for a different vertical position between adjacent frame images, the time of the detected motion is different.

  FIG. 4 shows an example in which motion detection is performed for two vertical positions. In this case, there are a total of four sampling points (sampling times) for motion detection with respect to the detection target image. Since the motion of the image derived from camera shake is the same for the entire image, if the sampling time for motion detection is the same, the same motion is detected regardless of the pixel signal in any region in the image. . For this reason, if the detection target image and the first and second reference images are divided into regions in the vertical direction, and based on the motion detection result in each divided region, it is possible to estimate the locus of camera shake at three or more sampling points. It becomes possible.

  The number of area divisions in the vertical direction (the value of M to be described later) is arbitrary. As the number of area divisions is increased, the number of motion detection sampling points increases and finer motion detection can be realized. FIG. 5 shows an example in which each image is divided into four in the vertical direction. The number of sampling points can also be referred to as the number of samplings (or resolution) in the time direction of motion detection. By increasing the number of sampling points, more accurate detection is possible for uniform motion on the image, such as camera shake. It becomes.

[Detection and restoration configuration]
Next, a configuration for realizing camera shake detection and image restoration will be specifically described. FIG. 6 shows a block diagram of parts involved in camera shake detection and image restoration. The motion detection unit 51 and the camera shake trajectory estimation unit 52 in FIG. 6 are provided in the video signal processing unit 13 in FIG. 1 or a detection processing unit (not shown) connected to the video signal processing unit 13. The image deterioration function calculation unit 53 and the image restoration filter coefficient calculation unit 54 in FIG. 6 are realized by the CPU 23 in FIG. The filtering unit 55 and the ringing removal unit 56 in FIG. 6 are provided in the video signal processing unit 13 in FIG.

  The motion detector 51 detects a motion between adjacent frame images. Each frame image is captured by dividing it into M in the vertical direction. For this reason, each frame image is considered to be divided into M vertical division regions. M is an integer of 2 or more.

  FIG. 7 shows how each frame image is divided. Each vertical division area is represented by AR [j] with reference to the origin O of the frame image. Here, j is an integer satisfying 1 ≦ j ≦ M. Vertical division areas AR [j] having the same j are composed of pixels on the same horizontal line. For example, the pixel signals of the pixels in the vertical division area AR [1] are represented by the pixel signals of the pixels forming the first to third horizontal lines in FIG. 3 to form the fourth to sixth horizontal lines. The pixel signal of each pixel in the vertical division area AR [2] is represented by the pixel signal of each pixel,..., By the pixel signal of each pixel forming the (n-2) th to nth horizontal lines, The pixel signal of each pixel in the vertical division area AR [M] is represented.

  The motion detection unit 51 uses, for example, a known image matching method (for example, a block matching method or a representative point matching method) and compares the video signals between the adjacent frame images, thereby moving the motion vector between the adjacent frame images. Are detected for each vertical division area AR [j]. The motion vector detected for each vertical division area AR [j] is particularly referred to as an area motion vector. The area motion vector for a certain vertical division area AR [j] specifies the magnitude and direction of the movement of the image in the vertical division area AR [j] between adjacent frame images.

  When calculating the region motion vector for a certain vertical division region, the region motion vector may be calculated based on the pixel signals of all the pixels in the vertical division region. The region motion vector may be calculated based only on pixel signals of some pixels.

  In addition, as shown in FIG. 8, each frame image is divided into M in the vertical direction and N in the horizontal direction, and a motion vector is calculated for each (M × N) divided regions, and the motion on the same horizontal line is calculated. You may make it handle the average vector obtained by averaging a vector as said area | region motion vector.

  FIG. 9 shows a scene of shooting by the imaging unit 11. In FIG. 9, reference numeral 100 represents a locus of camera shake applied to the imaging apparatus 1 in the vicinity of the imaging timing of the detection target image. Due to this camera shake, the subject that falls within the imaging area of the imaging unit 11 changes from moment to moment.

  In FIG. 9, a rectangular area denoted by reference numeral 101 represents a shooting area of the imaging unit 11 at a timing when exposure of the first reference image is started (that is, exposure start timing of the first horizontal line of the first reference image). A rectangular area denoted by reference numeral 102 represents a shooting area of the imaging unit 11 at a timing at which exposure of the detection target image is started (that is, exposure start timing of the first horizontal line of the detection target image). The rectangular area marked with represents the shooting area of the imaging unit 11 at the timing when the exposure of the second reference image is started (that is, the exposure start timing of the first horizontal line of the second reference image). The rectangular area thus obtained represents the imaging area of the imaging unit 11 at the timing when the exposure of the next frame image of the second reference image is started.

As shown in FIG. 10, the timing of exposure of the first reference image is started (i.e., the exposure start timing of the first horizontal line of the first reference image) to represent at T _A, the timing of exposure of the detection target image is started (i.e., the exposure start timing of the first horizontal line of the detection target image) represents at T _B, the timing of exposure of the second reference image is started (i.e., the exposure start timing of the first horizontal line of the second reference image) Is represented by T _C.

  Now, assume that the number of area divisions in the vertical direction is 3 (that is, M = 3). FIG. 11 shows examples of detected region motion vectors. In FIG. 11, reference numerals 121, 122, and 123 represent region motion vectors for vertical division regions AR [1], AR [2], and AR [3] between the first reference image and the detection target image, respectively. Reference numerals 131, 132, and 133 denote area motion vectors for the vertical division areas AR [1], AR [2], and AR [3] between the detection target image and the second reference image, respectively.

  When considering the movement of an image derived from camera shake between the first reference image and the detection target image, the three region motion vectors are the same if the exposure timing of each pixel of the image sensor is the same. However, since the image sensor 33 in FIG. 2 exposes each pixel with the rolling shutter as described above, the three region motion vectors 121 to 123 can be different vectors as shown in FIG. The same applies to the detection target image and the second reference image. Such a phenomenon becomes prominent when the frame period is close to the camera shake period or when the frame period is longer than the camera shake period as in the example shown in FIG.

  In such a case, if the hand movement trajectory of the detection target image is estimated from the average vector of the region motion vectors 121 to 123 and the average vector of the region motion vectors 131 to 133, a conventional method using a CCD is used. Similarly, the camera shake trajectory during the exposure period of the detection target image cannot be accurately estimated. This is because there are two sampling points for motion detection during the exposure period of the detection target image. In the present embodiment, taking advantage of the exposure characteristics of the rolling shutter, the sampling point is set to 3 or more (when M = 3, the sampling point is 6).

  The region motion vector between the first reference image and the detection target image and the region motion vector between the detection target image and the second reference image are transmitted to the camera shake locus estimation unit 52 in FIG. 6 as region motion information. The hand movement trajectory estimation unit 52 estimates the hand movement trajectory applied to the imaging device 1 during the exposure period of the detection target image based on the region motion information. Actually, the locus of the blur (blur on the image) included in the detection target image derived from this camera shake is estimated.

  Although the estimated blur is a two-dimensional amount, a detailed description will be given focusing on the vertical component of the blur trajectory.

Please refer to FIG. In FIG. 12A, the horizontal axis represents time, and the vertical axis represents the vertical position of the imaging region of the imaging unit 11 in real space. A curve 150 represents a temporal change in the vertical position of the imaging region in the vicinity of the imaging timing of the detection target image. Timings T _A , T _B and T _C are the same as those in FIG. The timing T _Ai represents the average exposure timing of the pixels corresponding to the vertical division area AR [i] of the first reference image, and the timing T _Bi is the average of the pixels corresponding to the vertical division area AR [i] of the detection target image. This represents the exposure timing, and the timing T _Ci represents the average exposure timing of the pixels corresponding to the vertical division area AR [i] of the second reference image (where i = 1, 2, 3).

In FIG. 12A, points 161, 162, 171, 172, and 181 are points on the curve 150 at timings T _A1 , T _A2 , T _B1 , T _B2, and T _C1 , respectively. The vertical component of the region motion vector 121 in FIG. 11 takes a value corresponding to the difference in vertical position between the point 161 and the point 171. Similarly, the vertical component of the region motion vector 122 in FIG. 11 takes a value corresponding to the difference in vertical position between the point 162 and the point 172. Similarly, the vertical component of the region motion vector 131 in FIG. 11 takes a value corresponding to the difference in vertical position between the point 171 and the point 181. The same applies to other region motion vectors.

Since the region motion vector 121 in FIG. 11 represents the magnitude and direction of the image motion between the timings T _{A1 and} T _B1 , the region motion vector 121 has a vertical component between the timings T _{A1 and} T _B1 . It is possible to calculate the average speed and direction of the vertical movement of the image. The same applies to other timings. Although it is necessary to consider time, the speed of motion and the magnitude of motion (motion amount) are concepts that can be considered interchangeably. In the following description, attention is paid to speed.

  FIG. 12B shows the vertical motion speed (including the motion direction) of the image calculated from the region motion vectors 121 to 123 and 131 to 133 in FIG. In FIG. 12B, the horizontal axis represents time, and the vertical axis represents the speed of the movement. In FIG. 12B, the point on the line 200 that matches the horizontal axis indicates that the velocity is zero, the upward direction of the vertical axis corresponds to the movement of the image in the upward direction, and the downward direction of the vertical axis is Corresponds to the downward movement of the image.

In FIG. 12B,
A point 201 represents an average motion speed and direction of the image in the vertical direction between timings T _{A1 and} T _B1 calculated from the vertical component of the region motion vector 121 in FIG.
Point 202 represents the average motion speed and direction of the vertical image between timings T _{A2 and} T _B2 , calculated from the vertical component of the region motion vector 122 of FIG.
A point 203 represents the average motion speed and direction of the image in the vertical direction between timings T _{A3 and} T _B3 calculated from the vertical component of the region motion vector 123 in FIG.
A point 211 represents the average motion speed and direction of the vertical image between timings T _{B1 and} T _C1 calculated from the vertical component of the region motion vector 131 in FIG.
A point 212 represents the average motion speed and direction of the vertical image between timings T _{B2 and} T _C2 calculated from the vertical component of the region motion vector 132 of FIG.
A point 213 represents the average motion speed and direction of the image in the vertical direction between timings T _{B3 and} T _C3 calculated from the vertical component of the region motion vector 133 in FIG.

  A solid curve 230 in FIG. 12C represents a camera shake trajectory in the vertical direction on the image (vertical component of the camera shake trajectory) estimated by the camera shake trajectory estimation unit 52 during the exposure period of the detection target image. Starting from the point 202 in FIG. 12B, the vertical position change due to the movement between the points 202-203 (in this example, a relatively large change in the downward direction), the vertical position due to the movement between the points 203-211 By joining together the change (in this example, a relatively small change in the downward direction) and the vertical position change due to the movement between points 211-212 (in this example, almost no change), the curve 230 Is drawn.

  In FIG. 12C, a dashed curve 231 represents a true camera shake locus in the vertical direction on the image (a vertical component of the true camera shake locus) during the exposure period of the detection target image. By increasing the number of samplings in the time direction, it can be seen that the estimated camera shake trajectory corresponding to the curve 230 substantially matches the true camera shake trajectory corresponding to the curve 231.

  Although the method for estimating the camera shake locus has been described focusing on the vertical component, the same estimation is performed on the horizontal component. Accordingly, the camera shake trajectory estimation unit 52 in FIG. 6 estimates the trajectory of the shake (blur on the image) included in the detection target image, which is derived from the camera shake. Information representing the blur locus is sent to the image deterioration function calculating unit 53 as shake locus information.

  The estimated blur trajectory can be expressed as a point spread function (hereinafter referred to as PSF). An operator or a spatial filter in which an ideal point image is weighted in accordance with a trajectory drawn on the image by a shake of the imaging apparatus 1 is called a PSF, and is generally used as a mathematical model of camera shake.

  When the detection target image is considered as a deteriorated image including blur, the PSF is interpreted as an image deterioration function. The image deterioration function calculation unit 53 in FIG. 6 calculates a PSF as an image deterioration function from the camera shake trajectory information.

  An image restoration filter for removing blur included in the detection target image can be generated from the PSF. In the present embodiment, the image restoration filter itself is mounted in the filtering unit 55, and the image restoration filter coefficient calculation unit 54 in FIG. 6 calculates the filter coefficient of the image restoration filter based on the PSF. The calculated filter coefficient is used as a filter coefficient of the image restoration filter implemented in the filtering unit 55.

  The filtering unit 55 in FIG. 6 generates a filtered image in which the blur included in the detection target image is removed or reduced by filtering the detection target image using the image restoration filter. Since the filtering image may include ringing associated with filtering, the ringing removing unit 56 removes the ringing.

  That is, the ringing removal unit 56 generates a restored image by performing weighted averaging of the filtered image and the detection target image. For example, the weighted average is performed for each pixel, and the ratio of the weighted average for each pixel is determined according to the edge strength of each pixel of the detection target image. The generated restored image is an image in which blur included in the detection target image is removed or reduced, and ringing due to filtering is removed or reduced. However, the filtering image generated by the filtering unit 55 can be handled as a restored image as it is. In this case, the ringing removal unit 56 is omitted.

  Note that a method for calculating a PSF from a blur locus, a method for generating an image restoration filter from the PSF (a method for calculating a filter coefficient of the image restoration filter from the PSF), and a degraded image (detection target image) using the image restoration filter. Since a technique for generating a filtered image from which the blur is included and a technique for removing ringing that can be included in the filtered image are all known, detailed description thereof is omitted. As those methods, for example, the method described in Patent Document 1 may be used.

  Further, the hand movement trajectory estimated by the above-described method corresponds to the hand movement trajectory with respect to the central region of the detection target image. Since the exposure timing is different if the vertical position of the image is different, the camera shake trajectory during the exposure period is strictly different if the vertical position is different. However, in this embodiment, it is assumed that the influence of ignoring this strictness is insignificant, and the camera shake trajectory with respect to the central region is estimated and applied to the entire region of the detection target image.

[Apply high-frequency enhancement]
As described above, by using a rolling shutter and dividing each image in the vertical direction, the number of motion detection samples can be increased and the accuracy of motion detection can be improved. However, since the time interval between adjacent frames used for motion detection is not shortened compared to the conventional case, the estimated camera shake trajectory is temporally averaged (in other words, high-frequency attenuation processing or low-pass filter processing). ) Is the camera shake trajectory. For example, the detected motion speed corresponding to the point 201 in FIG. 12B is not the motion speed in a minute section shortened according to the number of area divisions, but is merely an average between the timings T _{A1 and} T _B1. (This average time follows the time interval between adjacent frames).

  Therefore, in order to compensate for the high-frequency component attenuated by the averaging, it is preferable to perform high-frequency emphasis processing in the time direction on the motion data calculated from the region motion information. Then, using the motion data after the high-frequency emphasis processing, the locus of the blur of the detection target image (the blur on the image) is estimated. A high-pass filter (correction unit) that performs this high-frequency emphasis processing is provided, for example, between the motion detection unit 51 and the camera shake trajectory estimation unit 52 or inside the camera shake trajectory estimation unit 52. The motion data represents the speed of motion of the image at each timing on the image.

  With reference to FIG. 13, a numerical example corresponding to the high frequency emphasis processing will be described. For simplification of description, attention is paid only to the vertical movement. FIG. 13A shows each numerical value in this numerical example, and FIG. 13B shows a graph corresponding to this numerical example. Each frame image is vertically divided into five equal parts, the first reference image is exposed between time 0.0 and time 5.0, and the detection target image is exposed between time 5.0 and time 10.0. Assume that the second reference image is exposed between time 10.0 and time 15.0.

  In FIG. 13A, each numerical value described in the second column represents the vertical position of the imaging region at each time, and each numerical value described in the third column represents the true motion speed at each time. Each numerical value shown in the fourth column represents a motion speed at each time detected from the region motion information (hereinafter referred to as a detected motion speed).

  A time difference “1” is considered as a unit time. For example, the true movement speed (in other words, the true movement amount) between time 0.0 and 1.0 is represented by the vertical position difference between time 0.0 and 1.0. On the other hand, the detected motion speed between times 0.0 and 5.0 calculated from the region motion information is a value obtained by multiplying the vertical position difference between times 0.0 and 5.0 by 1/5 (= 2. 2).

  The true movement speed between time 0.0 and 1.0 calculated by the difference in the vertical position is the average movement speed between time 0.0 and 1.0. Therefore, it may be appropriate to interpret the movement speed as the movement speed at time 0.5, but for convenience, the movement speed is delayed by half of the unit time and is considered as the movement speed at time 1.0. Accordingly, the detected motion speed between time 0.0 and 5.0 is also delayed by half of the unit time, and this is considered as the detected motion speed at time 3.0.

  In FIG. 13B, a solid line 251 plots the vertical position of the imaging region, a broken line 252 plots the true movement speed, and a solid line 253 indicates the detected motion. The velocity is plotted. The horizontal axis in FIG. 13B represents time. Then, by applying high-frequency emphasis processing in the time direction to the detected motion speeds arranged in time series, a solid line and a thick broken line 254 are drawn. It can be seen that the broken line 254 is more similar to the broken line 252 corresponding to the true movement speed than the broken line 253.

  In order to optimize this similarity, it is necessary to optimize the characteristics of the high-frequency enhancement processing, that is, the filter characteristics of the high-pass filter. The characteristics of the high-frequency enhancement processing include the frame period and the sampling interval for motion detection. It should be set appropriately in consideration. That is, the characteristic (frequency response characteristic) of the high-frequency emphasis processing should be appropriately changed according to the number of area divisions M in the vertical direction.

<< Deformation, etc. >>
As modifications or annotations of the above-described embodiment, notes 1 to 4 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values.

[Note 2]
The timing at which each part in FIG. 6 functions effectively is arbitrary. This is because the restored image can be generated through the calculation of the region motion information at any timing as long as there is each image data of the detection target image and the first and second reference images.

  For example, the following processing procedure can be adopted. Only region motion information or camera shake trajectory information is created in the shooting mode and stored in the memory card 18 of FIG. 1, and when there is a restored image generation instruction in the playback mode, the stored region motion information or camera shake information is stored. A restored image is generated from the detection target image based on the trajectory information.

[Note 3]
The imaging apparatus 1 in FIG. 1 can be realized by hardware or a combination of hardware and software. In particular, the function of each part shown in FIG. 6 can be realized by hardware, software, or a combination of hardware and software. It can also be realized.

  When the imaging apparatus 1 is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. 6 is described as a program, and the program is executed on a program execution device (for example, a computer), whereby all or one of the functions is realized. You may make it implement | achieve a part.

[Note 4]
In the above-described embodiment, the motion detection unit 51 in FIG. 6 functions as a region motion detection unit, and the restoration unit that generates a restored image includes an image degradation function calculation unit 53, an image restoration filter coefficient calculation unit 54, a filtering 55, and The ringing removal unit 56 is formed.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. It is an internal block diagram of the imaging part of FIG. It is a figure for demonstrating the rolling shutter by the image pick-up element of FIG. It is a conceptual diagram showing the principle of the motion detection process which concerns on embodiment of this invention. It is a conceptual diagram showing the principle of the motion detection process which concerns on embodiment of this invention. It is a block diagram of the site | part which is mounted in the imaging device of FIG. It is a figure which shows the mode of the division | segmentation of each frame image defined by the motion detection part of FIG. It is a figure which shows the modification of the division | segmentation method shown by FIG. It is a figure showing the scenery of imaging | photography by the imaging part of FIG. 1, and is a figure which shows a mode that an imaging | photography area | region changes derived from camera shake. It is a figure which shows each exposure timing of a detection target image and a 1st and 2nd reference image. It is a figure which shows each area | region motion vector calculated regarding a detection target image. A diagram (a) showing the vertical position of the imaging region in the vicinity of the imaging timing of the detection target image, a diagram (b) showing the vertical motion speed of each timing calculated from each region motion vector, and movements arranged in time series It is a figure (c) showing the hand movement locus | trajectory estimated based on speed. It is a figure for demonstrating the significance by performing the high region emphasis process of the time direction with respect to the detected motion data. It is a figure showing the scenery of imaging | photography with the conventional imaging device, and is a figure which shows a mode that an imaging | photography area | region changes from camera shake. It is a figure showing the time change of the perpendicular | vertical position of an imaging region corresponding to the change of the imaging region shown by FIG. It is a figure which shows the sampling point of the motion detection by the conventional imaging device. It is a figure which shows the camera shake locus | trajectory estimated with the conventional imaging device, and a true camera shake locus | trajectory.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 13 Video signal processing part 23 CPU
51 Motion Detection Unit 52 Camera Shake Trajectory Estimation Unit 53 Image Degradation Function Calculation Unit 54 Image Restoration Filter Coefficient Calculation Unit 55 Filtering Unit 56 Ringing Removal Unit

Claims (7)

A motion detection device that performs motion detection on a detection target image obtained by shooting based on an output of an image pickup device that performs shooting while varying exposure timing between different horizontal lines,
Each image obtained by photographing before and after the detection target image is a reference image,
A region motion detection unit that divides the detection target image and each reference image in a vertical direction and detects a motion of an image in each divided region between the detection target image and each reference image as a region motion;
A motion detection apparatus that performs motion detection on the detection target image based on each region motion. The motion detection apparatus according to claim 1, wherein motion detection is performed on the detection target image by regarding each region motion as movement at different times with respect to the entire detection target image. 3. The motion detection according to claim 2, further comprising correction means for correcting each region motion by performing high-frequency emphasis processing in a time direction on data representing each region motion arranged in time series. apparatus. The motion detection apparatus according to claim 3, wherein characteristics of the high-frequency emphasis process are changed according to the number of area divisions in the vertical direction with respect to the detection target image and each reference image. 5. The motion detection apparatus according to claim 1, wherein each region motion arranged in time series represents a blur locus included in the detection target image. A motion trajectory estimation unit configured to estimate a motion trajectory included in the detection target image based on a motion detection result by the motion detection device according to claim 1;
An image pickup apparatus comprising: a restoration unit configured to generate a restored image in which the blur in the detection target image is reduced based on the estimated blur locus. A motion detection method for performing motion detection on a detection target image obtained by shooting based on an output of an image pickup device that performs shooting while varying exposure timing between different horizontal lines,
Each image obtained by photographing before and after the detection target image is a reference image,
The detection target image and each reference image are divided into regions in the vertical direction, and the movement of the image in each divided region between the detection target image and each reference image is detected as a region motion,
A motion detection method, wherein motion detection is performed on the detection target image by regarding each region motion as movement at different times with respect to the entire detection target image.