JP2011205215A - Image pickup device and control method thereof - Google Patents

Abstract

Motion is detected without using past images.
Two CMOS sensors of a rolling shutter system are provided in scanning directions opposite to each other. When the subject has movement, the moving body image is distorted or enlarged or reduced depending on the direction of movement. Based on the difference image between the forward image and the backward image, the presence / absence of movement, the direction thereof, and the like are detected.
[Selection] Figure 2

Description

  The present invention relates to an imaging apparatus and a control method thereof.

  2. Description of the Related Art A motion detection process for detecting a motion (existence of a moving object) in an image represented by image data is known. Japanese Patent Application Laid-Open No. H10-228561 describes a technique for detecting motion using a difference signal obtained by taking a difference between images taken at different timings. Since two images picked up at different timings are used, if a moving body suddenly appears at a high speed, there is a possibility that the detection of the movement may fail.

  In Patent Document 2, pixel signals are read from scanning directions that are opposite to each other in a CMOS image sensor to obtain an image of an odd-numbered row region and an image of an even-numbered row region. An apparatus that analyzes the movement of an object using an image (an image for one frame) is described. In Patent Document 2, since the scanning direction is changed for each row, for example, when a Bayer array filter is used as a color filter, an image in an odd row region and an image in an even row region are red (R) components. Or, any of the blue (B) components is missing. A color shift occurs when an image of an odd-numbered row area and an image of an even-numbered row area are combined to form an image for one frame.

Japanese Patent Laid-Open No. 9-130819 JP 2006-217213 A

  An object of the present invention is to be able to detect a motion (moving body) without using a plurality of frame images taken at different timings and to obtain a color image with little color shift.

  The image pickup apparatus according to the present invention sequentially scans an image pickup signal for each horizontal pixel in the pixel region in which a large number of photoelectric conversion elements are arranged in the horizontal direction and the vertical direction, and outputs the image pickup signal for each horizontal pixel. , Two solid-state imaging devices that are provided so as to be able to image a common imaging target range and have different scanning directions as scanning directions, and the two solid-state imaging devices described above Between the first scan image and the second scan image represented by the color filters formed on the respective light receiving surfaces, the first scan image data and the second scan image data output from the two solid-state image sensors. Difference image data generating means for calculating difference and generating difference image data, and difference image generated by the difference image data generating means Data using the one in which a motion detecting means for detecting at least one of the presence or absence of movement and movement direction of the imaging target range.

  The present invention also provides a control method suitable for the imaging apparatus. This method is a rolling method in which an imaging signal for each horizontal pixel is sequentially scanned in a vertical direction in a pixel region in which a large number of photoelectric conversion elements are arranged in a horizontal direction and a vertical direction, and output of the imaging signal is repeated for each horizontal pixel. A method for controlling an image pickup apparatus including a shutter-type solid-state image pickup element, wherein the image pickup apparatus is provided so as to be able to pick up a common image pickup target range and has different directions as scanning directions, And color filters formed on the respective light-receiving surfaces of the two solid-state image sensors, and the differential image data generation means outputs the first scan image data and the second scan output from the two solid-state image sensors. The difference between the first scanned image and the second scanned image represented by the image data is calculated to generate difference image data, and the motion detection means generates And it detects at least one of the presence or absence of movement and movement direction of the imaging target range by using the differential image data.

  The solid-state imaging device used in the imaging apparatus of the present invention repeats the output of the imaging signal for each horizontal pixel by sequentially scanning the imaging signal for each horizontal pixel (pixels for one horizontal row) in the vertical direction. A rolling shutter type. Since the exposure timing is different for each row, the line image represented by the imaging signal output first in the image for one frame and the line image represented by the imaging signal output lastly have a difference in imaging timing.

  According to the present invention, a shift in imaging timing that occurs in a rolling shutter type imaging apparatus is used. That is, when a subject (moving body) accompanied by a motion is imaged by a rolling shutter imaging device, the motion (moving body) in the captured image is caused by image distortion of the image portion (moving body image) representing the image or enlargement of the image. Or it appears as a reduction. Since at least part of the distorted image, enlarged image, or reduced image remains in the difference image, it is possible to detect the presence of motion, and it is not necessary to continuously capture images of two or more frames at different imaging timings. Even motion can be detected. Moreover, since the scanning directions of the two solid-state imaging devices are different from each other, the shapes of the moving body images appear different from each other in the first scanning image and the second scanning image. As a result, by using the difference image, the direction of movement within the imaging target range can also be added to the detection. Further, since color filters are formed on the light receiving surfaces of the two solid-state imaging devices, a color image for one frame can be formed by each of the first scanning image and the second scanning image. It is also possible to acquire (record, display, etc.) two color images with little color misregistration.

  Preferably, the scanning directions of the two solid-state imaging devices are opposite to each other. That is, if the first scanned image represented by the first scanned image data is an image configured by scanning from the top to the bottom, the second scanned image represented by the second scanned image data is from the bottom to the top. The image is formed by directed scanning.

  There are the following two methods for making the scanning directions of the two solid-state imaging devices opposite to each other.

  One is to provide the image pickup apparatus with two solid-state image pickup elements having the same scanning direction so as to be turned upside down. The second is to use a solid-state imaging device having an address circuit capable of controlling the scanning direction. In any manner, the scanning directions of the two solid-state imaging devices can be reversed.

  The two solid-state imaging devices may capture the imaging target range from different viewpoints (so-called compound eyes), or may capture the imaging target range from the same viewpoint (so-called monocular). .

  By making the scanning directions of the two solid-state imaging elements opposite to each other, the difference image has a specific shape according to the direction of movement. In one embodiment, the motion detection means detects at least one of the presence / absence of motion and the direction of motion according to the image shape appearing in the difference image represented by the difference image data.

  If there is a movement in the horizontal direction (direction parallel to the horizontal pixel array, left-right direction), horizontal image distortion appears in the difference image. In one embodiment, at least one of the presence / absence of motion and the direction of motion (horizontal motion) is detected in accordance with horizontal image distortion appearing in the difference image according to the motion in the horizontal direction.

  When there is a movement in the vertical direction (up and down direction), an enlargement of the vertical image appears in the difference image. In another embodiment, at least one of presence / absence of motion and direction (vertical motion) is detected according to the vertical image enlargement appearing in the difference image in accordance with the vertical motion.

  In the case of a compound-eye imaging device, if there is a movement in the front-rear direction (direction approaching or moving away from the imaging device), horizontal image distortion appears in the difference image. In another embodiment, the presence / absence of the motion and the direction of the motion (the motion in either the horizontal direction or the front-back direction) according to the horizontal image distortion appearing in the difference image according to the motion in the horizontal direction and the front-back direction. ) Is detected.

  In the case of a compound eye imaging device, a difference image may be obtained even when there is no movement within the imaging target range. This is a case where there is a parallax shift due to the two solid-state imaging devices being provided so as to capture the same region from different viewpoints.

  If there is a parallax shift in a compound-eye imaging device, an image area (parallax shift area) indicating the presence of the parallax shift appears in the difference image. In one embodiment, the apparatus further includes a parallax shift detection unit that detects the presence or absence of a parallax shift based on a parallax shift region that appears in the differential image represented by the differential image data.

  When there is a parallax shift in the horizontal direction (left-right direction), the parallax shift area appears in the horizontal direction (left-right direction), and when there is a parallax shift in the vertical direction (vertical direction), the parallax shift area is in the vertical direction (vertical direction). appear. In one embodiment, the parallax deviation detecting means detects the presence and direction of the parallax deviation according to the difference in the position of the parallax deviation area appearing in the difference image according to the direction of the parallax deviation.

  In another embodiment, the first scanning image represented by the first scanning image data and the second scanning image represented by the second scanning image data are overlapped by the overlapping means, and obtained by the overlapping means. In the superimposed image, the center of gravity position of the moving body image included in the first scanned image, the center of gravity position calculating means for calculating the center of gravity position of the moving body image in the second scanned image, and the disparity shift detected by the disparity shift detecting means. A center-of-gravity distance calculating unit that calculates a distance between the two center-of-gravity positions in a direction along the direction, and a first scanned image in a direction in which the center-of-gravity distance decreases according to the center-of-gravity distance calculated by the center-of-gravity distance calculating unit; It further includes parallax correction means for moving at least one of the second scanned images. The above-described parallax deviation can be eliminated or reduced.

  Preferably, a display that generates a display image for one frame by scanning both line images represented by imaging signals for each horizontal pixel output from each of the two solid-state imaging elements in opposite directions. An image generating means and a signal switching means for switching between an imaging signal output from one solid-state imaging device and an imaging signal output from the other solid-state imaging device each time an imaging signal of the same line is output are further provided. One of the two solid-state imaging devices outputs line image data in a direction from the upper end to the lower end of the image, and the other solid-state imaging device outputs line image data from the lower end to the upper end of the image. By constructing a display image for one frame using both line image data from the two solid-state imaging devices, it is possible to shorten the image update time.

It is a schematic diagram of a CMOS image sensor. This is a Bayer color filter. It is a block diagram which shows the electric constitution of an imaging device. The moving body image obtained when the subject including the moving body M that moves to the left is imaged by the imaging device is shown. A moving body image obtained when an object including a moving body M moving upward is imaged by an imaging device is shown. A moving body image obtained when a subject including a moving body M moving in a direction away from the imaging apparatus is captured is shown. It is a flowchart which shows the process sequence of a moving body detection circuit. Each of (A) to (E) shows a difference image represented by difference image data. An image obtained by superimposing two captured images obtained when the moving object M moves to the left is shown. (A) is a display image displayed on the display device by two CMOS sensors 1A and 1B whose scanning directions are inverted vertically, and (B) is a graph with the vertical axis representing the scanning line and the horizontal axis representing the time axis. ing. It is a block diagram which shows schematically the structure of a monocular imaging device.

  FIG. 1 shows an embodiment of the present invention and is a schematic diagram of a CMOS image sensor. FIG. 1 is an enlarged view of an equivalent circuit diagram of a photosensitive portion (a part of the pixel portion) for four pixels included in the CMOS image sensor together with a schematic diagram of the COMS image sensor. FIG. 2 shows an example of a color filter provided on the light receiving surface of the CMOS image sensor.

  A CMOS image sensor (hereinafter referred to as a CMOS sensor) 1 reads a pixel portion (pixel region) 2 in which a large number of photosensitive portions 2A are arranged in the horizontal direction and the vertical direction, and signal charges accumulated in the photosensitive portion 2A. Vertical scanning circuit 3 and horizontal scanning circuit 4, and an analog / digital converter (hereinafter referred to as ADC) 5 for converting signal charges into digital image data. In this specification, “horizontal direction” and “vertical direction” mean directions corresponding to the arrangement direction of the multiple photosensitive portions 2A in the CMOS image sensor 1 and orthogonal to each other.

  Each of the multiple photosensitive portions 2A includes a photodiode 2a, an amplifying element 2b, and a row selection switch 2c. A horizontal signal line 2d is provided above each photosensitive portion 2A, and a vertical signal line 2e is provided on the right side.

  The CMOS sensor 1 of this embodiment is of the rolling shutter type in which the shutter is sequentially released for each horizontal scanning line (one row line). When a row selection pulse is applied to one horizontal signal line 2d by a row selection switch pulse generated by the vertical scanning circuit 3, a large number of row selection switches 2c for one scanning line are turned on. Each of the signal charges stored in the photodiode 2a for one scanning line is amplified by the amplifying element 2b and then sent to the vertical signal line 2e. The signal charge sent to the vertical signal line 2e is converted into digital image data by the ADC 5, and then output from the CMOS sensor 1 by a horizontal scanning pulse generated by the horizontal scanning circuit 4. Digital image data (line image data) representing an image for one scan line (line image data) is scanned in the vertical direction, and output of the line image data is repeated to obtain digital image data representing an image for one frame. It is done.

  Referring to FIG. 2, color filter 6 provided on the light receiving surface of CMOS sensor 1 is a so-called Bayer color filter, and has a characteristic of transmitting a red light component (labeled R). ), A filter having a characteristic of transmitting a green light component (labeled G) and a filter having a characteristic of transmitting a blue light component (labeled B). R and G filters are arranged in the order of RGRG ··· on the odd rows (or even rows), and the G and B filters and the G filters are arranged in the column direction on the even rows (or odd rows) of GBGB ··· Are arranged so that they are not consecutive.

  FIG. 3 is a block diagram illustrating an electrical configuration of the imaging apparatus.

  The imaging device 10 includes two CMOS sensors 1A and 1B. On the respective light receiving surfaces of the CMOS sensors 1A and 1B, the above-described Bayer color filters (not shown in FIG. 3) are formed. Lenses 11A and 11B are provided in front of the CMOS sensors 1A and 1B, respectively.

  Ray bundles representing the subject image are incident on the light receiving surfaces of the CMOS sensors 1A and 1B through the lenses 11A and 11B and the color filter, respectively. A subject image is formed on the light receiving surfaces of the CMOS sensors 1A and 1B, and color digital image data representing the subject image is output from the COMS sensors 1A and 1B.

  The CMOS sensors 1A and 1B are both the above-described CMOS sensor 1, but one CMOS sensor 1B is provided by being rotated 180 degrees (upside down) with respect to the other CMOS sensor 1A. The sensors 1A and 1B have the scanning direction opposite (upside down). In the following description, the CMOS sensor 1A reads line image data in the order of the 0th row, the 1st row, the 2nd row,..., The N-2th row, the N-1th row, and the Nth row ( This reading direction is called “forward direction”, and this scanning in this order is called “forward scanning”. On the other hand, the CMOS sensor 1B reads line image data for each scanning line in the order of the Nth row, the N-1th row, the N-2th row, ..., the second row, the first row, and the 0th row. This reading direction is called “reverse direction”, and this sequential scanning is called “reverse scanning”. A frame image obtained by the CMOS sensor 1A is called a forward image, and a frame image obtained by the CMOS sensor 1B is called a backward image.

  The color line image data output from the forward scanning CMOS sensor 1A and the color line image data output from the backward scanning CMOS sensor 1B are input to the signal processing devices 12A and 12B, respectively. Predetermined signal processing is performed. The line image data signal-processed by the signal processing devices 12A and 12B is input to the main memory 14 and stacked to become image data for one frame.

  A rearrangement control circuit 13 is connected to the main memory 14. The rearrangement control circuit 13 rearranges the upper and lower arrangements of the line image data output from the backward-scanning CMOS sensor 1B (reverses the stacking order of the line image data). Thereby, the vertical direction of the forward direction image and the reverse direction image constituted by a large number of line images obtained in the CMOS sensors 1A and 1B coincide with each other.

  The CMOS sensors 1A and 1B are provided apart from each other by a predetermined distance in the imaging device 10. A stereoscopic image can be obtained from the forward image obtained by the CMOS sensor 1A and the backward image obtained by the CMOS sensor 1B. The stereoscopic image is displayed on the display screen of the display device 15. A set of forward image data and backward image data is recorded in the memory card 16.

  The imaging apparatus 10 further includes a moving object detection circuit 17. The moving object detection circuit 17 detects the presence or absence of movement (moving object) included in the captured image in real time using the forward image and the reverse image obtained by the two CMOS sensors 1A and 1B whose scanning directions are opposite to each other. And it is a circuit which detects the direction of the movement. In addition, when a parallax shift (left-right parallax shift and vertical parallax shift) is included between the forward image and the reverse image, the presence and direction of the parallax are also detected.

FIG. 4 is obtained by the moving body image G ₁ included in the forward image obtained by the CMOS sensor 1A and the CMOS sensor 1B when a subject including the moving body M moving leftward is imaged by the imaging device 10. It shows a mobile image G ₂ contained in the reverse direction image. For ease of understanding, the moving body M is represented by a rectangular shape. In this specification, “leftward” means a direction parallel to the horizontal direction (see FIG. 1) in the CMOS sensors 1A and 1B, and means the leftward direction when the subject (moving body M) is viewed from the imaging device.

  When the moving body M that moves to the left is picked up by the imaging device, distortion occurs in the image portion (moving body image) representing the moving body M in the forward and backward images obtained by the CMOS sensors 1A and 1B. This is because, as described above, the CMOS sensors 1A and 1B are of the rolling shutter type, and the imaging timing of each scanning line is different.

The CMOS sensor 1A performs forward scanning, and the subject is sequentially imaged in the direction from the top to the bottom. Since imaging timing toward the lower end position from the upper end position of the moving body M is delayed, the mobile image G ₁ obtained by the CMOS sensor 1A will represent the moving body M after movement toward the lower end. Distorted to the direction of movement of the moving body M moving object image G ₁ is imaged.

On the other hand, since the CMOS sensor 1B performs reverse scanning, the subject is sequentially imaged in the direction from the bottom to the top. Moving object image G ₂ obtained by the CMOS sensor 1B will represent the moving body M after movement toward the upper end. Distorted to the direction of movement of the moving body M moving object image G ₂ is imaged.

When the moving object M is moving to the right also, mobile image G ₁ obtained by the CMOS sensor 1A in the forward scan becomes represent the moving body M after movement toward the lower end, of the moving body M become distorted in the moving direction (the same shape as the moving object image G ₂ in FIG. 4). Moving object image G ₂ obtained by the CMOS sensor 1B of the reverse scan becomes represent the moving body M after movement toward the upper end, the distorted in the moving direction of the moving object M (object image in FIG. 4 the same shape as the G _1). “Rightward” means a direction parallel to the horizontal direction (see FIG. 1) in the CMOS sensors 1A and 1B and the right direction when the subject (moving body M) is viewed from the imaging device.

A moving object image _{G 1} obtained by the CMOS sensor 1A, on the basis of the moving object image _{G 2} obtained by the CMOS sensor 1B, (presence or absence of the moving object M) movement of existence by motion detection circuit 17 detects Is done. Further, the shape of the image distortion of the moving body image G ₁ obtained by the CMOS sensor 1A, based on the shape of the image distortion of the moving object image G ₂ obtained by the CMOS sensor 1B, the movement direction (movement direction of the moving body M ) Can also be detected (details will be described later).

FIG. 5 shows moving body images G ₃ included in the forward image and the backward image obtained by the CMOS sensors 1A and 1B, respectively, when a subject including the moving body M moving upward is imaged by the imaging device 10. It shows the G _4. In this specification, “upward” means a direction parallel to the vertical direction (see FIG. 1) in the CMOS sensors 1A and 1B and means an upward direction when the subject (moving body M) is viewed from the imaging device. “Downward” means downward when the subject (moving body M) is viewed from the imaging apparatus.

As described above, since the imaging timing of the CMOS sensor 1A for forward scanning is delayed from the upper end position to the lower end position of the moving body M, when the moving body M moving upward is imaged, it is shrunk in the vertical direction (collapsed). the shape of the) mobile image G ₃ is obtained. On the other hand, since the imaging timing toward the upper position from the lower end position of the CMOS sensor 1B in the moving body M in the reverse scan is delayed, the vertical direction is expanded (extended shape) mobile image G ₄ is obtained.

When the moving object M is moving downward, the moving object image G ₃ which is enlarged from the CMOS sensor 1A in the forward scan in the vertical direction obtained (same shape as the moving object image G ₄ shown in FIG. 5), The moving body image G ₄ (the same shape as the moving body image G ₃ shown in FIG. 5) reduced in the vertical direction is obtained from the CMOS sensor 1B for backward scanning.

When the moving body M moves in the vertical direction, as described above, the two moving body images G ₃ and G ₄ representing the moving body M obtained by the CMOS sensors 1A and 1B are enlarged and reduced in the vertical direction. Also in this case, the moving object detection circuit 17 detects the presence / absence of movement (presence / absence of the moving object M), and based on the image shapes (enlarged shape and reduced shape) of the moving object images G ₃ and G ₄ , The direction of movement (movement direction of the moving body M) can also be detected (details will be described later).

6, the forward image G ₅ and the reverse image G ₆ obtained when imaged by the imaging device 10 of a subject comprising the moving body M to move away from the imaging device 10, CMOS sensor 1A, 1B, and The positional relationship between the lenses 11A and 11B and the moving body M is shown. In this specification, the direction approaching the imaging device 10 is referred to as “front direction”, and the direction away from the imaging device is referred to as “rear direction”.

  Unlike the case where the moving body M moves in the horizontal direction or the vertical direction of the CMOS sensors 1A and 1B (imaging devices), the moving direction of the moving body M is the direction perpendicular to the light receiving surface of the CMOS sensors 1A and 1B (optical axis direction). When moving in the back-and-forth direction, the above-described image distortion or enlargement or reduction does not occur in the moving object image. However, if the movement of the moving body M in the front-rear direction is a movement that deviates from the optical axes of the CMOS sensors 1A and 1B, the moving body image in the forward image and the backward image obtained by the CMOS sensors 1A and 1B Image distortion accompanying the movement of the moving body M in the front-rear direction appears.

Consider a case where the moving body M moves in a direction away from the optical axis of the CMOS sensor 1A (that is, a movement not along the optical axis of the CMOS sensor 1B). In this case, the moving body image G ₅ (left image) obtained by the CMOS sensor 1A is hardly affected by the movement of the moving body M. On the other hand, in the moving body image G ₆ (right image) obtained by the CMOS sensor 1B, image distortion due to parallax that changes with the movement of the moving body M appears. That is, when moving to the moving body M in a direction away from the imaging device, the moving body M is imaged as if it moved relatively rightward in the reverse direction image captured by the CMOS sensor 1B due to a change in parallax. The Moving object image G ₆ distorted in the lateral direction is obtained.

  On the contrary, if the moving body M moves in the direction approaching the imaging device, the moving body M is imaged as if it moved relatively leftward in the reverse direction image captured by the CMOS sensor 1B. .

Like the moving body M to be moved in the lateral direction (see FIG. 4), even if the mobile M to move in the longitudinal direction of the image pickup apparatus is present, the moving object image G ₅ of the moving object M obtained by the CMOS sensor 1A , based on the moving object image G ₆ of the moving object M obtained by the CMOS sensor 1B, the presence of motion (moving object M) is detected by the motion detection circuit 17. Further, the shape of the image distortion of the moving body image G ₁ obtained by the CMOS sensor 1A, based on the shape of the image distortion of the moving object image G ₂ obtained by the CMOS sensor 1B, the movement direction (movement direction of the moving body M ) Can also be detected.

  In the case of the movement of the moving body M in the left-right direction (FIG. 4) and the movement of the moving body M in the front-rear direction (FIG. 6), the moving body image has a horizontal Distortion occurs. If the shape of the moving object M is known in advance, the shape of the moving object M is compared with the shape of the moving object image in the forward image and the shape of the moving object image in the backward image. It is possible to distinguish whether the moving direction is the left-right direction or the front-rear direction. If the shape of the moving object M known in advance is different from the shapes of the two moving object images and the shapes of the moving object images are symmetrical, it can be detected that the moving object M is moving in the left-right direction. If only one of the two moving body images substantially matches the shape of the moving body M, or if the two moving body images do not have a symmetric shape, it means that the moving body M has moved in the front-rear direction. Can be detected.

  FIG. 7 is a flowchart showing an example of the processing procedure of the moving object detection circuit 17.

  The moving object detection circuit 17 uses the forward image data and the backward image data output from the forward scan CMOS sensor 1A and the backward scan CMOS sensor 1B and stored in the main memory 14, and uses the following five states. Is detected.

(1) There is no movement (moving body) and there is no parallax deviation. (2) There is movement in the left-right direction or the front-back direction (moving body moving in that direction). (3) Vertical movement (that (4) There is a parallax shift in the left-right direction (horizontal direction). (5) There is a parallax shift in the vertical direction (vertical direction).

  Referring to FIG. 7, first, difference image data representing the difference between the forward image and the backward image obtained by CMOS sensors 1A and 1B is generated (step 21). In the following description, the forward image is represented by + (plus), and the backward image is represented by-(minus). The difference image S represented by the difference image data is calculated by subtracting the pixel value of the pixel at the corresponding position in the backward image from the pixel value (level data) of each of the plurality of pixels constituting the forward image. Is done.

8A to 8E show the difference image S for the moving body image represented by the difference image data, and the five states (modes) detected by the moving body detection circuit 17 are shown in FIGS. Difference images S corresponding to the respective images (identified by reference signs S _{1 to} S ₅ ) are shown. 8A to 8E, a range surrounded by a solid line represents a portion of the moving body image obtained by only one of the CMOS sensors 1A and 1B. A broken line portion represents a portion of the moving body image obtained by both the CMOS sensors 1A and 1B. In the following description, FIGS. 8A to 8E are referred to as appropriate.

  All the pixel values included in the difference image S are added, and the added value is compared with a predetermined threshold value TH1 (step 22). Since the pixel values of all the pixels in the vertical and horizontal directions included in the difference image S are added, this addition is expressed as “Σ vertical and horizontal directions | S |”.

  When the motion (moving body image) is included, the added value increases as compared with the case where the motion is not included. Even when motion is not included, if there is a parallax shift (for example, a parallax shift other than a parallax shift due to a difference in the positions (viewpoint positions) of the CMOS sensors 1A and 1B), a large value is calculated as the addition value. Is done. If the added value is greater than the predetermined threshold value TH1, it is determined that there is a motion (moving body M) or that there is a parallax shift (YES in step 22). On the other hand, if the added value is equal to or less than the predetermined threshold value TH1, there is no motion in the captured image, and the difference image S has a parallax shift and noise due to differences in the arrangement positions (viewpoint positions) of the CMOS sensors 1A and 1B. , It is determined that there is no difference other than errors (hereinafter, this state is referred to as “still”) (NO in step 22).

  If it is determined that there is motion (moving body image) or that there is a parallax shift (YES in step 22), the pixel values in the horizontal direction (horizontal direction) of the difference image S are added, and this value is added in the vertical direction ( Further addition is performed in the vertical direction (step 23). Hereinafter, the addition of the pixel values in the horizontal direction is referred to as horizontal addition, and the value is referred to as the horizontal addition value. The addition of pixel values in the vertical direction is called vertical addition, and the value is called vertical addition value.

The horizontal addition and the vertical addition will be described with reference to the difference images S _{1 to} S _{5 in five} modes shown in FIGS.

FIG. 8 (A) shows a differential image S ₁ when no there is no movement (mobile M), and also the parallax displacement exists. Figure 8 (B) a is the difference image S ₂ which motion is obtained when the lateral direction or rear direction, FIG. 8 (C) is a differential image S ₃ when there is a disparity shift in the horizontal direction, FIG. 8 (D) the difference image S ₄ which movement is obtained when the vertical direction, FIG. 8 (E) is the difference image S ₅ when there is a disparity shift vertically, respectively.

Referring to FIG. 8 (A), as described above, there is no moving object image in the captured image, and the parallax displacement does not exist, all the longitudinal and transverse directions contained in the difference image S ₁ Judgment is made when the sum of the pixel values of these pixels is equal to or less than a predetermined threshold TH1 (NO in step 22).

When there is a motion and the direction of the motion is the left-right direction or the front-back direction, as described above, in the forward direction image and the reverse direction image obtained by the CMOS sensors 1A, 1B, Distortion appears (no distortion appears in the vertical direction) (see FIGS. 4 and 6). Referring to FIG. 8B, the difference image S2 obtained when the movement is in the left-right direction or the front-rear direction is the pixel value ( ₂ ) obtained by the forward-scanning CMOS sensor 1A in the distorted portion (left and right end portions). It includes an image region composed only of (+)) and an image region composed only of pixel values (denoted by-) obtained by the backward scanning CMOS sensor 1B.

When there is a parallax shift in the left-right direction, an image shift appears in the left-right direction in the forward image and the reverse image obtained by the CMOS sensors 1A, 1B. Referring to FIG. 8 (C), the differential image S ₃ when parallax displacement in the horizontal direction exists, the pixel value obtained by the CMOS sensor 1A of either one forward scan of the right and left end portions (+ And an image region composed only of pixel values (denoted by −) obtained by the reverse-scanning CMOS sensor 1B at the other of the left and right ends. Become a thing.

When there is a motion and the direction is the vertical direction, as described above, in the forward image and the reverse image obtained by the CMOS sensors 1A and 1B, the moving body image is enlarged or reduced in the vertical direction ( There is no enlargement or reduction in the left-right direction) (see FIG. 5). Referring to FIG. 8 (D), the differential image S ₃ that motion is obtained when the vertical direction has, in its upper and lower ends, forward scan (+) or reverse scan (-) of the CMOS sensor 1A Alternatively, it includes an image area composed only of pixel values obtained by the CMOS sensor 1B.

When there is a parallax shift in the vertical direction, a shift in the image appears in the vertical direction in the forward and reverse images obtained by the CMOS sensors 1A and 1B. Referring to FIG. 8 (E), a differential image S ₄ when parallax displacement in the vertical direction are present, are pixel values obtained by the forward scanning the CMOS sensor 1A on one of the upper and lower ends (+ And an image region composed only of pixel values (denoted by −) obtained by the reverse-scanning CMOS sensor 1B at the other of the upper and lower ends. .

Returning to FIG. 7, the difference image S ₂ (FIG. 8B) regarding the movement in the left-right direction or the front-rear direction or the difference image S ₃ when the parallax shift in the left-right direction exists (FIG. 8C). If there is, when the absolute value (| Σlateral direction S |) of the lateral direction addition value of the difference image S is calculated, the pixel value (+) obtained by the CMOS sensor 1A and the pixel value (−) obtained by the CMOS sensor 1B. ) Cancel each other, and the value becomes small. A value obtained by adding the horizontal direction addition value in the vertical direction (Σvertical direction | Σhorizontal direction S |) is also a small value.

On the other hand, if the difference image S ₄ (FIG. 8 (D)) regarding the movement in the vertical direction or the difference image S ₅ (FIG. 8 (E)) in which there is a vertical disparity shift, the horizontal direction of the difference image S A large value is calculated as the absolute value of the addition value (Σ horizontal direction S), and a value (Σ vertical direction | Σ horizontal direction S |) obtained by adding the absolute values is also a large value.

  The value of the Σ vertical direction | Σ horizontal direction S | is compared with the second threshold value TH2 (step 23). If the value of the Σ vertical direction | Σ horizontal direction S | is smaller than the second threshold value TH2, it is determined that there is a movement in the left-right direction or the front-rear direction or there is a parallax shift in the left-right direction. (YES at step 23, step 24). If the value of Σ vertical direction | Σ horizontal direction S | is equal to or greater than the second threshold value TH2, it is determined that there is either vertical movement or vertical parallax deviation (step). NO at 23, step 25).

  If the value of Σvertical direction | Σhorizontal direction S | is smaller than the second threshold value TH2 (YES in step 23), whether there is further movement in the left-right direction or the front-rear direction, or whether there is a parallax shift in the left-right direction Is determined (step 24).

Referring to FIG. 8 (B), when there is a lateral direction or longitudinal direction of movement, the absolute value of the vertical direction addition value of the difference image S ₂ ((| Σ longitudinal S |)) After calculating the, CMOS sensor 1A The pixel value (+) obtained by the above and the pixel value (−) obtained by the CMOS sensor 1B cancel each other, and a small value is calculated. A value obtained by adding the vertical direction addition value in the horizontal direction (Σ horizontal direction | Σ vertical direction S |) is also a small value.

Referring to FIG. 8 (C), the if the parallax displacement in the horizontal direction exists, the absolute value of the vertical direction addition value of the difference image S ₃ relatively large when calculating the ((|) | Σ longitudinally S) A value obtained by calculating the value and further adding it in the horizontal direction (Σhorizontal direction | Σvertical direction S |) is also a large value.

  The value of Σ horizontal direction | Σ vertical direction S | is compared with the third threshold value TH3 (step 24). If the value of the Σ horizontal direction | Σ vertical direction S | is smaller than the third threshold value TH3, it is determined that there is a movement in the left-right direction or the front-rear direction (YES in step 24, FIG. 8B). If the value of the Σ horizontal direction | Σ vertical direction S | is equal to or greater than the third threshold TH3, it is determined that there is a parallax shift in the left-right direction (NO in step 24, FIG. 8C).

  Note that when there is a movement in the left-right direction or the front-rear direction and there is also a parallax shift in the left-right direction, the value of Σhorizontal direction | Σvertical direction S | This case is included in the determination of the presence of a left-right direction parallax shift.

  On the other hand, if the value of the Σ vertical direction | Σ horizontal direction S | is equal to or greater than the second threshold value TH2 (NO in step 23), whether there is vertical movement or whether there is vertical parallax shift. Judgment is made (step 25).

Referring to FIG. 8 (D), the case where there is a vertical movement, the absolute value of the vertical direction addition value of the difference image S ₄ (| Σ longitudinal S |) is a relatively large value when calculating the calculated, this The value obtained by adding the values in the horizontal direction (Σ horizontal direction | Σ vertical direction S |) is also a large value. Referring to FIG. 8 (E), if the parallax displacement in the vertical direction is present, the absolute value of the vertical direction addition value of the difference image S ₅ ((| Σ longitudinal S |)) CMOS sensor 1A when calculating the The pixel value (+) obtained by the above and the pixel value (−) obtained by the CMOS sensor 1B cancel each other, and the value becomes a small value. A value obtained by adding the vertical direction addition value in the horizontal direction (Σ horizontal direction | Σ vertical direction S |) is also a small value.

  The value of the Σ horizontal direction | Σ vertical direction S | is compared with the fourth threshold value TH4 (step 25). If the value of the Σ horizontal direction | Σ vertical direction S | is smaller than the fourth threshold value TH4, it is determined that there is a vertical movement (YES in step 25, FIG. 8D). If the value of the Σ horizontal direction | Σ vertical direction S | is equal to or greater than the fourth threshold TH4, it is determined that there is a vertical parallax shift (NO in step 25, FIG. 8E).

  When vertical motion is included and vertical parallax shift also exists, the value of Σhorizontal direction | Σvertical direction S | is equal to or greater than the fourth threshold value TH4. This case is included in the determination of the presence of vertical disparity shift.

  In this way, by using two CMOS sensors 1A and 1B whose scanning directions are opposite to each other (inverted vertically), the movement (moving object) can be performed without using a plurality of frame images including past frame images. ), And its direction (whether it is a movement in the left-right direction or the front-rear direction, or a movement in the up-down direction). It is also possible to determine whether or not there is a parallax shift and its direction (a parallax shift in the horizontal direction or a parallax shift in the vertical direction).

  When it is determined that there is a parallax shift, image correction for eliminating the parallax shift may be performed, and then the moving object detection process may be performed again. FIG. 9 shows the forward image obtained by the CMOS sensor 1A and the reverse image obtained by the CMOS sensor 1B in an overlapping manner, and shows the entire image Ga obtained when the moving body M moves leftward. (See FIG. 4).

In FIG. 9, moving body images G ₁ and G ₂ representing the same moving body M are shown at different positions in the vertical direction and the horizontal direction in the entire image Ga. When both the left-right direction parallax shift and the vertical-direction parallax shift exist, the whole image Ga as shown in FIG. 9 is obtained by superimposing the forward image and the reverse image.

For example, when the presence of a left-right parallax shift is detected (NO in step 24 in FIG. 7), the forward image data or the backward image is set so that the centroids of the moving body images G ₁ and G ₂ match in the entire image a. If data correction is performed, that is, image correction is performed by translating horizontally, the two pieces of corrected image data are those in which the parallax deviation in the left-right direction has been eliminated. For example, if the entire image Ga shown in FIG. 9, since the center of gravity C ₁ of the moving object image G ₁ and the center of gravity C ₂ of the moving object image G ₂ is laterally spaced by a distance L, and the moving body image G ₁ is moved by 1 / 2L in the right direction, and by performing the moving object image G ₂ image correction is moved by 1 / 2L in the left direction, the parallax displacement in the horizontal direction is solved. By performing the moving object detection process again using the two corrected forward and backward images, the accuracy of moving object detection can be improved.

The same applies to the parallax shift in the vertical direction (NO in step 25 in FIG. 7). Note that the vertical parallax shift is considered to be caused by the optical axis shift of the two optical members (lenses 11A, 11B, etc.), so that when the vertical parallax shift is detected, the vertical center of gravity C ₁ , C The angle of the optical member may be adjusted according to the distance between the _two .

When correcting the disparity shift is forward image and the image weight adjusting process a reverse image, the position and range of the detection process of the moving object image G _1, G _2, the center of gravity of the moving object image G _1, G ₂ calculation processing of the position C _1, C _2, calculation of the position of the center of gravity C _1, C ₂ distance between L, and the calculated distance L image correction corresponding to the (mobile) processing, to perform the above motion detection circuit 17 Alternatively, a hardware circuit that performs these processes may be separately provided in the imaging apparatus.

  FIG. 10A shows an example in which a display image is formed on the display device 15 by line images obtained by two CMOS sensors 1A and 1B whose scanning directions are opposite (inverted vertically). FIG. 10B shows the display timing of the line image on a graph with the vertical axis representing the scanning line and the horizontal axis representing the time axis. In FIG. 10B, the display timing of the line image represented by the line image data output from the forward scanning CMOS sensor 1A is represented by the solid line, and the line image data output from the backward scanning CMOS sensor 1B. The display timing of the line image to be displayed is indicated by a broken line.

  When only one CMOS sensor, for example, the CMOS sensor 1A is used, it is assumed that a time of t seconds is required to display an image for one frame. In this case, the update time is t seconds for all line images from the 0th line to the Nth line.

  On the other hand, if a display image for one frame is formed using both line images obtained by scanning from the top and bottom in the two CMOS sensors 1A and 1B, the topmost, The update time of the line image other than the lower end and the center can be shortened. Referring to FIG. 10B, the update time is t seconds at the uppermost end (Nth line), the lowermost end (0th line) and the center (1 / 2Nth line) of the display image. However, for line positions other than these, the line image obtained by the CMOS sensors 1A and 1B is alternately used, so that the display image update time is shorter than t seconds. The display image update time can be locally shortened without changing the exposure time.

  When an image is displayed on the display device 15 in units of line images as described above rather than in units of frame images, the display device 15 or a circuit provided separately for line image reading and line image switching (display image generation circuit) Forward and backward line image data stored in the main memory 14 are sequentially given (scanned) from the top to the bottom and from the bottom to the top in the display device 15, respectively. ) And every time line image data of the same line is stored in the main memory 14, the line image data used for image display is switched (updated).

  In the above-described embodiment, the compound eye imaging device in which the two CMOS sensors 1A and 1B whose scanning directions are vertically inverted are provided at different positions has been described. However, the two CMOS sensors 1A and 1B are provided in a monocular imaging device. It may be. FIG. 11 schematically shows a mechanism when two CMOS sensors 1A and 1B are provided in a monocular imaging device.

  The monocular imaging device includes a half mirror 18 having a half mirror surface directed at an angle of 45 degrees with respect to the optical axis of incident light. Incident light passes through the lens 11 and enters the half mirror surface. Incident light incident on the half mirror surface passes through the half mirror 18 as it is and is reflected on the half mirror surface. CMOS sensors 1A and 1B are respectively provided at positions where incident light directed in two directions is incident. The CMOS sensors 1A and 1B are provided upside down as described above. Each of the CMOS sensors 1A and 1B can image a subject at the same angle of view. In the case of a monocular imaging device, the above-described parallax shift does not occur, and erroneous detection of a moving object due to the presence of the parallax shift is prevented.

  In the above-described embodiment, the CMOS sensors 1A and 1B themselves are vertically inverted and provided in the imaging apparatus to realize forward scanning and reverse scanning. However, the CMOS sensors 1A and 1B themselves are inverted vertically. Instead, it may be provided in the imaging device, and the order (direction) of the vertical scanning in the two CMOS sensors 1A and 1B may be controlled by the address circuit so that they are directed in opposite directions. In the CMOS sensors 1A and 1B, an Nth row, an N-1th row, an Nth row from the CMOS sensor 1A by an address circuit (not shown) that determines the generation order of row selection switch pulses generated by the vertical scanning circuit 3. -2 line,..., 2nd line, 1st line, and 0th line are output in the order of line image data of each scanning line, and the CMOS sensor 1B outputs the 0th line, 1st line, 2nd line, ..., line image data of each scanning line may be output in the order of the (N-2) th row, the (N-1) th row, and the Nth row.

1,1A, 1B CMOS (Solid-state imaging device)
2 Pixel part (area)
2a Photodiode 6 Color filters 11A and 11B Lenses 12A and 12B Signal processing circuit 14 Main memory 15 Display device 17 Moving object detection circuit

Claims (15)

A rolling shutter type solid that sequentially scans an imaging signal for each horizontal pixel in the vertical direction in a pixel region where a large number of photoelectric conversion elements are arranged in a horizontal direction and a vertical direction, and repeats the output of the imaging signal for each horizontal pixel. An imaging device provided with an imaging element,
Two solid-state imaging devices provided so as to be able to image a common imaging target range and having different directions as scanning directions;
Color filters formed on the respective light receiving surfaces of the two solid-state imaging devices,
A difference image for generating difference image data by calculating a difference between the first scan image and the second scan image represented by the first scan image data and the second scan image data output from the two solid-state imaging devices. Data detection means, and motion detection means for detecting at least one of the presence or absence of motion and the direction of motion within the imaging target range using the difference image data generated by the difference image data generation means,
An imaging apparatus comprising: The scanning directions of the two solid-state imaging devices are opposite to each other.
The imaging device according to claim 1. Two solid-state imaging devices having the same scanning direction are provided in a state where they are turned upside down.
The imaging device according to claim 2. An address circuit for controlling the scanning direction of the solid-state imaging device;
The imaging device according to claim 2. The two solid-state imaging devices capture the imaging target range from different viewpoints.
The imaging device according to claim 2. The two solid-state imaging devices capture the imaging target range from the same viewpoint.
The imaging device according to claim 2. The motion detection means is
Detecting at least one of the presence or absence of movement and the direction of movement according to the image shape appearing in the difference image represented by the difference image data,
The imaging device according to claim 2. The motion detection means includes
Detecting at least one of the presence or absence of movement and the direction of movement according to horizontal image distortion appearing in the difference image according to the movement in the horizontal direction;
The imaging device according to claim 2. The motion detection means includes
Detecting at least one of the presence or absence of movement and the direction of movement according to the vertical image enlargement appearing in the difference image according to the vertical movement;
The imaging device according to claim 2. The moving body detecting means is
Detecting at least one of the presence or absence of movement and the direction of movement according to the horizontal image distortion appearing in the difference image according to the movement in the horizontal direction and the front-rear direction;
The imaging device according to claim 5. Parallax deviation detecting means for detecting the presence or absence of parallax deviation based on a parallax deviation region appearing in the difference image represented by the difference image data,
The imaging device according to claim 2. The parallax deviation detecting means includes
Detecting the presence and direction of the parallax shift according to the difference in position of the parallax shift area appearing in the difference image according to the direction of the parallax shift;
The imaging device according to claim 11. Display image generating means for generating a display image for one frame by scanning both line images represented by imaging signals for each horizontal pixel output from each of the two solid-state imaging elements in opposite directions. And a signal switching means for switching between an imaging signal output from one solid-state imaging device and an imaging signal output from the other solid-state imaging device each time an imaging signal of the same horizontal line is output.
The imaging device according to claim 2. Superimposing means for superimposing the first scanned image represented by the first scanned image data and the second scanned image represented by the second scanned image data;
A center-of-gravity position calculating unit that calculates a center-of-gravity position of the moving body image included in the first scanning image and a center of gravity position of the moving-body image in the first scanning image in the overlapping image obtained by the above-described overlapping unit;
The centroid distance calculating means for calculating the distance between the two centroid positions in the direction along the parallax deviation direction detected by the parallax deviation detecting means, and the centroid distance according to the centroid distance calculated by the centroid distance calculating means. Parallax correction means for moving at least one of the first scanned image and the second scanned image in a direction in which the distance becomes shorter;
The imaging device according to claim 12, further comprising: A rolling shutter type solid-state image pickup that sequentially scans an image pickup signal for each pixel in the horizontal direction in the vertical direction in a pixel region in which a large number of photoelectric conversion elements are arranged in the horizontal direction and the vertical direction, and repeats output of the image pickup signal for each pixel in the horizontal direction. A method for controlling an imaging device including an element,
The image pickup device is provided so as to be able to pick up a common image pickup target range, and has two solid-state image pickup elements whose scanning directions are different from each other, and a color formed on a light receiving surface of each of the two solid-state image pickup elements. With a filter,
The difference image data generation means calculates a difference between the first scan image and the second scan image represented by the first scan image data and the second scan image data output from the two solid-state image sensors, and calculates the difference. Generate image data,
The motion detection means detects at least one of the presence or absence of motion and the direction of motion within the imaging target range using the generated difference image data;
Control method of imaging apparatus.

Images