JP2011097204A - Imaging apparatus - Google Patents

Abstract

An object of the present invention is to efficiently detect striped flicker that occurs when shooting is performed using a rolling shutter-type CMOS imaging device in a shooting environment in which a light source whose light emission varies periodically is used as illumination, To realize an imaging device capable of highly accurate suppression.
A first photodiode for photographing a subject and a second photodiode for detecting a change in light emission intensity of a light source are arranged on one image sensor, and the first photodiode is arranged. And an imaging device having means capable of individually setting the exposure time of the second photodiode as imaging means of the imaging device, and flicker obtained from the second photodiode of the imaging device The image pickup apparatus is provided with means for correcting the image pickup signal of the subject acquired by the first photodiode based on the feature amount.
[Selection] Figure 1

Description

The present invention relates to an imaging apparatus.

  As background art of this technical field, for example, there is JP-A-2008-147713 (Patent Document 1). In this publication, “[Problem] In an imaging apparatus that captures an image using an image sensor, it is possible to determine the flicker frequency without taking time. [Solution] An image is captured using the image sensor (4). In the imaging device for imaging, readout is performed to sequentially read out electrical signals having the same charge accumulation time for two periods at a plurality of different time differences respectively corresponding to a plurality of frequencies from a preset region of the imaging element (S21, S23). S28), the difference between the electrical signals for the two periods read in the reading step is within a range set in advance for one time difference, and out of the range set in advance for at least one other time difference. (S22, S24, S29), the frequency corresponding to the time difference within the preset range is determined as the flicker frequency. To (S25). "Is described as (see Abstract).

JP 2008-147713 A

  In a video camera with a shooting format of High-Definition (generally expressed as HD) and a digital still camera with a large number of pixels, a CMOS imaging device suitable for high-speed reading is regarded as a suitable imaging device. Since a general CMOS image sensor is driven by a rolling shutter operation in which an exposure time is shifted in units of lines, in a shooting environment in which the illumination intensity varies periodically, the shutter speed and the illumination flashing intensity variation period In the case where the images are not synchronized with each other, an exposure spot is generated in line units, which may cause line flicker during moving image shooting (exposure spots in line units during still image shooting), thereby degrading image quality. Therefore, in an imaging apparatus using a CMOS imaging element, it is required to predict and reduce the occurrence of exposure spots in order to solve such a problem.

  As a prior art for solving the problem, Japanese Patent Laid-Open No. 2004-228867 discloses pre-shooting that drives a CMOS image sensor with a special drive pattern in order to detect the line flicker frequency before shooting a subject, and determines the presence or absence of flicker. Techniques to detect and respond are described.

  However, since fluctuations in emission intensity that occur in actual lighting often show complex waveforms, in order to suppress line flicker well during movie shooting, not only the line flicker frequency is determined, but also the line in real time. It is important to calculate the correction amount of the line flicker for each time in order to take a picture with an appropriate exposure amount, and Patent Document 1 does not include a description regarding handling in real time.

  To suppress striped flicker that occurs when shooting using a rolling shutter CMOS image sensor in a shooting environment that uses a light source whose light emission varies periodically as illumination, whether or not flicker occurs Is detected and the shutter speed is set to an integral multiple of the blinking period of the blinking light source, or gain adjustment is performed so as to cancel out the fluctuation amount of the signal level in units of lines due to exposure spots.

  In an imaging apparatus using a conventional imaging device that uniformly sets the exposure time of the photodiode of the imaging device, it should be originally set in the process of estimating the amount of signal level fluctuation for flicker detection and gain adjustment. Therefore, there is a problem that it is not possible to take a moving image with appropriate exposure.

  Therefore, the problem to be solved by the present invention is, for example, by detecting a flicker feature amount while maintaining an appropriate exposure state in photographing a subject even when the imaging apparatus is in a flicker detection process. An imaging apparatus capable of suppressing flicker is provided.

The present invention employs the configurations described in the claims.
For example, a first photodiode for photographing a subject and a second photodiode for detecting a change in light emission intensity of a light source are arranged on one image sensor, and the first photodiode and the An imaging device having means capable of individually setting the exposure time of the second photodiode is used as imaging means of the imaging device, and characteristics of flicker obtained from the second photodiode of the imaging device The imaging apparatus is provided with means for correcting the imaging signal of the subject acquired by the first photodiode according to the amount.

  According to the present invention, it is possible to provide an imaging apparatus capable of suppressing flicker. For example, in the image pickup apparatus according to the present invention, an appropriate exposure time is set for the first photodiode of the image pickup element, and the second photodiode is exposed to a value suitable for detecting a feature of fluctuation in light emission intensity. You can set the time. For this reason, the second photodiode is used to dynamically trace the blinking cycle and fluctuation amount of the blinking light source in parallel with the photographing of the subject while the first photodiode is kept in an appropriate exposure state. There is an advantage that the correction means can correct the imaging signal of the first photodiode in real time using the trace result of the fluctuation amount of the light source by the second photodiode.

An example of imaging measures represented by functional block diagram Block diagram for explaining the features of the image signal processor Block diagram for explaining the characteristics of the image sensor The figure explaining the exposure timing of an image sensor The figure which showed the connection structure of the pixel which 1st photodiode group consists of The figure which showed the connection structure of the pixel which 2nd photodiode group consists of Illustration explaining exposure time setting Illustration explaining the light emission characteristics of illumination A diagram for explaining the relationship (asynchronous) between fluctuations in emission intensity and exposure timing The figure explaining the relationship (synchronization) of fluctuation of light emission intensity and exposure timing A diagram for explaining the exposure timing and light emission timing when extracting the characteristics of illumination light (when asynchronous) Diagram explaining fluctuations in signal level when asynchronous The figure explaining the exposure timing and light emission timing when extracting the feature of illumination light Flowchart of light source feature extraction procedure

  In the image pickup apparatus according to the present invention, an image pickup device including a photodiode for extracting a feature amount of flicker and a photodiode for shooting a subject is used, and exposure control and subject shooting for extracting the feature amount of flicker are performed. The exposure control to be performed can be performed in parallel.

  An image signal processor that processes an imaging signal simultaneously performs subject image signal processing while extracting a flicker feature amount from a photodiode for extracting a flicker feature amount.

  A microprocessor serving as a control unit of the imaging apparatus controls the imaging apparatus so as to suppress the occurrence of flicker based on the extracted flicker feature amount.

  FIG. 1 is an example of a functional block diagram illustrating the imaging measures of the present invention. In the figure, 101 is an imaging lens, 102 is an imaging device, 103 is an image signal processor, 104 is an image compression / decompression processor, 105 is a storage device, 106 is a user interface, 107 is a microprocessor, and 108 is a display device. Yes.

  The outline of the image pickup apparatus according to the present invention shown in FIG. 1 will be described below.

  The image pickup lens 101 has a lens function for condensing light, a telephoto function for adjusting the angle of view, and an aperture function for limiting the amount of light. An image of the subject is formed on the light receiving unit of the image sensor 102 at the angle of view.

  In the light receiving portion of the image sensor 102, photodiodes are arranged at regular intervals, and the photodiodes convert light energy imaged on the light receiving portion into electrical energy by photoelectric conversion. The electrical energy accumulated by the photoelectric conversion is periodically read out in line units as an imaging signal. On the other hand, the electrical energy is reset at a timing set in order to keep the imaging signal level appropriate and discharged to the substrate. In addition, as the photodiode, there are a first photodiode for photographing a subject and a second photodiode for extracting a feature quantity of the light source, which are arranged on the same line so as to divide the area for each purpose. Yes. For these first and second photodiodes, it is possible to set an individual exposure time for the reset timing of electrical energy in accordance with the purpose of use of the photodiode, and for photographing a subject. An appropriate exposure time is set for the first photodiode when photographing a subject, and an appropriate exposure time is set for the second photodiode for extracting the feature quantity of the light source. Is done.

  An image signal processor 103 converts an image signal read from the image sensor 102 into a luminance signal and a color signal and outputs the signal as an image signal, while extracting a feature amount of illumination light. Further, when it is necessary to correct the exposure spot of the image as a result of extracting the feature amount, the exposure spot is also corrected.

  The image compression / decompression processor 104 is an image composed of luminance signals and color signals output from the image signal processor 103 using an image compression format such as MPEG (Moving Picture Experts Group) or JPEG (Joint Photographic Experts Group). The signal is compressed and written into the storage device 105 as compressed image data, or the compressed image data read from the storage device is expanded into an image signal composed of a luminance signal and a color signal. Further, the image compression / decompression processor is composed of an image signal composed of a luminance signal and a color signal supplied from the image processor, or a luminance signal and a color signal read out from a storage device and decompressed. The image signal is supplied to the display device 108 and converted into an external signal input format generally used in a display device for displaying television or video and output.

  A storage device 105 is a storage unit that uses a nonvolatile storage unit such as a magnetic disk or a semiconductor memory, and records and holds compressed image data supplied from the image compression / decompression processor 104.

  A user interface 106 is command input means including a circuit open / close switch, volume resistance, pressure sensor, and the like, and is command input means used by the user to control the operation state of the imaging apparatus.

  A microprocessor 107 determines the operating state of the imaging apparatus by a command input from the user interface 106, and also provides a focus degree obtained from the image signal processor 103, the brightness of the subject, the color temperature of the light source, 101 imaging lens, 102 imaging device, 103 image signal processor, and 104 image compression / decompression processor so that a suitable shooting and recording image quality can be obtained based on the image quality evaluation information of the flicker detection status of the light source. It is a control means to control.

  A display device 108 is a lightweight and low power consumption image display device such as a liquid crystal display or an organic EL display, and displays an image signal supplied from an image compression / decompression processor 104 as an image.

  The above is the outline of the operation of the imaging apparatus according to the present invention.

  Next, the details of the imaging apparatus according to the present invention will be described.

  FIG. 2 is a block diagram for explaining the characteristics of the image signal processor 103 according to the present invention. In the figure, 201 indicates a line delay, 202 indicates a noise filter, 203 indicates a correction gain generation unit, 204 indicates a detection unit, 205 indicates a multiplier, 206 indicates an imaging signal processing unit, and 207 indicates a microprocessor interface.

  FIG. 3 is a block diagram for explaining the features of the image sensor 101 according to the present invention. In the figure, 301 is a first photodiode group, 302 is a second photodiode group, 303 is a pixel amplifier group, 304 is a signal readout switch group, 305 is a first charge discharge switch group, and 306 is a second photodiode discharge group. Charge discharge switch group, 307 is a third charge discharge switch group, 308 is a column selection switch group, 309 is a switch opening / closing means, 310 is an analog gain adjustment amplifier, 311 is a CDS and ADC circuit, 312 is a microprocessor interface, 313 is The setting register of each part is shown.

  Hereinafter, how each function shown in FIG. 2 and FIG. 3 works to solve the conventional problem in this imaging apparatus will be described. First, each block of the imaging device in FIG. The operation will be described.

  The first photodiode group 301 and the second photodiode group 302 are arranged on the imaging surface of the imaging device so as to be arranged at regular intervals. The first photodiode group is a generic term for photodiodes for capturing an image of a subject, and a part of them is a photodiode for specifying a black level in which a light receiving portion of the photodiode is shielded with a metal such as aluminum. Used as. The second photodiode group is a generic term for photodiodes for detecting flicker of a light source and extracting feature amounts. In the photodiodes constituting the first photodiode group and the second photodiode group, photoelectric conversion for converting light into electrical energy is performed. At this time, the electrical energy generated by the photoelectric conversion is proportional to the product of the intensity of light applied to the photodiode and the exposure time, so that the exposure time for each photodiode is kept constant in the photodiode. If the stored electrical energy is read as an electrical signal, an imaging signal that forms an image of the light of the subject projected on the imaging surface of the imaging element can be obtained.

  A pixel amplifier group 303 is a general term for amplifiers for amplifying weak imaging signals read from the photodiodes, and is connected to the photodiodes forming the first photodiode group and the second photodiode group.

  The signal readout switch group 304 is a general term for electronic switches that are opened when the imaging signal amplified by the pixel amplifier is read out, and is connected to the pixel amplifier and electronically opened and closed by a switch opening / closing means 309. Control is performed. The switch open / close control is configured to be shared in units of rows.

  A first charge discharging switch group 305, a second charge discharging switch group 306, and a third charge discharging switch group 307 are provided for discharging unnecessary charges onto a substrate connected to each photodiode. This is a general term for electronic switches, and switch opening / closing control is electronically performed by switch opening / closing means 309 as in the case of the signal readout switch group 304. Each switch group has a circuit configuration in which the discharge timing is controlled for each switch group and an individual exposure time can be set according to the purpose of use of the imaging signal read from the photodiode.

  The column selection switch group 308 is a general term for electronic switches that control which column is selected and output from among the imaging signals read from the photodiodes in units of rows. Open / close control is performed.

  The image pickup signal read out electronically by the switch opening / closing means 309 is subjected to gain adjustment by an analog gain adjustment amplifier 310.

  After the gain-adjusted image signal is adjusted by a CDS circuit of 311 CDS (Correlated Double Sampling) and ADC (analog-to-digital converter) circuit to make the level of correlated double sampling black constant. The analog signal is converted into a digital signal by the ADC circuit, and the digital image signal is output from the image sensor.

  The above is the operation of each block of the image sensor. Next, a method for reading out signals independently from individual pixels in the image sensor will be described.

  When signals are read out independently from individual pixels in the image sensor, the following switch open / close control is performed. First, a signal readout switch for selecting a row of pixels from which a signal is read out is opened by a switch opening / closing means 109. Next, a column selection electronic switch for selecting a column for outputting a signal is opened. The imaging signals in a desired row and column can be read out by the above switch opening / closing operation.

  In addition, when reading imaging signals from all pixels, the operation of shifting the selection of columns from which signals are read in order from the end while fixing the row selection in order from all ends to all rows in order. Just do it.

  By the way, since the capacity of electrical energy that can be stored in the photodiode is limited, when the electrical energy by photoelectric conversion satisfies the capacity by exposing the photodiode, the photodiode is irradiated. It is impossible to maintain a proportional relationship between the product of the intensity of the emitted light and the exposure time and the imaging signal level output from the photodiode. This state is generally called saturation, but the operation of the image sensor for avoiding this saturation will be described below.

  In a general solid-state imaging device including the imaging device according to the present invention, a function of resetting the electrical energy stored in the photodiode by periodically discharging the electrical energy stored in the photodiode onto the substrate. Is provided. This function of resetting electrical energy is generally called an electronic shutter function. For the electronic shutter function, a global shutter system that performs shutter control uniformly on a screen and a rolling shutter system that performs shutter control on a line basis are generally used. In the image sensor used in the present invention, the latter rolling shutter system is used. The method is used.

  In a general rolling shutter system, photodiode reset timing management by a ROW reset control pulse and imaging signal readout timing management by a ROW readout control pulse are performed on the basis of a frame reset pulse as shown in FIG. Tex, which is an exposure period for accumulating electrical energy indicating an image of a subject with a photodiode, is a time until a ROW reset control pulse is output based on a frame reset pulse. Is determined by setting Tread, which is the time until output.

  The exposure time in the image sensor used in the present invention is controlled by controlling the timing of the ROW reset control pulse and the ROW readout control pulse, as in the case of an image sensor employing a general rolling shutter. A combination of photodiodes that are reset simultaneously with timing management has the following characteristics.

  FIG. 5 is a diagram showing a connection configuration of pixels formed by the first photodiode group of the image sensor used in the present invention. In the figure, 501 is a pixel, 502 is a control line for controlling the first charge discharge switch group, 503 is a control line for controlling the signal readout switch group, 504 is a signal line for reading the imaging signal, Reference numeral 505 denotes an electronic switch forming a column selection switch. As shown in the figure, the connection configuration of the pixels formed by the first photodiode group in the imaging device used in the present invention is a rolling shutter type imaging device that reads an imaging signal by designating general coordinates. The same configuration is taken.

  FIG. 6 is a diagram showing a connection configuration of pixels formed by the second photodiode group of the image sensor used in the present invention. As shown in the figure, the connection configuration of the pixels formed by the second photodiode group is different from that of the first photodiode group, and a control line for controlling the charge discharge switch group is provided for even and odd pixels. It is configured to be shared in groups. The control lines for controlling the signal readout switch group are shared in units of rows as in the connection configuration of FIG. 5, and further shared with the same row of pixels formed by the first photodiode group shown in FIG. Is done.

  FIG. 7 shows an example of the exposure control timing of the rolling shutter when attention is paid to an arbitrary row of pixels of the image sensor applied to the present invention. The ROW reset control pulse A is the first photodiode group. Reset, ROW reset control pulse B is reset of even pixels of the second photodiode group, ROW reset control pulse C is reset of odd pixels of the second photodiode group, and ROW readout control pulse is The timing at which reading is performed in units of rows of the photodiode and the second photodiode is shown.

  These ROW reset control pulse A, ROW reset control pulse B, ROW reset control pulse C, and ROW readout control pulse are controlled by the setting of the switch opening / closing means 309, and each pulse has a timing as shown in FIG. When taking the exposure times of the pixels belonging to the first photodiode group, the even pixels belonging to the second photodiode group, and the odd pixels belonging to the second photodiode group in the same row, respectively, TexA, TexB, TexC In this way, three exposure times can be set.

  Then, the imaging signals photographed with these three exposure times are supplied to the 103 image signal processor, and the image signal processor 103 extracts the feature of the illumination light from the imaging signal obtained from the second photodiode. While performing, it is possible to visualize the target subject using the imaging signal obtained from the first photodiode.

  In the feature extraction of illumination light in the image signal processor 103, in order to reduce the influence of random noise superimposed on the imaging signal, in this embodiment, input is performed in units of lines using a line delay 201 as shown in FIG. The image pickup signal is delayed, and a function of performing a two-dimensional noise reduction process by a noise filter 202 is provided.

  Since the noise reduction process needs to be performed for each image signal taken with the same exposure time, in this embodiment, the second photodiode is divided into even pixels and odd pixels, and filtering for noise reduction is performed. I do.

  The feature of the illumination light extracted in the present embodiment is whether or not the light emission intensity is periodically changed, and if it is a light source that periodically changes, what is the blinking cycle? It is.

  Next, a procedure for deriving the characteristics of the illumination light will be described. First, an environment in which the emission intensity of the illumination light varies periodically and the influence of the environment on the imaging signal will be described.

  50 Hz and 60 Hz are mixed in commercial power supply frequencies that are maintained as social infrastructure. At this time, the blinking speed of the illumination such as a fluorescent lamp whose light emission intensity changes following the voltage fluctuation is generally double the power supply frequency, and the light emission intensity of the light source changes at 100 Hz with a 50 Hz power supply and 120 Hz with a 60 Hz power supply. .

  FIG. 8 is an example of a graph of the light emission characteristics of illumination in which the light emission intensity varies according to the variation of the power supply voltage in an area or facility where a 50 Hz commercial power supply is employed.

  As shown in the figure, the fluctuation period in the case of illumination in which the light emission intensity periodically fluctuates according to the voltage fluctuation of the 50 Hz commercial power supply is 1/100 seconds, that is, the fluctuation frequency is 100 Hz.

  On the other hand, when driving an image pickup device in an image pickup apparatus for performing moving image shooting, a frame frequency of 50 Hz or 60 Hz is generally adopted because of its good compatibility with a television broadcast format.

  FIG. 9 shows an arbitrary change in the light emission intensity of the illumination when shooting is performed with the frame rate set to 60 Hz and the exposure time set to 1/120 seconds in an illumination environment where the light emission intensity changes at 100 Hz. The relationship of the timing at which the pixels of the line are exposed is illustrated.

  As shown in the figure, when the light emission timing of the illumination and the exposure timing of the image sensor (period indicated by Tex) are not synchronized, even if a stationary subject is photographed, it is photoelectrically converted by the photodiode. Since the signal level taken out as the imaging signal varies from frame to frame, there is an effect that the luminance level of the image signal varies from frame to frame.

  FIG. 10 shows an arbitrary change in the emission intensity of illumination when shooting was performed with the frame rate set to 50 Hz and the exposure time set to 1/120 seconds in an illumination environment where the emission intensity changed at 100 Hz. FIG. 5 is a diagram illustrating a relationship between a line and a pixel exposure timing of a line from which a signal is read out at a timing shifted by A time from a signal reading timing of the arbitrary line.

  In the drawing, exposure timing A indicates an arbitrary line, and exposure timing B indicates an exposure timing of a line from which a signal is read out at a timing shifted by A time from the signal reading timing of the arbitrary line.

  According to the figure, since the light emission timing of the illumination and the exposure timing of the image sensor (period indicated by Tex) are synchronized in each line, the luminance level of the image signal does not vary from frame to frame, but the same Even when the exposure time is set, the amount of light emitted during the exposure period varies depending on the line.For example, even when an object with uniform light reflectance is photographed on the entire screen, the luminance level of the image signal varies depending on the line. Will come out.

  The above is the environment in which the emission intensity of the illumination light periodically varies and the influence of the environment on the imaging signal.

  Therefore, in this embodiment, a function for extracting the feature of illumination light is provided by an algorithm that outlines the following. Assuming various models of illumination, 107 microprocessors set the exposure time for feature extraction on the second photodiode of the image sensor via the 312 microprocessor interface. The microprocessor compares the prediction of the imaging signal according to the setting state of the exposure time and the imaging result of the second photodiode from the detection unit 204. A plausible model with little difference between the prediction and the imaging result is defined as a feature of the illumination light.

  Therefore, an example of the feature extraction algorithm will be described below.

  An example of the first illumination light feature extraction algorithm will be described below.

  FIG. 11 is a diagram illustrating the relationship between the exposure time set for the second photodiode when extracting the characteristics of illumination light, the exposure timing in an arbitrary line, and the light emission timing of illumination.

  As shown in the figure, in an even number of pixels with an exposure time set to 1/100 second, the exposure timing shifts for each photographing frame with respect to the illumination emission timing, but the illumination emission amount varies 1 Since the period is exposed, the exposure amount of each frame is always constant. On the other hand, in the odd-numbered pixels whose exposure time is set to 1/120 seconds, the exposure amount of each frame varies because one cycle in which the illumination emission amount varies is not exposed.

  FIG. 12 shows the transition of the imaging signal obtained from any odd pixel and even pixel in the second photodiode for each frame when the imaging element is driven under the conditions shown in FIG. It is a figure explaining about.

  As shown in the figure, when the shooting timing of the frame and the fluctuation timing of the illumination light emission amount are not synchronized, the level of the imaging signal obtained from either the even pixel or the odd pixel is different for each frame. Therefore, the detection unit 204 detects the signal level of the second photodiode by dividing the signal level into even and odd pixels, and compares the detection results for each frame to determine whether the flashing light source is present or not. The blinking frequency can be recognized by the 207 microprocessor.

  FIG. 13 is a diagram showing the relationship between the exposure time set for the second photodiode when extracting the characteristics of illumination light, the exposure timing in an arbitrary line, and the illumination emission timing.

  As shown in the figure, when the shooting timing of the frame is synchronized with the fluctuation timing of the light emission amount of the illumination light, the exposure time is set to 1/120 second even for the even pixels whose exposure time is set to 1/100 second. The exposure amount of each frame does not fluctuate even for odd-numbered pixels. Therefore, if the exposure amount of each frame does not vary, it can be estimated that the variation in illumination emission intensity and the timing of frame shooting are synchronized. Can be estimated.

  The above is an example of the first illumination light feature extraction algorithm.

  Next, an example of the second illumination light feature extraction algorithm will be described.

  In the example of the algorithm described above, the signal level is assumed on the assumption that the light source is blinking. However, it is necessary to explain a little about the case where the light source is not blinking.

  When the light source does not blink, the exposure amount is simply proportional to the exposure time, so the exposure time is set to 1/100 second as in the case where the frame shooting timing and the illumination light emission amount fluctuation timing are synchronized. The exposure amount of each frame does not fluctuate even for odd-numbered pixels whose exposure times are set to 1/120 seconds.

  Therefore, it is impossible to distinguish between the case where the light source does not blink and the case where the photographing timing of the frame and the variation timing of the light emission amount of the illumination light are synchronized only by monitoring the fluctuation of the exposure amount of each frame.

  From the above, in the example of the second algorithm, attention is paid to the next part in the case where the shooting timing of the frame and the fluctuation timing of the light emission amount of the illumination light are synchronized to distinguish them.

  When shooting a general subject, the correlation between image signals obtained from adjacent pixels is extremely strong, so the intensity of light applied to a given pixel and the intensity of light applied to an adjacent pixel Are likely to be at the same level, and if the exposure times of adjacent pixels are equal, there is a high probability that the levels of the imaging signals obtained from the respective pixels will be equal.

  On the other hand, the level of the imaging signal obtained from each pixel is proportional to the product of the light intensity irradiated to the pixel and the exposure time. When the light intensity irradiated to the pixel is constant, the exposure time fluctuation ratio Accordingly, the level of the obtained imaging signal also varies in proportion.

  Thus, when the light emission intensity of the light source is constant, consider the case where the exposure times of the even and odd pixels of the second photodiode are set to 1/100 (even pixels) and 1/120 (odd pixels), respectively. The following are expected:

  When the signal level of an arbitrary even pixel is Dc (n) and the signal levels of odd pixels adjacent to the even pixel are Dc (n−1) and Dc (n + 1), Dc (n) is Dc (n− The probability of 1/5 and 6/5 times Dc (n + 1) is very high.

  On the other hand, in the case of FIG. 13 where the light emission intensity of the light source is not constant, the exposure times of the even and odd pixels of the second photodiode are respectively 1/100 (even pixel) and 1/120 (odd pixel). The following can be said when considering

  Focusing on the average brightness of the exposure timing of even-numbered pixels and odd-numbered pixels when exposure is performed under illumination whose brightness varies periodically, the even-numbered pixels are exposed for one cycle of brightness variation. Although it is constant in any line, in an odd-numbered pixel, since the exposure timing is not one brightness fluctuation cycle, the average luminance level varies depending on the line.

  In other words, this denies that the ratio of the signal level obtained from the adjacent even-numbered pixel and the odd-numbered pixel is highly proportional to the ratio of the exposure time, and the ratio of the signal level of the adjacent even-numbered pixel and odd-numbered pixel. If the deviation rate between the exposure time ratio and the exposure time is examined statistically, it is possible to confirm the presence or absence of blinking of the light source.

  Therefore, in the second illumination light feature extraction algorithm, the blinking light source feature is extracted by confirming this deviation rate.

  The above is an example of the second illumination light feature extraction algorithm.

  FIG. 14 summarizes the procedure for extracting the feature quantity of the light source in a flowchart. From the viewpoint of increasing detection accuracy, it is preferable to make a judgment by taking statistics of the fluctuation history of “whether there is a fluctuation in signal level in units of frames” over several frames, but it is quick even if the accuracy slightly decreases If it is desired to make a determination, the determination of “whether there is a change in signal level in units of frames” may be omitted.

  According to the above, in the present embodiment, it is possible to specify the presence / absence of a light source whose emission intensity periodically varies and the variation frequency.

  Next, exposure control of the image pickup apparatus of the present invention when a light source whose emission intensity periodically changes is detected will be described.

  In the imaging apparatus according to the present invention, when a light source whose emission intensity varies periodically is detected, image quality control is performed so as to suppress image quality degradation due to exposure spots caused by the influence of the emission intensity fluctuation as much as possible.

  As a means for suppressing image quality deterioration due to exposure spots, a method based on exposure time management and a method of predicting exposure spots and correcting the level of the imaging signal so as to cancel the exposure spots are adopted.

  Hereinafter, specific means for suppressing image quality degradation will be described.

  First of all, it is a method based on the management of exposure time. In the explanation of the procedure for extracting the feature quantity of illumination, if the exposure time of the image sensor is matched to the light emission intensity fluctuation cycle, it depends on the timing of signal readout. It is already clear that an image signal corresponding to the subject can be obtained.

  Therefore, as management of the exposure time, the exposure time of the first photodiode is set so as to match the integral multiple of the light emission intensity fluctuation period of the light source whose light emission intensity fluctuates, and the shutter speed priority exposure control is performed. To do.

  Next, it is a method of correcting the level of the imaging signal so that the exposure spots are predicted and offset the exposure spots, but this is effective when appropriate exposure cannot be realized by the exposure control with priority on the shutter speed. .

  A method of correcting the level of the imaging signal so as to predict the exposure spots and cancel out the exposure spots is a correction coefficient for each line calculated from the imaging signal obtained from the second photodiode by the correction gain generation unit 203. This is realized by multiplying the image pickup signal obtained from the first photodiode by a multiplier.

  The calculation of the correction coefficient utilizes the property that the level of the imaging signal obtained from the photodiode is proportional to the exposure time if the intensity of the light applied to the photodiode is the same.

  In order to make use of the above property, the even number pixels of the second photodiode have the same exposure time as the fluctuation period of the emission intensity of the illumination light, and the odd number pixels have the first photodiode having the proper exposure time. The same exposure time as the exposure time set for the first photodiode is set.

  An image pickup signal is read from the image pickup device for each line. At this time, first, a signal indicating the black reference of each line, then an image pickup signal from the second photodiode, and finally from the first photodiode. In this embodiment, the description is continued on the premise that the signals are read in this ideal order.

The correction coefficient is calculated as follows. A signal level serving as a black reference for a line to be corrected by the correction gain generation unit 203 is detected. By subtracting the black reference signal level from the imaging signal output levels of the even and odd pixels of the second photodiode, an imaging signal component generated by exposure is extracted. One or more signal level ratios of the even and odd pixels that are present on the same line and are adjacent to each other are sampled, and the average value Rm is calculated by Equation 1.
(Expression 1) Rm = Σ {(signal level of even pixel) ÷ (signal level of odd pixel)} ÷ (number of samples))
The ratio of the exposure time of even pixels and odd pixels is calculated by Equation 2.
(Expression 2) Rex = (exposure time of even pixels) / (exposure time of odd pixels)
The correction coefficient α calculated at this time is calculated by Equation 3. (Equation 3) α = Rm ÷ Rex
The above is the calculation of the correction coefficient.

  The calculated correction value α is immediately used as a correction coefficient for the imaging signal output from the first photodiode on the same line, and the level of the imaging signal can be corrected so as to cancel out the exposure spots.

  From the above, this imaging apparatus uses an imaging device including a photodiode for extracting a flicker feature amount and a photodiode for photographing the subject, and exposure control for extracting the flicker feature amount and the subject. Exposure control for shooting can be performed in parallel, and the imaging device can be controlled and corrected in real time so as to suppress the occurrence of flicker based on the extracted flicker feature quantity. is there.

  It can be used in an imaging apparatus, and is useful for efficiently detecting the occurrence of exposure spots that may occur due to the influence of a light source such as a fluorescent lamp that blinks periodically, and to suppress it with high accuracy.

DESCRIPTION OF SYMBOLS 101 Image pick-up lens 102 Image pick-up element 103 Image signal processor 104 Image compression / expansion processor 105 Storage device 106 User interface 107 Microprocessor 108 Display device 201 Line delay 202 Noise filter 203 Correction gain generation part 204 Detection part 205 Multiplier 206 Imaging signal processing part 207 Microprocessor interface 301 First photodiode group 302 Second photodiode group 303 Pixel amplifier group 304 Signal readout switch group 305 First charge discharging switch group 306 Second charge discharging switch group 307 Third charge discharging switch Group 308 Column selection switch group 309 Switch opening / closing means 310 Analog gain adjustment amplifier 311 CDS and ADC circuit 312 Microprocessor Interface 313 Setting register 501 of each part Pixel 502 Control line 503 for controlling first charge discharge switch group Control line 504 for controlling signal readout switch group Signal line 505 for reading imaging signal Forming column selection switch Electronic switch

Claims (4)

An imaging lens that collects light and forms an image of the subject on the light receiving surface of the imaging device;
An imaging device having a first photodiode group for photographing a subject and a second photodiode group for detecting a variation in the light emission intensity of the light source;
Image signal processing means for generating an image signal from an imaging signal obtained from the first photodiode group;
Detecting means for detecting a light emission pattern of a light source from an imaging signal obtained from the second photodiode group;
A microprocessor that controls the imaging lens, the imaging device, the image signal processing means, and the detection means;
Have
A sensor for detecting the light emission pattern of the light source and photographing the subject by allowing the first photodiode group and the second photodiode group to set individual exposure times for the image sensor. To drive together,
An imaging apparatus characterized by the above. The imaging apparatus according to claim 1,
The imaging device is configured such that two or more different exposure times can be set for the second photodiode group arranged on the same line, so that the second photodiode has two or more exposures. An imaging apparatus capable of detecting the presence or absence of a flashing light source whose light emission intensity varies periodically by setting time and monitoring the signal level output from the second photodiode group. The imaging apparatus according to claim 1 or 2,
By detecting a blinking light source whose emission intensity varies periodically and specifying the blinking cycle, and controlling the exposure time of an integral multiple of the identified blinking cycle to be set in the first photodiode, An image pickup apparatus that prevents image quality deterioration caused by exposure spots. The imaging apparatus according to any one of claims 1 to 3,
For the second photodiode of the image sensor, an exposure time set for the first photodiode and a blinking cycle of the blinking light source whose emission intensity periodically changes detected by the imaging device are set. And predicting the exposure spot of each line from the imaging signal obtained from the photodiode and the exposure time set in the second photodiode, calculating a coefficient for suppressing the exposure spot, and calculating the calculated coefficient An image pickup apparatus that suppresses image quality deterioration due to exposure spots caused by blinking of a light source by applying correction to an image pickup signal output from a first photodiode.

Images