JP5885384B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus capable of improving the image quality of an image taken by a camera or the like when the image is deteriorated due to fluctuations caused by heat.

In recent years, when a certain subject is photographed using a camera or the like, there are many image processing methods for improving the image quality degradation (restoring the image quality) so that the subject can be perceived satisfactorily when the photographed image is degraded. Proposed.
Related to this is the prior art (referred to as Prior Art 1) listed in Patent Document 1. Prior art 1 discloses a method for restoring image degradation using the blind deconvolution method for events such as camera shake that occurs during camera shooting. The blind deconvolution method is used to detect the amount of shaking caused by camera shake. It is shown that a good image free from camera shake can be obtained with a simple configuration without using a sensor that detects a physical quantity such as a gyro sensor.

JP 2001-333326 A

Masatoshi Okutomi (15 others), "Digital Image Processing", Foundation for Image Information Education, 2004, p.202-204, p.243-245 Jean-Yves Bouguet, "Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the algorithm", OpenCV Reference Manual, [online], [October 8, 2010 search], Internet <URL: http: // trac. assembla.com/dilz_mgr/export/272/doc/ktl-tracking/algo_tracking.pdf>

According to the prior art 1, even when image quality is deteriorated due to camera shake or the like when photographing using a camera, the deterioration can be improved and a good image can be obtained. However, the image quality deterioration factors include not only camera shake but also various other factors such as events caused by natural atmospheric factors such as the sun flame. A hot flame is a phenomenon in which air at different temperatures mixes, causing a difference in the density of the air locally, and light is refracted at the boundary to make the object appear to fluctuate. There is a problem that correction cannot be performed in the conventional method 1 because there is fluctuation.
In view of the above-described problems, an object of the present invention is to provide an image processing apparatus that can improve image quality deterioration due to an event caused by natural atmospheric factors such as a hot flame and obtain a good image.

The present invention provides an image processing apparatus that processes video captured by a camera, a reference image estimation unit that estimates a reference image by exponentially moving an input image sequentially input from the imaging apparatus in a time domain, Based on the low-pass filter that gives the input image the same blur as the blur on the reference image, and the input image and the reference image given the blur, the local fluctuation of the input image is calculated and the An optical flow calculation unit for detecting a motion vector, and correcting the input image by moving pixels of the input image by the motion vector whose sign is inverted so as to remove local fluctuation of the input image The low-pass filter is a Gaussian filter represented by a Gaussian function having a variance substantially equal to the variance of the motion vector. Characterized in that it is a filter.
The optical flow calculation unit may calculate the optical flow using at least one of a block matching method and a gradient method.
The present invention is an image processing program that causes a computer to process video captured by a camera, and which estimates a reference image by exponentially moving an input image sequentially input from an imaging device in the time domain. An estimation step, a filtering step for applying a low-pass filter that gives the input image the same blur as the blur on the reference image, and based on the input image and the reference image that are given the blur, An optical flow calculation step of detecting a motion vector due to a hot flame by calculating a dynamic fluctuation, and moving the pixels of the input image by the motion vector with the sign inverted so as to remove the local fluctuation of the input image by a motion compensation step of correcting the input image, it was executed in the computer, the low pass filter Characterized in that it is a Gaussian filter represented by substantially equal variance Gaussian function of the variance of the motion vector.

  According to the image processing apparatus of the present invention, it is possible to correct the fluctuation of the image due to the heat flame that could not be corrected by the conventional method 1, and to provide a good image.

1 is a configuration diagram of a monitoring apparatus to which an image processing apparatus of the present invention is applied. It is a figure explaining an example of the internal block diagram of the image processing apparatus of this invention. It is a figure explaining an example of the operation | movement flow of the image processing apparatus of this invention. It is a figure explaining operation | movement of the reference image estimation part of this invention. It is a figure explaining the calculation result of the optical flow calculation part of this invention. It is a figure explaining the calculation method of the filtration characteristic of a low pass filter in the optical flow calculation part of the present invention.

  Embodiments according to the present invention will be described with reference to the drawings.

As a specific embodiment to which the present invention is applied, a monitoring apparatus using an imaging apparatus such as a camera will be described below as an example.
FIG. 1 is a functional block diagram showing the configuration of a monitoring apparatus 100 to which the present invention is applied. The monitoring device 100 includes an imaging device 101 such as a surveillance camera, an image processing device 102 that inputs image data from the imaging device 101 and executes image processing of the present invention, image data (input image) input from the imaging device 101, An image output unit 103 that inputs image data (reference image, corrected image) output from the image processing apparatus 102 and outputs the image data is provided. The image output unit 103 can be configured by a monitor device including a video display unit. Although not shown, the image processing apparatus 102 includes a microcomputer, executes various programs by executing programs stored in the memory, and implements functions described below.

  FIG. 2 shows an example of an internal block diagram of the image processing apparatus 102 of the present embodiment. The image processing apparatus 102 according to the present exemplary embodiment includes a reference image estimation unit 1, an optical flow calculation unit 2, and a motion correction unit 3. FIG. 3 shows an example of a flowchart of the operation of the image processing apparatus 102 of the present embodiment. The operation of the reference image estimation unit 1 is executed in the reference image estimation step S1, the operation of the optical flow calculation unit 2 is executed in the optical flow calculation step S2, and the operation of the motion correction unit 3 is executed in the motion correction step S3. Hereinafter, the operation of each processing unit will be described.

First, in the reference image estimation step S1, the reference image estimation unit 1 estimates an image (reference image) having no motion from an input image sequentially input from the imaging device 101 such as a camera. The operation will be described with reference to FIG. An image F _t (x, y) input at a certain time t is input to the gain block 5 and multiplied by r, while a reference image R _t (x, y) output from the delay unit 7 at a certain time t. y) is multiplied by 1-r in the gain block 6, and the output values of the gain block 5 and the gain block 6 are weighted and added by the adder 8. The output value of the adder 8 is input to the delay unit 7, and the delay unit 7 updates and outputs the data as the reference image R _{t + 1} (x, y) in the next frame (time t + 1). Here, r in the gain blocks 5 and 6 is a weight, and is a constant that takes a value between 0 and 1. The delay unit 7 holds an image for a time interval of one frame. Therefore, the reference image calculated by the reference image estimation unit 1 is expressed as in Expression (1).

R _{t + 1} (x, y) = r × F _t (x, y) + (1−r) × R _t (x, y)
(1)

  Equation (1) is called Exponential Moving Average, and has the effect of reducing (averaging) high-frequency components of input images that are sequentially input. Here, when the weight r is, for example, r = 0.01 (1/100), the input image finally weighted and added to the reference image has a weight equivalent to that obtained by averaging 100 frames of the input image. Imposed. That is, when the reference image estimation unit 1 is executed for each input image that is sequentially input, the input image having the number of frames indicated by the weight r is converged as an averaged image. This image is referred to as a reference image, and the reference image is an image having no motion by removing moving objects, fluctuations, and the like in the input image.

  Next, the optical flow calculation unit 2 compares the input image with the reference image and calculates an optical flow in the optical flow calculation step S2. The optical flow represents a local motion distribution between two images, for example, as shown in FIG. FIG. 5 is a diagram for explaining the calculation result of the optical flow calculation unit 2, FIG. 5 (a) shows a reference image, FIG. 5 (b) shows an input image, and FIG. 5 (c) shows an optical flow. Yes. As described above, the reference image is an image that does not move, and the input image is an image that is fluctuating due to a hot flame or the like. Therefore, the optical flows of the two images are locally random as shown in FIG. Distribution with a lot of movement.

  As a method of calculating the optical flow in FIG. 5C, the local block pattern (template) of the reference image (FIG. 5A) is located at which position in the input image (FIG. 5B). A block matching method based on a process (referred to as a template matching process) for searching whether there is a local area (for example, 10 pixels vertically and 10 pixels horizontally) of two images or the luminance of a certain pixel of interest There is a gradient method for calculating a motion vector from the difference between the spatial gradient of the level and the luminance level of the image of two frames. By applying these calculations to a target pixel (for example, a pixel at a position of 10 pixels vertically and horizontally) on the input image (FIG. 5B), the entire input image (FIG. 5B) is localized. A motion (motion vector) distribution is obtained. These methods are widely used in the technical field. For example, for the template matching method, pages 202 to 204 of Non-Patent Document 1, and for the optical flow, pages 243 to 245 of Non-Patent Document 1, and non-patents. Document 2 describes in detail.

  That is, according to the optical flow calculation unit 2, it is possible to calculate a local motion between two images of the input image (FIG. 5B) and the reference image (FIG. 5A). That is, it is possible to estimate how much fluctuation (motion vector) is generated by the hot flame in the target pixel on the input image (FIG. 5B).

Next, the motion correction unit 3 corrects the fluctuation due to the heat flame using the motion vector obtained by the optical flow calculation unit 2 in the motion correction step S3. For example, when the motion vector at a certain pixel of interest in the input image (FIG. 5B) is v = (vx, vy), the local region centered on the pixel of interest in the input image (FIG. 5B) For example, a partial image that is moved by (−vx, −vy) is generated with respect to the partial image of 10 pixels in the above example and 10 pixels in the horizontal direction. That is, the moved partial image becomes a partial image in which the amount of movement due to the hot flame is corrected. This partial image generation process is performed for all the target pixels, and a superposition of all the partial images is output as a corrected image.
When the motion vector is represented by a real number, an appropriate interpolation process may be performed in the process of moving the partial image described above.

  By doing as described above, a reference image (FIG. 5 (a)) having no motion is estimated from an input image (FIG. 5 (b)) sequentially input, and the input image (FIG. 5 (b)) and the reference image The optical flow (FIG. 5C) is calculated based on (FIG. 5A) to calculate the local fluctuation (motion vector) of the input image (FIG. 5B), and based on the motion vector. Then, a partial image of the input image (FIG. 5B) is moved so as to cancel the fluctuation of the heat flame, and a corrected image is generated by removing the fluctuation of the heat flame existing in the input image (FIG. 5B). Can do.

  The present embodiment improves the accuracy of the calculation result of the optical flow calculation unit 2 in the image processing apparatus shown in the first embodiment. Since the configuration of the processing unit of the present embodiment is the same as that of FIG. 2, the description of the configuration is omitted.

  As described above, the reference image estimation unit 1 averages the sequentially input images by the exponential moving average to obtain a reference image without motion. As a result, in addition to the fluctuation of the heat flame existing in the input image, the movement of the moving object and the moving background component is removed. However, a phenomenon that constantly fluctuates like a hot flame appears as a blur on the reference image by averaging the input images. On the other hand, the optical flow calculation using the gradient method calculates a motion vector from the difference between the spatial gradient of the luminance level of a certain pixel of interest and the luminance level of the image of two frames. Are superimposed or there is a difference in luminance level between the input image and the reference image, there is a problem that the accuracy of the calculated optical flow is lowered.

  In order to solve this problem, there is a method in which a high-pass filter that reduces blur on the reference image is applied to the reference image, or a low-pass filter that superimposes the same blur on the input image is applied to the input image. is there. In the present embodiment, considering that noise is mixed in the input image, a low-pass filter (for example, a Gaussian filter) is applied to the input image input to the optical flow calculation unit 2.

Next, a method for calculating the filtration characteristics of the low-pass filter (Gaussian filter) will be described with reference to FIG. FIG. 6 shows a histogram representing the amount of motion vector on the horizontal axis and the frequency of motion vector generation on the vertical axis. The motion vector amount is at least one of the elements vx or vy of the motion vector v at each point of interest, calculated in a predetermined number of frames in the past (for example, 10 frames) in the optical flow calculation unit 2 (the elements are real numbers). If it is represented, it is converted to an integer by a method such as rounding off or truncating to a small number), and this is set as the motion amount d. The occurrence frequency of the motion vector is the number of motion vectors taking the value of the motion amount d of the element. This histogram represents the distribution of the amount of fluctuation caused by the heat, and the frequency of occurrence decreases as the fluctuation increases, and thus the distribution is mountain-shaped around zero. Next, using this histogram, the variance σ ² (standard deviation σ) of the shake distribution is calculated. σ ² is expressed by Expression (2).

σ ² = Σd ² × h (d) / N (2)

Here, d represents the amount of movement, h (d) represents the frequency of occurrence of the amount of movement d, and N represents the total number of target elements. The Gaussian filter is realized by a smoothing filter whose weight is a Gaussian function represented by the variance σ ² . This Gaussian function is expressed by Equation (3).

h _g (x, y) = 1 / 2πσ ² × exp (− (x ² + y ² ) / 2σ ² )
(3)

That is, by using the Gaussian filter described above, the difference in luminance level due to blurring of the input image and the reference image is reduced, and the optical flow is calculated by reducing the noise of the input image, so the accuracy of the optical flow is improved. To do.

  The present embodiment improves the correction accuracy of the motion correction unit 3 in the image processing apparatuses shown in the first and second embodiments. Since the configuration of the processing unit of the present embodiment is the same as that of FIG. 2, the description of the configuration is omitted.

  As described above, the optical flow calculation unit 2 calculates the local similarity between the input image and the reference image, the spatial gradient of the luminance level of the input image at the target pixel, and the difference between the luminance levels of the input image and the reference image. Calculate the motion vector. However, even if an object that moves to the input image exists due to the effect of averaging the reference image, the object does not exist on the reference image. Therefore, the local similarity decreases in the area where the object exists, and the difference between the luminance levels of the input image and the reference image becomes large, and an accurate motion vector cannot be calculated in the area where the object exists. is there.

  In order to solve this problem, in this embodiment, the reliability of the motion vector is evaluated based on the local similarity between the input image and the reference image. The similarity is, for example, the sum of absolute values of differences in luminance level between the input image and the reference image (Sum of Absoluted Difference), or the normalized cross-correlation between the input image and the reference image. ). The degree of difference and the degree of similarity are described on pages 203 to 204 of Non-Patent Document 1.

  Next, when the reliability is low (for example, if the degree of difference is the difference, the sum of the absolute values of the differences is equal to or greater than a predetermined threshold (for example, 64 averaged by the number of pixels), When the correlation is a predetermined threshold value (for example, when 0.7 or less), it is assumed that there is a moving object on the input image, and the motion vector is set to 0 (v = (0, 0)) Setting the motion vector to 0 means that the partial image of the input image is directly reflected in the corrected image, that is, the moving object in the input image is output to the corrected image as it is. The

  In this way, even when there is a moving object in the input image and the motion vector cannot be accurately calculated, the object can be correctly output to the corrected image.

The characteristics of the above embodiment are summarized as follows.
(1) An image processing apparatus according to the present invention has no movement based on input images sequentially input from the imaging apparatus in an imaging apparatus that captures a target region and an image processing apparatus that processes video captured by the imaging apparatus. A reference image estimation unit for estimating a reference image; an optical flow calculation unit for calculating a local fluctuation of the input image based on the input image and the reference image; And a motion correction unit that corrects the input image based on the distribution of motion caused by the hot flame detected by the optical flow calculation unit so as to remove local fluctuations.
(2) Further, the reference image estimation unit of (1) may sequentially update the reference image by weighted addition of the current input image and the current reference image.
(3) Further, the optical flow calculation unit of (1) and (2) may calculate the optical flow using at least one of a block matching method and a gradient method.
(4) Further, in the optical flow calculation unit of (3), when the optical flow calculation unit calculates an optical flow using a gradient method, the optical flow calculation unit calculates the input image before calculating the optical flow. A filter (for example, a Gaussian filter) that reduces the high-frequency component of may be executed.
(5) Further, the filtering characteristics of the filter of the optical flow calculation unit in (4) may be calculated based on a motion distribution detected by the optical flow calculation unit based on a predetermined number of frames in the past. Good.
(6) Further, the optical flow calculation unit of (1) to (5) calculates the reliability of each motion vector of the detected motion distribution based on the local similarity between the input image and the reference image, The motion distribution may be corrected based on the reliability.
(7) The image processing apparatuses (1) to (6) may be provided in a monitoring apparatus that inputs and processes image data from the imaging apparatus.

  The present invention is not limited to the monitoring device described above, and can be widely used for preventing image deterioration due to the heat flame in various imaging devices.

DESCRIPTION OF SYMBOLS 1 ... Reference image estimation part 2 ... Optical flow calculation part 3 ... Motion correction part 5, 6 ... Gain block 7 ... Delay part 8 ... Adder 100 ... Monitoring apparatus 101 ... Imaging device 102 ... Image processing device 103 ... Image output unit S1 ... Reference image estimation step S2 ... Optical flow calculation step S3 ... Motion correction step

Claims (3)

In an image processing device that processes video captured by a camera,
A reference image estimation unit that estimates a reference image by exponentially moving an input image sequentially input from the imaging device in a time domain;
A low pass filter that gives the input image the same blur as the blur on the reference image;
An optical flow calculation unit that calculates a local fluctuation of the input image based on the input image given the blur and the reference image, and detects a motion vector due to a hot flame;
A motion correction unit that corrects the input image by moving pixels of the input image by the motion vector with the sign inverted so as to remove local fluctuations of the input image ;
The image processing apparatus , wherein the low-pass filter is a Gaussian filter expressed by a Gaussian function having a variance substantially equal to a variance of the motion vector . The image processing apparatus according to claim 1, wherein the optical flow calculation unit calculates an optical flow using at least one of a block matching method and a gradient method . An image processing program that causes a computer to process video captured by a camera ,
A reference image estimation step of estimating a reference image by exponentially moving an input image sequentially input from an imaging device in a time domain;
A filtering step of applying a low-pass filter that gives the input image the same blur as the blur on the reference image;
An optical flow calculation step of calculating a local fluctuation of the input image based on the input image given the blur and the reference image, and detecting a motion vector due to a hot flame,
A motion correction step of correcting the input image by moving pixels of the input image according to the motion vector whose sign is inverted so as to remove local fluctuations of the input image;
To the computer,
The low-pass filter is a Gaussian filter expressed by a Gaussian function having a variance substantially equal to the variance of the motion vector.

Images