CN103377472B - For removing the method and system of attachment noise - Google Patents

Abstract

Methods and systems for removing clinging noise are disclosed. The attached noise detection method includes: selecting any frame in the 3D spatio-temporal image I(x, y, t) as a reference frame, and performing perspective transformation on other frames in the 3D spatio-temporal image I(x, y, t) to obtain The transformed three-dimensional space-time image I'(x, y, t); use the transformed three-dimensional space-time image I'(x, y, t) to model the static background image, and transform the transformed three-dimensional space-time image I' (x, y, t) is subtracted from the static background image obtained by modeling to obtain the three-dimensional difference image I _d (x, y, t); the three-dimensional difference image I _d (x, y, t) is binary value processing to obtain a binarized three-dimensional difference image I _d '(x, y, t), wherein the modeling error in the binarized three-dimensional difference image I _d '(x, y, t) has been is removed; and performing inverse perspective transformation on the binarized three-dimensional difference image I _d '(x, y, t) to remove the influence of the moving object, thereby detecting the attached noise in the video to be detected.

Description

Method and system for removing attached noise

technical field

The present invention relates to the field of image processing, and more particularly to a method and system for removing attached noise.

Background technique

At present, surveillance systems are widely used in various public places. In the outdoor environment, the noise attached to the camera protective lens, such as raindrops or mud spots, will have a great impact on the quality of the surveillance video. Under the influence of attached noise, the observed scene is difficult to be monitored.

Usually, the removal of adhesion noise is achieved by cleaning the camera protective lens. However, the camera protective glasses of most surveillance systems cannot be cleaned automatically, and are difficult to be cleaned manually. Another general approach to remove adhesion noise is to avoid the generation of adhesion noise. Generally, professional cameras have lens protective glasses or special anti-adhesion oil. However, this cannot completely avoid the generation of attachment noise. Therefore, it is necessary to deal with the attachment noise through digital processing technology. Of course, before removing the adhesion noise, some methods need to be used to detect the adhesion noise.

The existing noise detection methods are not suitable for the above situation, the reasons are as follows: 1) Most of the adhesion noise detection methods are aimed at the noise of fixed shape or texture, and the adhesion noise caused by the complex outdoor environment is not all Fixed shapes and textures. 2) Some attachment noise detection methods are specifically aimed at the rain or snow that is falling, but these methods are based on the assumption that the noise is a constant motion, but in fact the attachment noise may be stationary relative to the camera. 3) Some adhesion noise detection methods are based on the assumption that the exact movement of the camera is known or detect the adhesion noise under the condition that the movement of the camera is at the level of the imaging plane, but the movement of the camera is usually complicated in the actual situation. unknowable.

Contents of the invention

In view of the problems described above, the present invention provides a method and system for removing attached noise.

The adhesion noise detection method according to the embodiment of the present invention is used to perform adhesion noise detection on the video to be detected, wherein all frames in the video to be detected are arranged in time order to obtain a three-dimensional space-time image I(x, y, t), the The attached noise detection method includes: selecting any frame in the 3D spatio-temporal image I(x, y, t) as a reference frame, and performing perspective transformation on other frames in the 3D spatio-temporal image I(x, y, t) to obtain the transformed The transformed three-dimensional space-time image I'(x, y, t); use the transformed three-dimensional space-time image I'(x, y, t) to model the static background image, and transform the transformed three-dimensional space-time image I'( x, y, t) is subtracted from the static background image obtained by modeling to obtain a three-dimensional difference image I _d (x, y, t); binary value is performed on the three-dimensional difference image I _d (x, y, t) process to obtain a binarized three-dimensional difference image I _d '(x, y, t), wherein the modeling error in the binarized three-dimensional difference image I _d '(x, y, t) has been removing; and performing inverse perspective transformation on the binarized three-dimensional difference image I _d '(x, y, t) to remove the influence of the moving object, thereby detecting the attached noise in the video to be detected.

The adhesion noise detection system according to the embodiment of the present invention is used to perform adhesion noise detection on the video to be detected, wherein all frames in the video to be detected are arranged in time order to obtain a three-dimensional space-time image I(x, y, t), the The attachment noise detection system includes: a perspective transformation unit, which is used to select any frame in the three-dimensional space-time image I (x, y, t) as a reference frame, and perform perspective transformation, to obtain the transformed three-dimensional space-time image I'(x, y, t); the static background modeling unit is used to utilize the transformed three-dimensional space-time image I'(x, y, t) to perform static background image Modeling, and subtracting the transformed three-dimensional space-time image I'(x, y, t) from the static background image obtained by modeling to obtain a three-dimensional difference image I _d (x, y, t); modeling error The removal unit is used to binarize the three-dimensional difference image I _d (x, y, t) to obtain a binarized three-dimensional difference image I _d '(x, y, t), wherein, in the binary The modeling error in the transformed three-dimensional difference image I _d '(x, y, t) is removed _; ) to perform anti-perspective transformation to remove the influence of moving objects, so as to detect the attached noise in the video to be detected.

The attached noise detection method and system according to the embodiments of the present invention can automatically detect the noise attached to the camera under irregular camera movement conditions, and is very suitable for outdoor monitoring systems.

Description of drawings

The present invention can be better understood from the following description of specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

FIG. 1 is a block diagram illustrating an attachment noise detection system according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a method for detecting adhesion noise according to an embodiment of the present invention.

detailed description

Features and exemplary embodiments of various aspects of the invention will be described in detail below. The following description covers numerous specific details in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is only to provide a clearer understanding of the present invention by showing examples of the present invention. The present invention is by no means limited to any specific configuration and algorithm presented below, but covers any modification, replacement and improvement of related elements, components and algorithms without departing from the spirit of the present invention.

For the noise attached to the protective lens of the camera, the position of the noise in the image does not change when the direction of the camera changes. This is because the noise is attached to the surface of the camera's protective lens and moves with the camera. On the other hand, when the camera moves, the position of the static background and the position of the moving object will change. The attached noise detection system and method according to the embodiments of the present invention attempt to use the above characteristics of attached noise, static background, and moving object to perform attached noise detection on the video to be detected.

Fig. 1 shows a block diagram of an adhesion noise detection system according to an embodiment of the present invention. Fig. 2 shows a flowchart of a method for detecting adhesion noise according to an embodiment of the present invention. The system and method for detecting adhesion noise according to the embodiments of the present invention will be described in detail below with reference to FIG. 1 and FIG. 2 .

It should be noted that before the attachment noise detection method and system according to the embodiment of the present invention work, each frame F(x, y) in the video to be detected needs to be arranged in time order to obtain a three-dimensional space-time image I( x, y, t), as the input of the attachment noise detection method and system according to the embodiments of the present invention.

As shown in FIG. 1 , the attachment noise detection system according to the embodiment of the present invention includes a perspective transformation unit 102 , a static background modeling unit 104 , a modeling error removal unit 106 , and a moving object removal unit 108 . Among them, the perspective transformation unit 102 selects any frame in the three-dimensional space-time image I(x, y, t) as a reference frame, and performs perspective transformation on other frames in the three-dimensional space-time image I(x, y, t) to obtain the transformation The final three-dimensional spatio-temporal image I'(x, y, t) (that is, execute step S202). The static background modeling unit 104 uses the transformed three-dimensional space-time image I'(x, y, t) to model the static background image, and combines the transformed three-dimensional space-time image I'(x, y, t) with the modeling The resulting static background images are subtracted to obtain a three-dimensional difference image I _d (x, y, t) (that is, step S204 is performed). The modeling error removal unit 106 performs binarization processing on the three-dimensional difference image I _d (x, y, t) to obtain a binarized three-dimensional difference image I _d '(x, y, t), wherein, in two The modeling error in the valued three-dimensional difference image I _d '(x, y, t) is removed (that is, step S206 is performed). The moving object removal unit 108 removes the influence of the moving object by performing inverse perspective transformation on the binarized three-dimensional difference image I _d '(x, y, t), thereby detecting the attached noise in the video to be detected (that is, performing Step S208).

Next, the attachment noise detection process will be specifically described.

perspective transformation

Select any frame in the video to be detected as a reference frame (hereinafter also referred to as an R frame), and use the imaging plane of this frame as a reference imaging plane. Then, perspective transformation is performed on other frames in the video to be detected to project other frames on the reference imaging plane. For any target frame (hereinafter also referred to as i frame) in the video to be detected except the reference frame, the perspective transformation can be obtained by multiplying the perspective projection matrix between the target frame and the reference frame and the original coordinate plane of the target frame. accomplish.

Wherein, when the target frame and the reference frame are two adjacent frames in the three-dimensional space-time image I(x, y, t), the perspective projection matrix between the target frame and the reference frame can be estimated by an automatic image correction method, The specific steps are as follows: First, find the static points in the target frame and the reference frame by Speeded Up Robust Features (SURF), and use the K-Nearest Neighbor (KNN) matching algorithm to match these static points (ie , to find the matching points in the static points), and optimize the matching points with the Random Sampling Consensus Algorithm (RANSAC); then obtain the perspective projection matrix between the target frame and the reference frame by optimizing the back-projection error.

When the target frame and the reference frame are two far apart frames in the three-dimensional space-time image I(x, y, t), it is difficult to find the matching point. Therefore, an indirect perspective projection matrix estimation method is proposed here, that is, the global perspective projection matrix (the perspective projection matrix between any two frames) is obtained through the local perspective projection matrix (the perspective projection matrix between two adjacent frames in time) ). The calculation method is as follows:

${\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; =</span>\begin{array}{cc}\mathrm{<span\; class="notranslate">\; h</span>}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \_</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{}_{\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; R</span>}\\ \mathrm{<span\; class="notranslate">\; h</span>}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \_</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>& \mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; R</span>}\\ & \mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; R</span>}\end{array}$

in, is the identity matrix, H _{i_R} is the global perspective projection matrix between the target frame i and the reference frame R, H _{j_(j+1)} is the target frame i and the reference frame in the three-dimensional space-time image I(x, y, t) The local perspective photography matrix between j frame and (j+1) frame between R, and H _{j_(j-1)} is the target frame i and reference frame in the three-dimensional space-time image I(x, y, t) The local perspective projection matrix between frames j and (j-1) frames between R.

After performing perspective transformation on each frame in the 3D spatiotemporal image I(x, y, t), the transformed 3D spatiotemporal image I'(x, y, t) can be obtained.

static background modeling

Next, model the static background. According to the characteristics of the attached noise (that is, attached to the protective mirror of the camera, its trajectory is consistent with the trajectory of the camera), we can see that after perspective transformation, all static backgrounds are aligned along the time axis, while the trajectory of the attached noise is becomes a space curve.

Therefore, the attachment noise can be detected according to the difference between the real static background and the transformed three-dimensional space-time image I'(x, y, t). The modeling process of the static background is as follows: first, the transformed 3D spatiotemporal image I'(x, y, t) is regarded as composed of a series of pixel sequences along the time axis, and these sequences can be divided into two categories - unimodal Sequences and multimodal sequences. A single-modal sequence refers to a sequence in which the gray value of a pixel changes little, such as the sky, the ground, etc.; a multi-modal sequence refers to a sequence in which the gray value of a pixel changes sharply and frequently, such as an area where a moving target passes or canopy etc.

Here, the variation magnitude of a sequence of pixels is measured as σ is the variance of the gray value of the pixel sequence, and μ is the mean value of the gray value of the pixel sequence. The sequence of pixels composing the transformed 3D spatio-temporal image I'(x, y, t) can be classified into two categories using an unsupervised K-means clustering method. For a single-modal sequence, the median gray value can be considered as the real static background value; for a multi-modal sequence, it can be modeled by a background modeling method based on a mixed Gaussian model.

After obtaining the static background image, the three-dimensional difference image I _d (x, y, t) can be obtained by subtracting the transformed three-dimensional space-time image I'(x, y, t) from the static background image. Obviously, these differences are caused by modeling errors, moving objects, and attachment noise. In order to detect the attachment noise, it is necessary to remove the difference caused by the modeling error and the moving target respectively.

remove modeling errors

The modeling error can be removed by binarizing the three-dimensional difference image I _d (x, y, t). The specific operation is as follows: first, the three-dimensional difference image I _d (x, y, t) is regarded as a multi-frame image arranged along the time axis; then, for each frame image, an adaptive threshold binarization is performed. The strategy of adaptive threshold is: the area occupied by the moving target and attached noise in each frame is less than 15% of the whole frame area. After removing the modeling error, the remaining area becomes the potential attachment noise area.

remove moving target

The binarized three-dimensional difference image I _d '(x, y, t) is subjected to inverse perspective transformation, that is, each frame of image is projected onto its original imaging plane. At this point, it can be seen that all the attached noises have been aligned along the time axis. Then, along the time axis, by voting with the probability of occurrence of potential attachment noise, the area of attachment noise can be obtained. The voting strategy is: the area occupied by the attached noise is less than 10%.

The attached noise detection system and method according to the embodiments of the present invention can automatically detect the noise attached to the camera under the condition of irregular camera movement, and is very suitable for an outdoor monitoring system.

The present invention has been described above with reference to the specific embodiments of the present invention, but those skilled in the art will understand that various modifications, combinations and changes can be made to these specific embodiments without departing from the requirements set by the appended claims or their equivalents. The spirit and scope of the present invention defined by the material.

The steps can be performed by hardware or software as desired. Note that steps may be added to, removed from, or modified in the flowcharts presented in this specification without departing from the scope of the present invention. In general, a flowchart is only used to indicate one possible sequence of basic operations for implementing a function.

Embodiments of the present invention may utilize programmed general purpose digital computers, utilize application specific integrated circuits, programmable logic devices, field programmable gate arrays, optical, chemical, biological, quantum or nanoengineered systems, components and mechanisms accomplish. Generally speaking, the functions of the present invention can be realized by any means known in the art. Distributed or networked systems, components and circuits can be used. Communication or transfer of data may be wired, wireless or by any other means.

It will also be appreciated that one or more of the elements shown in the figures may be implemented in a more separate or integrated manner, or even removed or disabled in some cases, depending on the needs of a particular application. It is also within the spirit and scope of the present invention to implement a program or code storable in a machine-readable medium to allow a computer to perform any of the methods described above.

Furthermore, any signal arrows in the figures should be considered as illustrative only, and not restrictive, unless specifically indicated otherwise. Combinations of components or steps are also considered to have been recited when terms are foreseen to obscure the ability to separate or combine.

Claims (10)

1. A method of attaching noise detection, used for attaching noise detection to a video to be detected, wherein, all frames in the video to be detected are arranged in time order to obtain a three-dimensional spatio-temporal image I (x, y, t), the Attachment noise detection methods include: Select any frame in the three-dimensional space-time image I (x, y, t) as a reference frame, and perform perspective transformation on other frames in the three-dimensional space-time image I (x, y, t) to obtain the transformed Three-dimensional space-time image I'(x, y, t); Utilize the transformed three-dimensional space-time image I'(x, y, t) to model the static background image, and combine the transformed three-dimensional space-time image I'(x, y, t) with the modeling result The static background image of is subtracted to obtain the three-dimensional difference image I _d (x, y, t); Binarize the three-dimensional difference image I _d (x, y, t) to obtain a binarized three-dimensional difference image I _d '(x, y, t), wherein, in the binarization The modeling error in the three-dimensional difference image I _d '(x, y, t) of has been removed; and By performing inverse perspective transformation on the binarized three-dimensional difference image I _d '(x, y, t) to remove the influence of moving objects, the attached noise in the video to be detected is detected. 2. The attachment noise detection method according to claim 1, wherein the processing of carrying out perspective transformation to any target frame in other frames in the three-dimensional space-time image I (x, y, t) comprises: Obtaining perspective projection matrices between all adjacent frames between the target frame and the reference frame; obtaining a perspective photography matrix between the target frame and the reference frame using a perspective projection matrix between all adjacent frames between the target frame and the reference frame; and Multiplying the perspective photography matrix between the target frame and the reference frame by the original coordinate plane of the target frame, so as to realize the perspective transformation of the target frame. 3. attachment noise detection method according to claim 1, is characterized in that, utilizes the three-dimensional spatio-temporal image I ' (x, y, t) after the transformation to carry out the processing of modeling static background image to comprise: The transformed three-dimensional space-time image I'(x, y, t) is regarded as composed of multiple pixel sequences along the time axis, and the multiple pixel sequences are divided into single-modal sequences and multi-modal sequences two types; For any pixel sequence belonging to the unimodal sequence, using the median value of the gray value of the pixel sequence as the static background value corresponding to the pixel sequence; For any pixel sequence belonging to the multi-modal sequence, the pixel sequence is modeled by a background modeling method based on a mixed Gaussian model. 4. The adhesion noise detection method according to claim 1, wherein the binarization of the three-dimensional difference image I _d (x, y, t) is performed based on an adaptive threshold. 5. The adhesion noise detection method according to claim 1, characterized in that the probability of the potential adhesion noise region appearing along the time axis in the three-dimensional space image of the inverse perspective transformation result is voted to detect the adhesion noise region . 6. A system for attaching noise detection, used for attaching noise detection to a video to be detected, wherein all frames in the video to be detected are arranged in time order to obtain a three-dimensional space-time image I (x, y, t), the The attached noise detection system includes: A perspective transformation unit, configured to select any frame in the three-dimensional space-time image I (x, y, t) as a reference frame, and perform perspective transformation on other frames in the three-dimensional space-time image I (x, y, t) , to obtain the transformed three-dimensional space-time image I'(x, y, t); A static background modeling unit, configured to use the transformed three-dimensional space-time image I'(x, y, t) to model the static background image, and convert the transformed three-dimensional space-time image I'(x, y , t) is subtracted from the static background image obtained by modeling to obtain a three-dimensional difference image I _d (x, y, t); A modeling error removal unit, configured to perform binarization processing on the three-dimensional difference image I _d (x, y, t) to obtain a binarized three-dimensional difference image I _d '(x, y, t), Wherein, the modeling error is removed in the binarized three-dimensional difference image I _d '(x, y, t); and The moving object removal unit is configured to perform inverse perspective transformation on the binarized three-dimensional difference image I _d '(x, y, t) to remove the influence of the moving object, thereby detecting the attached noise in the video to be detected. 7. The attachment noise detection system according to claim 6, wherein the perspective transformation unit processes any target frame in other frames in the three-dimensional space-time image I (x, y, t) by the following processing Do a perspective transformation: Obtaining perspective projection matrices between all adjacent frames between the target frame and the reference frame; obtaining a perspective photography matrix between the target frame and the reference frame using a perspective projection matrix between all adjacent frames between the target frame and the reference frame; and Multiplying the perspective photography matrix between the target frame and the reference frame by the original coordinate plane of the target frame, so as to realize the perspective transformation of the target frame. 8. The attachment noise detection system according to claim 6, wherein the static background modeling unit models the static background image by the following processing: The transformed three-dimensional space-time image I'(x, y, t) is regarded as composed of multiple pixel sequences along the time axis, and the multiple pixel sequences are divided into single-modal sequences and multi-modal sequences two types; For any pixel sequence belonging to the unimodal sequence, using the median value of the gray value of the pixel sequence as the static background value corresponding to the pixel sequence; For any pixel sequence belonging to the multi-modal sequence, the pixel sequence is modeled by a background modeling method based on a mixed Gaussian model. 9. The adhesion noise detection system according to claim 6, wherein the modeling error removal unit is based on an adaptive threshold value for the three-dimensional difference image I _d (x, y, t) Binarization. 10. The adhesion noise detection system according to claim 6, characterized in that the probability of the potential adhesion noise region appearing along the time axis in the three-dimensional space image of the inverse perspective transformation is voted to detect the adhesion noise region .