CN105450950B - Unmanned plane video jitter removing method - Google Patents

Abstract

The invention discloses a method for deshaking a UAV aerial video, comprising step 1: preprocessing of road aerial images; step 2: extracting straight lines based on aerial images; step 3: calculating straight line direction histograms; step 4: correcting and removing aerial images Shaking; the present invention utilizes the characteristic that most of the straight lines extracted by UAV aerial photography road image detection are parallel to the road direction, establishes a straight line relative angle histogram, and can obtain the angle of the road in the aerial photography image by detecting the angle corresponding to the maximum peak point of the histogram Direction, rotate the image according to the angle to adjust the road to the horizontal direction, and realize the deshaking of the aerial image. Based on the image processing technology, the invention realizes the deshaking of the road aerial image of the UAV by intelligently sensing the road direction and rotating the aerial image.

Description

Method for Deshaking UAV Aerial Video

technical field

The invention belongs to the technical field of image processing, and relates to a method for deshaking aerial video of an unmanned aerial vehicle, in particular to a straight line extraction method based on image edge detection and Hough transform, a road direction detection method based on a straight line angle histogram and a method based on A correction method for images rotated in the direction of the road to achieve the purpose of image deshaking.

Background technique

UAVs are widely used in surveying and mapping, aerial photography, traffic monitoring and other fields because of their advantages such as strong mobility, wide field of vision, and flight routes not restricted by terrain. The application of two-axis and three-axis stabilized gimbals eliminates the problem of image jitter caused by drone attitude adjustment and external conditions (gusts), but in many application scenarios, such as UAV highway traffic monitoring, due to the lack of The human-machine does not have the ability to intelligently identify the road, causing the angle between the flying direction of the UAV and the direction of the road to change all the time, thus causing the problem of road jitter in the aerial image. , or the extraction of traffic parameters based on image processing methods have brought great obstacles. Therefore, it is particularly important to invent an intelligent road aerial image deshaking method.

At present, the image deshaking methods of aerial images mainly include mechanical deshaking and registration method deshaking. Hardware deshaking means that the image acquisition equipment is installed on the mechanical gimbal with self-stabilizing function. This method eliminates the image jitter caused by the attitude adjustment of the drone and external conditions (gusts) of the aerial image. The image registration method obtains the motion of the image background by tracking the feature points in the image, so as to eliminate the shaking through the affine transformation method. This type of shaking needs to select the basic registration frame in advance. In the actual UAV traffic monitoring application, the user is interested in the road area, and what needs to be obtained is a stable road image. For this application scenario, because the existing de-shaking method cannot perceive the road area, it cannot be targeted Eliminate the shaking problem of road aerial photography images.

Contents of the invention

Aiming at the problem that the existing image de-shaking methods cannot intelligently eliminate the road jitter in the road aerial image of the UAV, the present invention proposes an image de-shaking method based on the road direction histogram, and adopts the method based on the road direction histogram to de-shake the road aerial image. shake. The invention is a method for deshaking a road aerial image based on a road direction histogram. Firstly, the road direction is detected and extracted through the road direction histogram, and then the image is rotated based on the road direction to realize deshaking of the road aerial image. The image de-shaking method of the present invention can intelligently sense the direction of the road, thereby realizing the image de-shaking that cannot be realized by the existing method.

The present invention uses a brand-new research starting point to propose a method for deshaking road aerial photography images that can be generally applied, and is realized through the following steps:

Step 1: Road aerial image preprocessing

Deframe the drone aerial video, obtain a single frame RGB color road aerial image, and convert the RGB color image into a grayscale image;

Step 2: Extract straight lines based on aerial images

Use the Canny edge detection operator to process the grayscale image in the previous step to obtain the edge contour map, and detect the edge contour map based on the Hough transform to obtain a straight line;

Step 3: Compute the line direction histogram

Calculate the angle of the line detected in the previous step, and then calculate the line relative angle histogram. The angle corresponding to the largest peak point in the histogram is extracted, and the angle is the direction of the road in the aerial image of the frame.

Step 4: Aerial image correction and deshaking

By rotating the aerial image of the road clockwise by the angle obtained in the previous step, the roads facing different directions in the aerial image can be adjusted to the horizontal direction consistently, so as to realize the deshaking of the aerial image of the road.

The advantages of the present invention are:

(1) The present invention utilizes the characteristic that most of the straight lines extracted from the aerial photography road image detection of the UAV are parallel to the road direction, establishes a straight line relative angle histogram, and can obtain the road in the aerial photography image by detecting the angle corresponding to the maximum peak point of the histogram Rotate the image according to the angle to adjust the road to the horizontal direction to achieve deshaking of the aerial image;

(2) The present invention is based on image processing technology, by intelligently sensing the road direction and rotating the aerial image, thereby realizing the deshaking of the road aerial image of the UAV;

(3) The present invention can be adapted to deshake aerial images in various road scenes, has good robustness, fast operation speed and does not require external data support (such as GIS map data), and has multi-point innovation.

Description of drawings

Fig. 1 is the grayscale image of the aerial image;

Figure 2 is an edge contour map based on Canny edge detection operator detection;

Fig. 3 is a straight line diagram based on Hough transform detection;

Fig. 4 is a definition diagram of an image coordinate system and a straight line angle direction;

Fig. 5 is the relative histogram of straight line direction;

Figure 6 is a horizontal correction map based on road direction;

Fig. 7 is a flow chart of the method of the present invention.

Detailed ways

The present invention will be further described in detail with reference to the accompanying drawings and embodiments.

The present invention provides a method for de-shaking aerial images based on road direction histograms. The de-shaking method first preprocesses the extracted road aerial images and converts them into grayscale images; then obtains grayscale images based on Canny edge detection operator detection. Edge contour map, and then perform Hough transform on the edge contour map to detect straight lines; calculate the angle of the detected straight line and calculate the relative angle histogram of the straight line direction, extract the angle of the straight line corresponding to the maximum peak value of the relative angle histogram, and obtain Road direction; then based on the detected road direction, the original aerial image is rotated instantaneously, and the road can be rotated to a horizontal direction to achieve deshaking of the aerial image. The above-mentioned aerial image deshaking method based on the road direction histogram, the process flow is shown in Figure 7, and the specific processing steps are as follows:

Step 1: Road aerial image preprocessing

Deframe the road aerial video to obtain RGB color images, and convert the RGB color images into grayscale images, as shown in Figure 1.

Step 2: Extract straight lines based on aerial images

After the grayscale image is obtained, the grayscale image is processed by the Canny edge detection operator to obtain a binary edge contour map, as shown in Figure 2, and the Hough transform is performed on the contour map in Figure 2 to detect and obtain The straight line graph is shown in Figure 3.

Step 3: Compute the line direction histogram

Calculate the angle of each straight line in Figure 3, where the definition of the straight line angle is shown in Figure 4, where O(0,0) is the origin of the image pixel coordinates, with O(0,0) as the starting point, and the right direction is the image's The column axis, down to the row axis of the image. For any straight line i in Figure 3, its two endpoints are P1 and P2 respectively, where the pixel coordinates of P1 and P2 are (c _{i_1} , r _{i_1} ) and (c _{i_2} , r _{i_2} ), is the angle between the straight line i and the horizontal direction, Indicates the angle through which a horizontal straight line is rotated counterclockwise to be parallel to straight line i, the angle of any straight line i The calculation method of is shown in the following formula (1):

in, is an integer, it is rounded up when calculating, and

Based on the calculated line angle, calculate the relative angle histogram, the detailed steps are as follows:

(1) Confirm the number n of straight lines detected in Fig. 3;

(2) Set 180 small intervals θ ₁ to θ ₁₈₀ , where: the intervals are: θ ₁ =[0°,1°),θ ₂ =[1°,2°),...,θ _i =[(i- 1)°,i°),...,θ ₁₈₀ ＝[179°,180°);

(3) For the n straight lines detected in Figure 3 (n is confirmed in step (1)), the angles of the straight lines are counted. If there are m straight lines with the angle θ _i , the number of times the angle θ _i appears is considered is m times, here h(θ _i ) represents the number of times that the straight line angle θ _i appears, that is, the frequency.

(4) Perform normalization processing on the frequency h(θ _i ) of the statistical line angles in step (3), and calculate the relative frequency H(θ _i ) of each line angle θ _i . The calculation method is H(θ i _i )=h(θ _i )/n. The purpose of frequency h(θ _i ) normalization is to simplify the calculation and reduce the value.

(5) Draw a relative linear angle histogram, as shown in Figure 5, the horizontal axis of the histogram represents the linear direction angle, and its definition domain is [0 °, 180 °), the vertical axis represents the relative frequency of the linear angle, and its value The domain is [0,1]. In the histogram shown in Figure 5, the height value of the straight line corresponding to each angle θ _i is the relative frequency value H(θ _i ) calculated in step (4), and the straight line with the largest height value in Figure 5 corresponds to The relative frequency value is called the maximum histogram peak.

(6) The maximum peak value H(θ _k ) in Figure 5 is obtained through the formula H(θ _k )=Max{H(θ ₁ ),H(θ ₂ ),…,H(θ _i ),…,H(θ ₁₈₀ ) }, where θ _k is the angle of the line corresponding to the maximum histogram peak.

The angle θ _k is the direction of the road in the frame image. The detection principle of the road direction is: in the aerial image, the contour lines of the road and surrounding structures are parallel to the direction of the road, so the angle with the highest relative frequency of the straight line angle in Fig. 3 is the direction of the road, and this principle is applicable to general All straight roads are applicable.

Step 4: Aerial image correction and deshaking

Rotating the road aerial image clockwise by θ _k degrees can correct the road in the image to the horizontal direction, and the corrected image is shown in Figure 6.

By repeating steps 1 to 4, the purpose of deshaking aerial video can be achieved based on road direction correction.

Claims (1)

1. An unmanned aerial vehicle aerial video jitter removal method comprises the following steps:

step 1: preprocessing of road aerial images

The unmanned aerial vehicle aerial video is unframed, a single-frame RGB color road aerial image is obtained, and the RGB color image is converted into a gray-scale image;

step 2: extracting straight lines based on aerial images

Acquiring a gray-scale image edge contour map by adopting a Canny edge detection operator, detecting the edge contour map based on Hough transform, and acquiring a straight line;

and step 3: calculating a straight-line direction histogram

Calculating the angle of the straight line, then calculating the histogram of the relative angle of the straight line, and extracting the angle theta corresponding to the maximum peak point in the histogram_kThe angle is the direction of the road in the frame of aerial image;

and 4, step 4: correction debouncing of aerial images

Rotating the road aerial image clockwise by an angle theta_kThe aerial images are adjusted to be in the horizontal direction in a consistent way towards different roads, so that the road aerial images are removed from shaking;

the step 3 specifically comprises the following steps:

let the upper left corner of the edge contour map be the origin O (0,0) of the pixel coordinates of the image, take O (0,0) as the starting point, the right side as the column coordinate axis of the image, and the down side as the row coordinate axis of the image, and let an arbitrary straight line i, whose two endpoints are P1 and P2, respectively, where the pixel coordinates of P1 and P2 are (c) respectively_{i_1},r_{i_1}) And (c)_{i_2},r_{i_2})，Is the included angle between the straight line i and the horizontal direction,indicating the angle rotated by rotating a horizontal straight line counterclockwise to be parallel to the straight line i, the angle of any straight line iThe calculation method of (2) is shown in the following formula (1):

wherein,is an integer, is rounded off when calculated, and

based on the calculated straight line angle, calculating a relative angle histogram, specifically:

(1) setting the number of the straight lines obtained in the step 2 as n;

(2) set 180 cell intervals theta₁～θ₁₈₀Wherein: the interval is as follows: theta₁＝[0°,1°),θ₂＝[1°,2°),…,θ_i＝[(i-1)°,i°),…,θ₁₈₀＝[179°,180°)；

(3) For n straight lines, counting the angle of the straight line, and if the angle is theta_iM straight lines of (a) are included, the angle theta is considered_iThe number of occurrences is m times, and h (theta) is used_i) Representing the angle theta of a straight line_iThe number of occurrences, i.e., the frequency;

(4) the frequency h (theta) of the linear angle obtained in the step (3)_i) Performing normalization processing to calculate each linear angle theta_iRelative frequency of occurrence H (θ)_i) The calculation method is H (theta)_i)＝h(θ_i)/n；

(5) Drawing a relative straight line angle histogram, wherein the horizontal axis of the histogram represents a straight line direction angle and has a definition domain of [0 degrees and 180 degrees ], and the vertical axis represents the relative frequency of the occurrence of the straight line angle and has a value domain of [0,1 ]; the relative frequency value corresponding to the straight line with the maximum height value in the relative angle histogram is the peak value of the histogram;

(6) maximum peak H (θ) of histogram_k) By the formula H (theta)_k)＝Max{H(θ₁),H(θ₂),…,H(θ_i),…,H(θ₁₈₀) Is calculated, wherein theta_kThe angle of the straight line corresponding to the peak value of the histogram;

angle theta_kI.e. the direction of the road in the frame image.