CN110514212A - A smart car map landmark positioning method integrating monocular vision and differential GNSS - Google Patents

Abstract

The invention discloses a method for locating a landmark on a map of an intelligent vehicle fused with monocular vision and differential GNSS, comprising five steps of sensor synchronization, data preprocessing, landmark detection and landmark feature point extraction, landmark tracking and positioning, and landmark descriptor calculation. The present invention utilizes the sensor to synchronously obtain the spatial coordinates of the vehicle and the camera corresponding to the image; detects the image landmarks and extracts the landmark feature points through data processing; uses the optical flow method to track the image feature points; and extracts multiple feature points from the same landmark is tracked and eventually averaged to calculate the position of this landmark. After the landmark positioning is successful, the features of the last detected landmark are extracted to uniquely describe the landmark, and the landmark position, type and feature description are stored in the database.

Description

A smart car map landmark positioning method integrating monocular vision and differential GNSS

technical field

The invention relates to the field of unmanned driving map positioning, in particular to a smart car map landmark positioning method that integrates monocular vision and differential GNSS.

Background technique

With the vigorous development of location-based services and the increasing number of large buildings, people's demand for location-based services continues to increase, and fast and accurate positioning has become an urgent need.

At present, smart car maps are mainly divided into two methods: lidar mapping and visual mapping. Based on lidar mapping, using point cloud coordinates to obtain the spatial position of landmarks, this method has a large amount of calculation, has certain requirements for hardware, and has low real-time performance; the landmark positioning of smart car maps based on vision uses binocular cameras, and can be achieved through binocular matching. Directly obtain the spatial position of landmarks, but this method has low robustness and large error.

Contents of the invention

The object of the present invention is to, in view of the above problems, propose a kind of intelligent vehicle map landmark location method of fusion monocular vision and differential GNSS.

A method for locating landmarks on a smart car map fused with monocular vision and differential GNSS, comprising the steps of:

S1: The sensor synchronously obtains the vehicle and camera space coordinates corresponding to the image;

S2: Data processing detects image landmarks and extracts landmark feature points;

S3: Use the optical flow method to track the image feature points;

S4: Calculate and obtain the landmark position and describe the sub-calculation.

A method for locating landmarks on a smart car map that combines monocular vision and differential GNSS, S1 includes the following sub-steps:

S11: Build the acquisition system, install the camera and differential GNSS dual positioning antennas on the roof, make the camera and GNSS antennas on the same plane, the direction of the camera is the same as that of the dual antennas, and the distance between the camera and the positioning antennas is d;

S12: Use the time stamps of different sensor data to perform temporal nearest neighbor synchronization, and obtain a set of standardized data for each frame of image, which needs to meet:

e _i =(p _i , V _i )

Among them, P _i represents the position and attitude of the vehicle corresponding to each frame image, namely ( _xi , y _i , _zi , α _i , β _i , γ _i ), where ( _xi , y _i , _zi ) is the Camera coordinates, (α _i , β _i , γ _i ) are the three angles of the camera pose on the vehicle, V _i represents the image corresponding to the current pose;

S13: Use the camera distortion parameters to correct the image to obtain the corrected image, and use the position obtained by GNSS to obtain the camera position by using the relative position transformation between the sensors.

S2 includes the following sub-steps:

S21: Use a deep learning algorithm to detect landmarks in the image, and the detection result is the landmark type (ID) and the two-dimensional position in the image;

S22: Extracting ORB features from the image region where the landmark is detected;

The step S3 includes the following sub-steps:

S31: Judging whether a local area appearing in two consecutive frames of images I and J is the same target, it needs to meet:

I(x,y,t)=J(x',y',t+Δ)

Among them, all (x, y) are moved in one direction (d _x , d _y ), thus obtaining (x′, y′).

S32: The (x, y) point at time t is (x+d _x , y+d _y ) at time t+τ, so the problem of finding a match can be reduced to finding the minimum value of the following formula, which needs to be satisfied:

Among them, w _x and w _y respectively represent 1/2 of the W window, u _x and u _y represent the image coordinates of the points to be matched respectively. In order to get the best match, make ε the smallest, let the derivative of the above formula be 0, find the minimum value, and the obtained d is the tracking offset.

Said step S4 also includes the following sub-steps:

S41: Acquiring imaging points from time t to t+N (N+1) during landmark positioning;

S42: Use the coordinate system transformation relationship to obtain the image landmark position, and the transformation relationship satisfies:

z ₀ =Z*cosθ ₁

Among them, (x0, y0, z0) is the same landmark point observed by using multiple image frames (the position is known), and the position of this landmark point can be calculated;

S43: Extract multiple feature points for the same landmark to track, and finally take the average value to calculate the position of the landmark.

Description of drawings

Figure 1: Framework diagram of landmark positioning system;

Figure 2: The image of landmark points in images at different times;

Figure 3: Schematic diagram of landmark localization.

Detailed ways

In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific implementation manners of the present invention will now be described with reference to the accompanying drawings.

In this embodiment, a method for locating landmarks on a smart car map that combines monocular vision and differential GNSS includes the following steps:

S1: The sensor synchronously obtains the vehicle and camera space coordinates corresponding to the image;

S2: Data processing detects image landmarks and extracts landmark feature points;

S3: Use the optical flow method to track the image feature points;

S4: Calculate and obtain the landmark position and describe the sub-calculation.

A method for locating landmarks on a smart car map that combines monocular vision and differential GNSS, S1 includes the following sub-steps:

S11: Build the acquisition system, install the camera and differential GNSS dual positioning antennas on the roof, make the camera and GNSS antennas on the same plane, the direction of the camera is the same as that of the dual antennas, and the distance between the camera and the positioning antennas is d;

S12: Use the time stamps of different sensor data to perform temporal nearest neighbor synchronization, and obtain a set of standardized data for each frame of image, which needs to meet:

e _i =(P _i , V _i )

Among them, P _i represents the position and attitude of the vehicle corresponding to each frame image, namely ( _xi , y _i , _zi , α _i , β _i , γ _i ), where ( _xi , y _i , _zi ) is the Camera coordinates, (α _i , β _i , γ _i ) are the three angles of the camera pose on the vehicle, V _i represents the image corresponding to the current pose;

S13: Use the camera distortion parameters to correct the image to obtain the corrected image, and use the position obtained by GNSS to obtain the camera position by using the relative position transformation between the sensors.

A method for locating landmarks on a smart car map that combines monocular vision and differential GNSS, S2 includes the following sub-steps:

S21: Use a deep learning algorithm to detect landmarks in the image (such as traffic lights, traffic signs, ground signs, etc.), the detection result is the landmark type (ID) and the two-dimensional position in the image, and two pixels are used to express the landmark in the image The position of Rect(T ₁ ,T ₂ ,B ₁ ,B ₂ ).

S22: Extracting ORB features from the image region where the landmark is detected;

A kind of intelligent vehicle map landmark location method of fusion monocular vision and difference GNSS, described step S3 comprises following sub-steps:

S31: Use the optical flow method to track inter-frame feature points. If a feature point appears outside the landmark detection frame in a certain frame, delete the feature point. The implementation of the optical flow method is based on the target in the video stream, and only consistent The assumption of constant small displacement, constant brightness and similar motion of adjacent frames. To judge whether a certain local area in two consecutive frames of images I and J is the same target, it needs to meet:

I(x,y,t)=J(x',y',t+Δ)

All (x, y) are moved in one direction (d _x , d _y ), thus obtaining (x′, y′).

S32: The (x, y) point at time t is (x+d _x , y+d _y ) at time t+τ, so the problem of finding a match can be reduced to finding the minimum value of the following formula, which needs to be satisfied:

Where w _x and w _y represent 1/2 of the W window, u _x and u _y respectively represent the image coordinates of the points to be matched. In order to get the best match, make ε the smallest, let the derivative of the above formula be 0, find the minimum value, and the obtained d is the tracking offset.

Said step S4 also includes the following sub-steps:

S41: As shown in Figure 2, during landmark positioning, the imaging point of the same 3D landmark point in different images in consecutive (N+1) images from time t to time t+N, O is the camera center at each time, and Z represents three-dimensional The distance between the landmark point and the camera at different times (Z is unknown) S42: Figure 3 is the principle diagram of landmark positioning. According to the camera pinhole imaging principle, the two angles from the landmark point to the camera center vector can be known from the image According to the coordinate system transformation relationship:

z ₀ =Z*cosθ ₁

Among them (x ₀ , y ₀ , z ₀ ), the position of this landmark point can be calculated by using the same landmark point observed by multiple image frames (with known positions);

S43: Extract multiple feature points for the same landmark to track, and finally take the average value to calculate the position of the landmark.

The invention proposes a method for locating landmarks on a smart car map that combines monocular vision and differential GNSS, which can realize fast and high-precision landmark positioning. The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the industry should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and the description only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Variations and improvements are possible, which fall within the scope of the claimed invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims (5)

1. a kind of intelligent vehicle map terrestrial reference localization method for merging monocular vision and difference GNSS, which is characterized in that including as follows Step:

S1: pretreatment carries out data acquisition according to the system built；

S2: image terrestrial reference detection carries out terrestrial reference detection according to deep learning algorithm and obtains terrestrial reference detection result；

S3: terrestrial reference feature point extraction carries out ORB feature point extraction according to above-mentioned image terrestrial reference detection result；

S4: terrestrial reference tracking carries out interframe using optical flow method and tracks to obtain previous frame characteristic point in the corresponding characteristic point of present frame；

S5: terrestrial reference positioning carries out triangulation calculation and is averaged to indicate current terrestrial reference to multiple characteristic points that a terrestrial reference extracts Position；

S6: terrestrial reference description calculates, and sub- extraction is described to landmark image and obtains the description of terrestrial reference uniqueness；

S7: terrestrial reference storage repeats S2-S6 step, determines and obtains different terrestrial reference positioning, wherein by it is described differently Calibration position is stored in landmark data library for constructing intelligent vehicle map.

2. a kind of intelligent vehicle map terrestrial reference localization method for merging monocular vision and difference GNSS according to claim 1, It is characterized in that, S1 includes following sub-step:

S11: building acquisition system, installs camera and the bis- positioning antennas of difference GNSS in roof；

S12: time arest neighbors is carried out using the timestamp of different sensors data and is synchronized, each frame image obtains one group of standardization Data need to meet:

e_i=(P_i, V_i)

Wherein, P_iIndicate the corresponding vehicle location of every frame image and posture, i.e. (x_i, y_i, z_i, α_i, β_i, γ_i), wherein (x_i, y_i, z_i) For camera coordinates on vehicle, (α_i, β_i, γ_i) be camera posture on vehicle three angles, V_iIndicate the corresponding figure of current pose Picture；

S13: it converts to obtain camera position using relative position between sensor.

3. a kind of intelligent vehicle map terrestrial reference localization method for merging monocular vision and difference GNSS according to claim 1, It is characterized in that, the terrestrial reference detection result is arranged to the two-dimensional position in terrestrial reference type and detection image, the detection figure Two-dimensional position as in is expressed with two pixels.

4. a kind of intelligent vehicle map terrestrial reference localization method for merging monocular vision and difference GNSS according to claim 1, It is characterized in that, the step S4 includes following sub-step:

S41: judgement appears in whether a certain regional area in two continuous frames image I, J is same target, however, it is determined that is to need completely Foot:

I (x, y, t)=J (x ', y ', t+ Δ)

Wherein, own (x, y) and all move (d to a direction_x, d_y), to obtain (x ', y ')；

(x, the y) point at S42:t moment is (x+d at the t+ τ moment_x, y+d_y), so the problem of seeking matching can turn to and seek to following formula It minimizes, needs to meet:

Wherein w_xAnd w_yRespectively indicate the 1/2, u of W window_xAnd u_yRespectively indicate the image coordinate of point to be matched.It is best in order to obtain Matching, so that ε is minimum, enabling above formula derivative is 0, seeks minimum, and the d solved is the offset tracked.

5. a kind of intelligent vehicle map terrestrial reference localization method for merging monocular vision and difference GNSS according to claim 1, It is characterized in that, the step S5 further includes following sub-step:

S51: continue the imaging point in (N+1) when obtaining terrestrial reference positioning from t moment to t+N；

S52: coordinate system transformation relationship is utilized, image landmark locations are obtained.Transformation relation meets:

z₀=Z*cos θ₁

Wherein (x₀, y₀, z₀) it is the position that this landmark point can be calculated using the same landmark point that multiple images frame observes It sets；

S53: multiple characteristic points are extracted to the same terrestrial reference and are tracked, are finally averaged to calculate this place target position.