CN112085790A - Point-line combined multi-camera visual SLAM method, equipment and storage medium - Google Patents

Abstract

The present invention proposes a multi-camera visual SLAM method, device and storage medium combining point and line. By collecting multi-angle image data of a target scene, extracting and matching point features and line features in the multi-angle image data, and obtaining The position information of the point feature and the line feature in the three-dimensional space, the camera pose is preliminarily estimated for each frame of the multi-angle image data, and the extracted and matched point features, line features and the camera are combined. Based on the preliminary estimation result of the pose, a graph structure is constructed, and a three-dimensional map is determined according to the extracted point features, line features and the graph structure. This embodiment adopts the joint solution method of point and line features. Since the line features contain more information, the stability and accuracy of the tracking can be improved, and the sparse feature map constructed by using the line features can make the scene more clearly and intuitively. abstract description.

Description

Multi-camera visual SLAM method, equipment and storage medium combined with point and line

technical field

The present invention relates to the technical field of computer vision, in particular to a point-line combined multi-camera visual SLAM method, device and storage medium.

Background technique

The SLAM algorithm is widely used in autonomous navigation and environment recognition of robots in augmented reality, aerospace, underwater and other scenarios, and has important theoretical significance and practical value. Among them, visual SLAM based on sequence image data has become a hot research topic due to its advantages of economy and portability. How to effectively extract representative and tracking meaningful features from image data, describe them using a suitable algebraic language, and how to fully utilize the above information to restore camera pose and scene structure are the key concerns of SLAM algorithms.

The feature method is currently the mainstream method in visual SLAM. By extracting abstract geometric features from images and performing data association, the relative pose relationship between different frames is calculated, the camera trajectory is recovered, and a sparse feature map is constructed. Common feature-based visual SLAM has the following deficiencies: (1) The camera angle of view used in ordinary visual SLAM systems is narrow, and the information obtained in a single frame is limited. (2) The point feature used in a large number of algorithms has a single dimension, and contains less scene information than other high-dimensional features. It is easy to be lost when the quality of image data is poor or the camera moves too fast, resulting in unstable tracking results and unqualified map construction. It is intuitive and difficult to reflect the real scene, so the feature-based visual SLAM in the prior art cannot meet the requirements of feature tracking stability and high precision.

Therefore, the prior art needs to be further improved.

SUMMARY OF THE INVENTION

In view of the deficiencies of the prior art, the purpose of the present invention is to provide a point-line combined multi-camera visual SLAM method, device and storage medium, so as to overcome the problems in the prior art that the stability and accuracy cannot meet the requirements when using the feature method for feature tracking. defect.

The technical scheme of the present invention is as follows:

In the first aspect, this embodiment discloses a multi-camera visual SLAM method combining point and line, including:

Collect multi-angle image data of the target scene;

Extracting and matching point features and line features in the multi-angle image data, and obtaining position information of the point features and line features in three-dimensional space;

Perform a preliminary estimation of the camera pose for each frame of the multi-angle image data, and combine the extracted and matched point features, line features and the preliminary estimation result of the camera pose to construct a graph structure;

A three-dimensional map is determined according to the extracted point features, line features and the graph structure.

Optionally, before the step of extracting point features and line features in the multi-angle image data, the method further includes:

Performing distortion correction processing on the acquired multi-angle image data to obtain intermediate image data after distortion correction;

The intermediate image data is masked by using a preset camera mask to obtain preprocessed multi-angle image data.

Optionally, the step of extracting point features and line features in the multi-angle image data includes:

Extracting the corner features in the multi-angle image data using a feature point extraction algorithm, and using the rBRIEF descriptor to describe the corner features, and performing feature matching based on the corner descriptions to obtain the extracted point features;

The line feature extraction algorithm is used to extract the line segment features in the image, and the LBD description operator is used to describe the extracted line segment features, and the line feature matching is performed based on the line segment feature description; the extracted line features are obtained.

Optionally, using a line feature extraction algorithm to extract line segment features in the image, and using an LBD description operator to describe the extracted line segment features, and performing line feature matching based on the line segment feature description The steps include:

Extracting structural line segments from the multi-angle image data by using a line feature extraction algorithm;

Selecting a preset number of structural line segments, and using descriptors to describe the image features of the selected structural line segments, and performing structural line segment matching according to the image feature description;

The straight line matched by the structural line segment is described using the four-dimensional orthogonal description of the Plück coordinate, and the extracted multiple line features are obtained.

Optionally, the step of performing structural line segment matching according to the image feature description includes:

Perform structural line segment matching in multiple channels of the image respectively, and mark structural line segments that meet preset matching conditions in at least one channel as valid line segments to obtain a first valid line segment set;

And, using the k-nearest neighbor method to match each line segment marked as the effective line segment again to obtain the effective line segment obtained by matching again, and obtain the second effective line segment set;

Bidirectional matching is performed on each valid line segment in the second valid line segment set to obtain the best matching pair.

Optionally, performing a preliminary estimation of the camera pose for each frame of the multi-angle image data, and combining the extracted and matched point features, line features and the preliminary estimation result of the camera pose to construct a graph structure. The steps include:

Use epipolar geometry to determine the relative pose relationship of the initial frame;

Estimate the pose relationship of the frame to be solved relative to the known frame through camera motion state estimation and EPnP method, and obtain the relative pose relationship of each frame to be solved;

Taking point features, line features and camera positions as vertices, a graph structure is constructed based on the projection relationship of the point features and line features.

Optionally, the step of constructing a graph structure with the point feature, line feature, and camera position as vertices and using the point feature and line feature projection relationship includes:

The line features and point features associated with the camera pose to be solved are added to the graph structure as vertices;

According to the visual relationship between point features and line features in the solution frame, multi-vertex edges are respectively constructed in the G2O open source library; wherein, the multi-vertex edges are the corresponding relative positions between point features, line features and cameras posture relationship;

According to the relative pose relationship between the point feature, the line feature and the camera corresponding to the multi-vertex edge, calculate the point feature reprojection error in each frame of image to solve the Jacobian matrix for the camera pose, and in each frame of image Calculate the reprojection error of the line feature to solve the Jacobian matrix for the camera pose;

According to the solved reprojection error, the optimization algorithm is used to iteratively solve the camera position and feature coordinates.

Optionally, the step of determining the three-dimensional map according to the extracted point features, line features and the graph structure further includes:

Based on the common view relationship and the similarity between adjacent frames, the adjacent frames in the multi-angle image data are screened to obtain a closed-loop spare group;

The bag-of-words closed-loop detection method is used to detect the matching area corresponding to the closed-loop spare group, and the common view relationship of the current frame is corrected according to the detection result, and the coordinate values of the feature points in the matching area are updated, and the closed-loop is corrected in the world coordinates. position in the system;

Update the common time domain and connection relationship between past frames in the graph structure according to the detected closed loop;

Combined with the updated map structure and the pre-saved end point coordinates of the line feature on the multi-angle image, the extracted straight line is clipped, and the clipped line segment is used to determine the three-dimensional map.

In a second aspect, this embodiment discloses an information processing device, which includes a processor and a storage medium communicatively connected to the processor, where the storage medium is suitable for storing a plurality of instructions; the processor is suitable for calling the The instructions in the storage medium are used to execute the steps of implementing the above-mentioned multi-camera vision SLAM method combined with dots and lines.

In a third aspect, this embodiment discloses a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors , to implement the steps of the multi-camera vision SLAM method as described.

Beneficial effects: The present invention proposes a multi-camera visual SLAM method, system and device combining point and line. By collecting multi-angle image data of a target scene, extracting and matching point features and line features in the multi-angle image data, And obtain the position information of the point features and line features in the three-dimensional space, carry out a preliminary estimation of the camera pose for each frame of the multi-angle image data, and combine the extracted and matched point features, line features and all According to the preliminary estimation result of the camera pose, a graph structure is constructed, and a three-dimensional map is determined according to the extracted point features, line features and the graph structure. This embodiment adopts the joint solution method of point and line features. Since the line features contain more information, the stability and accuracy of the tracking can be improved, and the sparse feature map constructed by using the line features can make the scene more clearly and intuitively. abstract description.

Description of drawings

Fig. 1 is the step flow chart of a kind of point-line combined multi-camera visual SLAM method according to the present invention;

2 is a schematic diagram of collecting image data using a camera mask in an embodiment of the present invention;

Fig. 3 (a) is the schematic diagram of the geometric meaning of parameter θ ₁ in the straight line orthogonal description method provided by the embodiment of the present invention;

Fig. 3 (b) is the schematic diagram of the geometric meaning of parameter θ ₂ in the straight line orthogonal description method provided by the embodiment of the present invention;

Fig. 3 (c) is the schematic diagram of the geometric meaning of parameter θ ₃ in the straight line orthogonal description method provided by the embodiment of the present invention;

4 is a schematic diagram of a result of a sparse feature map corresponding to a local scene provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an information processing device in an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer and clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

It will be understood by those skilled in the art that the singular forms "a", "an", "the" and "the" as used herein can include the plural forms as well, unless expressly stated otherwise. It should be further understood that the word "comprising" used in the description of the present invention refers to the presence of stated features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components and/or groups thereof. It will be understood that when we refer to an element as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Furthermore, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combination of one or more of the associated listed items.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It should also be understood that terms, such as those defined in a general dictionary, should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as herein, should not be interpreted in idealistic or overly formal meaning to explain.

The inventor found that the camera used in the SLAM algorithm in the prior art has narrow vision, and the information that can be obtained from a single frame of picture is limited, and due to the recovery of the camera trajectory based on point features, the result of the tracking trajectory is unstable, and the constructed map cannot be used. reflect the real scene.

In order to overcome the above problems in the prior art, the present embodiment discloses a multi-camera visual SLAM method, device and storage medium combining point and line, by extracting point features and line features in multi-angle image data, and obtaining point features and the position description of the line feature, use the matching algorithm to establish the correspondence between the point feature and the line feature in the frame; then use the model to mathematically describe the point and line features in the two-dimensional image plane and the three-dimensional world, establish the projection relationship, and according to The point feature, line feature, point feature and line feature, and the preliminary estimation result of the camera pose are used to construct a graph structure to realize the visualization of the camera trajectory and the three-dimensional feature map.

The method, system and device provided by the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

Exemplary method

In the first aspect, this embodiment discloses a multi-camera visual SLAM method combining point and line, as shown in FIG. 1 , including:

Step S1, collecting multi-angle image data of the target scene.

First, the multi-angle image data in the target scene area is collected. In order to obtain panoramic data with more information, the Ladybug 5+ multi-camera panoramic device is used in this step to collect the multi-angle image data in the target scene area.

Because the six cameras on the Ladybug panoramic device simultaneously collect 360° scene data. Compared with a system that uses a single ordinary camera to collect data, using multiple wide-angle cameras to collect data can obtain more scene information in the same period of time; the trajectory calculation results integrate information from various angles, improving the stability of the system.

In order to facilitate the processing of the acquired multi-angle image data, after the multi-angle image data is collected, the collected multi-angle image data is also preprocessed. The preprocessing steps include:

Step S01: Perform distortion correction processing on the acquired multi-angle image data to obtain intermediate image data after distortion correction.

When the Ladybug 5+ multi-camera panoramic device is used in this step, you can directly use the Ladybug SDK toolkit to perform distortion correction on the image.

Step S02 , performing mask processing on the intermediate image data by using a preset camera mask to obtain preprocessed multi-angle image data.

As the data was collected, the outlines of the collector's head and other sensors on the backpack appeared on the image. In order to avoid erroneously extracting features from the junction of the image and the frame and the photographed interfering objects, which affects the subsequent feature tracking and trajectory solution, after using the Laydbug SDK tool to correct the distortion of the image, then use the Laydbug SDK tool to correct the image distortion. After mask processing, the corrected image corresponds to the mask as shown in Figure 2. Due to the difference in the placement of the lens on the data collection backpack, the occluded part of the image is different, so the mask is made according to the difference of the lens. Each lens has a separate fixed mask. Images are adaptive.

Further, this step also includes: using the Ladybug SDK toolkit to obtain internal and external parameters of the Ladybug device, including the focal lengths of the six cameras carried by the Ladybug, the coordinates of the image center, and the rotation and translation parameters relative to the center of the device.

Step S2: Extract and match point features and line features in the multi-angle image data, and obtain position information of the point features and line features in the three-dimensional space.

The point features and line features are extracted from the multi-angle image data collected by the camera, and the position information of the point features and line features in the extracted multi-angle image data in the three-dimensional space is obtained.

Specifically, this step includes: extracting and matching point and line features respectively on the obtained sequence image data, and using an appropriate method to geometrically describe its position in the image and space.

Further, extracting and matching the point and line features includes: extracting the corner features in the multi-angle image data using a feature point extraction algorithm, and using the rBRI EF descriptor to describe the corner features, and based on the corner description. Feature matching to obtain the extracted point features. And use the line feature extraction algorithm to extract the line segment feature in the image, and use the LBD description operator to describe the extracted line segment feature, and perform line feature matching based on the line segment feature description; obtain the extracted line feature.

Specifically, this step includes the following steps:

Step S21, extracting point features in the sequence image, and performing matching and geometric description. The specific process is as follows:

Step S21.1, using the ORB algorithm to extract point features in the image. The steps for extracting point features using other algorithms are similar to ORB. Ensure that the point features are evenly distributed in the image. Before extracting the feature, the image needs to be divided into several grids. In each grid, the initial threshold is used to extract the FAST corner points. If the extraction fails, the smallest value slightly larger than the initial threshold is used instead. The threshold is extracted, and the threshold is adjusted according to the effect so that a similar number of point features can be extracted in each grid.

First, the FAST corner detection algorithm is used to extract several corner points in the image, and then the Harris response of each corner point is calculated, and the N points with the largest Harris response are selected as the best features, and then an image pyramid is established, generally an 8-layer pyramid is established. Set the scale of the 0th level of the pyramid to 1, the scale of the 1st level to 1.2, then the 2nd level to be 1.2 ² , and so on. Corner extraction is performed on 8 pyramid images respectively, the number of pyramid layers where the corner is located is recorded as the scale information of the point, and the moment is used to determine the direction of the corner.

For a certain corner point to be found, first calculate the brightness weighted centroid within the radius r with the point as the center of the circle, and use the direction of the vector moment formed from the centroid to the point as the direction of the corner point.

Use I(x, y) to represent the brightness of the corner (x, y), and the moment of the corner is:

The center of mass coordinates are

The direction of the corner point is θ=atan2(y _c , x _c ).

For each extracted point feature, a 5*5 neighborhood is selected as a judgment window, and only when all the pixels in the window are within the range of the mask passing portion, the feature is judged to be an effective feature.

Step S21.2, use the rBRIEF descriptor to generate a 256-dimensional binary descriptor to describe the image features around the corner, and estimate the similarity of the image performance of the corresponding feature points by calculating the Hamming Distance between the descriptors. feature matching.

或齐次坐标

描述点特征在世界坐标系中的位置。使用齐次坐标可以更方便地将点在不同坐标系间进行转换。Step S21.3, use three-dimensional space coordinates

or homogeneous coordinates

Describes the location of a point feature in the world coordinate system. Using homogeneous coordinates makes it easier to convert points between different coordinate systems.

Further, the method steps of extracting line features in the multi-angle image data include:

The line feature extraction algorithm is used to extract the line segment features in the image, and the LBD description operator is used to describe the extracted line segment features, and the steps of performing line feature matching based on the line segment feature description include:

Extracting structural line segments from the multi-angle image data by using a line feature extraction algorithm;

Selecting a preset number of structural line segments, and using descriptors to describe the image features of the selected structural line segments, and performing structural line segment matching according to the image feature description;

The straight line matched by the structural line segment is described using the four-dimensional orthogonal description of the Plück coordinate, and the extracted multiple line features are obtained.

Step S22, extract the line features in the sequence images, perform matching operations on them, and use a four-dimensional orthogonal expression based on Plück coordinates to geometrically describe their positions in the three-dimensional space. Different from the general coordinate transformation method, the method of coordinate transformation of line features in world space, camera space and image plane is introduced in detail. The specific process is as follows:

Step S22.1, using the LSD (Line Segment Detector) method to extract structural line segments from the image. Filter by a certain length, and take the two ends p _s , p _e and the midpoint p _m of the line segment, and judge whether all the points in its 3*3 neighborhood are in the range of the mask passing part described in step S101.2. Within, the extracted line segment is determined to be a valid feature only when all three points pass.

Step S22.2, after extracting a sufficient number of line segments (not less than 30 in the texture-rich area, and not less than 10 in the texture-poor area), then use the method based on the LBD descriptor (Line Band Descriptor) to generate a 256-dimensional The binary descriptor is used to describe the image features of the line segments extracted from different images, and match them by calculating the Hamming distance.

In step S22.3, the line segment features in the image are described using the homogeneous coordinates of the endpoints x ₁ (u ₁ , v ₁ , 1) and x ₂ (u ₂ , v ₂ , 1). For linear features in space, a four-dimensional orthogonal description based on Plück coordinates is used to describe them.

则空间直线的普吕克坐标可由6维向量L^T～(n^T，v^T)^T表示，其中：Homogeneous coordinates of two points in a given space

Then the Plück coordinate of a straight line in space can be represented by a 6-dimensional vector L ^T ～(n ^T , v ^T ) ^T , where:

Then the orthogonal coordinates of the line are:

则U∈SO(3)，W∈SO(2)，线段参数可简化为矩阵U的对数映射向量θ和W对应角度θ的组合

其中每个参数都具有各自的几何意义：矩阵W包含的比率信息σ₁/σ₂表示了从坐标原点O到直线的距离d，参数θ与d的方向相关。矩阵U包含了直线L的三维坐标信息，θ₁、θ₂、θ₃各自对应的几何关系为：let matrix

Then U∈SO(3), W∈SO(2), the line segment parameters can be simplified to the combination of the logarithmic mapping vector θ of the matrix U and the corresponding angle θ of W

Each parameter has its own geometric meaning: the ratio information σ ₁ /σ ₂ contained in the matrix W represents the distance d from the coordinate origin O to the straight line, and the parameter θ is related to the direction of d. The matrix U contains the three-dimensional coordinate information of the straight line L, and the corresponding geometric relationships of θ ₁ , θ ₂ , and θ ₃ are:

1. θ ₁ means that the straight line remains tangent to the circle with O as the center, d as the radius, and the circle on the OL plane rotates;

2. θ ₂ means that the straight line remains tangent to the circle with O as the center, d as the radius, perpendicular to the OL plane and intersecting the line on the point P and rotates;

3. θ ₃ represents the rotation of the line around the axis OP. The geometric meaning represented by each component in the orthogonal coordinates of the straight line is described by the change of the position of the straight line when the component changes, as shown in Figure 3a to Figure 3c.

进行更新，算法通过计算指数和对数映射，更新矩阵U和W来进行：In the subsequent graph optimization calculation, it is necessary to adjust the matrix according to the increment of δθ

To update, the algorithm does this by computing exponential and logarithmic mappings, updating matrices U and W:

表示世界到相机的坐标转换矩阵，依据

将普吕克直线L_w从世界坐标系转换到相机坐标系下直线L_c，其中R_cw∈SO(3)为转换旋转矩阵，

为平移向量，

表示普吕克坐标系下T_cw的变体形式。Step S22.4, with

Represents the coordinate transformation matrix from the world to the camera, according to

Convert the Plück line L _w from the world coordinate system to the line L _c in the camera coordinate system, where R _cw ∈ SO(3) is the transformation rotation matrix,

is the translation vector,

Represents a variant of T _cw in the Plück coordinate system.

in accordance with

为普吕克坐标系下相机内置矩阵，其中(u_c,v_c)表示相机主光轴在图像坐标系中的坐标，f_u、f_v分别为相机在影像u、v方向上的焦距，n_c直线在相机坐标系内普吕克坐标的分量。Transform the line from the camera coordinate system to the camera plane line l', where

is the built-in matrix of the camera in the Plück coordinate system, where (u _c , v _c ) represents the coordinates of the main optical axis of the camera in the image coordinate system, f _u and f _v are the focal lengths of the camera in the u and v directions of the image, respectively, The component of the Plück coordinate of the n _c line in the camera coordinate system.

In this step, in order to achieve a better line feature matching effect, the step of performing structural line segment matching according to the image feature description includes:

Step 22.21: Perform structural line segment matching in multiple channels of the image respectively, and mark structural line segments that meet preset matching conditions in at least one channel as valid line segments to obtain a first valid line segment set.

For multi-channel color images, line segment matching is performed in each channel. As long as the matching conditions are met in more than one channel, the line segment can be marked as an effective match. This step is to make full use of the information of color image data in each channel. , after the matching in this step, the first valid line segment set is obtained.

In step 22.22, each line segment marked as the valid line segment is matched again by using the k-nearest neighbor method to obtain valid line segments obtained by matching again, and obtain a second valid line segment set.

For each match, use the K Nearest-Neighbor method to find the line segment whose description operator distance is next to the match, and mark the match only when the difference between the two distances is greater than the set threshold. It is a valid match, otherwise the match result is discarded. This step of screening is to reduce the occurrence of matching misalignment between similar line segments, so as to increase the matching accuracy. After this matching, a second valid line segment set is obtained.

Step 22.23: Perform bidirectional matching on each valid line segment in the second valid line segment set to obtain the best matching pair.

Swap the position of the feature sequence of the frame to be matched and the feature sequence of the new input frame, perform two-way matching, and only retain the feature pair that is the best match in both directions. For example, assuming that there is a line segment l ₁ in frame A, the best matching line segment with l ₁ in frame B is calculated and screened to be l ₂ ; by exchanging the positions of A and B, only when the best match with l ₂ in frame A is When the line segment is still l ₁ , it is judged that the matching result {l ₁ , l ₂ } is a valid matching pair.

Step S3: Perform a preliminary estimation of the camera pose for each frame of the multi-angle image data, and combine the extracted and matched point features, line features and the preliminary estimation result of the camera pose to construct a graph structure.

In this step, the feature association information obtained according to the aforementioned method is used, the epipolar geometry is used to initialize the system, and the EPnP method is used to perform preliminary pose estimation. Initialization mainly includes the following steps:

Step S31, using the epipolar geometry to determine the relative pose relationship of the initial frame.

Extract point features from the input image. When the number of point features is sufficient (greater than the threshold τ _d ), it is used as the undetermined initial frame for the next step, otherwise the frame is skipped and the calculation continues; Features are extracted from the frame, and matched, and the matching results are subjected to nearest neighbor and bidirectional filtering. If the number of matching features is also sufficient (greater than τ _m ), it can be regarded as an undetermined initial frame pair, and the fundamental matrix F is solved by using epipolar geometry, and the essential matrix E is further solved, and then the relative poses of the two frames of cameras are preliminarily estimated.

Step S32, estimating the pose relationship of the frame to be solved relative to the known frame by estimating the motion state of the camera and the EPnP method, and obtaining the relative pose relationship of each frame to be solved.

Perform system initialization. The initial frame is added to the system as a key frame, and the matching feature points and lines in the initial frame are converted to the world coordinate system and added to the map. Then use the graph optimization method to optimize the global graph of the camera pose, point feature, and line feature, and adjust the pose relationship. When using the multi-camera sequence image data for actual operation, the initialization result of camera 0 is used as the standard, and the initialization pose is converted into the Ladybug device pose according to the camera calibration result obtained in the preprocessing, and is mapped to the remaining cameras.

In step S33, a graph structure is constructed with the point feature, the line feature and the camera position as vertices, and the projection relationship between the point feature and the line feature.

The pose relationship of the frame to be solved relative to the known frame is roughly estimated by the camera motion state estimation and EPnP method, which is used as the initial value of graph optimization.

According to the point-line feature matching relationship, a graph is constructed, which includes multiple point feature vertices, multiple line feature vertices, multiple (local or global estimation) or one (single-frame estimation) camera vertices, and feature-camera corresponding edge relationships.

Step S4: Determine a three-dimensional map according to the extracted point features, line features and the graph structure.

In this step, according to the point features, line features and the constructed graph structure extracted in the above steps S2 and S3, the trajectory tracking of the camera is implemented and the constructed three-dimensional map is obtained.

Further, this step also includes the step of using the G2O tool to optimize the graph constructed in the above step S3, by using the graph optimization algorithm to optimize the positioning and attitude information of the camera in each shooting frame, and update the constructed graph structure. , so as to optimize the 3D world coordinates of point features and line features in the graph structure, so as to obtain a more accurate 3D map.

Specifically, the step of constructing a graph structure with the point feature, line feature and camera position as vertices, and the projection relationship between the point feature and the line feature includes:

The line features and point features associated with the camera pose to be solved are added to the graph structure as vertices;

According to the visual relationship between point features and line features in the solution frame, multi-vertex edges are respectively constructed in the G2O open source library; wherein, the multi-vertex edges are the corresponding relative positions between point features, line features and cameras posture relationship;

According to the relative pose relationship between the point feature, the line feature and the camera corresponding to the multi-vertex edge, calculate the point feature reprojection error in each frame of image to solve the Jacobian matrix for the camera pose, and in each frame of image Calculate the reprojection error of the line feature to solve the Jacobian matrix for the camera pose;

According to the solved reprojection error, the optimization algorithm is used to iteratively solve the camera position and feature coordinates.

The specific steps of using the G2O graph structure for pose estimation are as follows:

根据向量加法进行更新；线特征的描述向量为：Step S41, adding line features and point features associated with the camera pose to be solved into the graph as vertices. The vertex structure needs to include the geometric position description vector of the feature, and the method to optimize the update of the pose according to the increment estimated by the Jacobian solution during the optimization process. The description vector of the point feature is the three-dimensional space coordinate

Update based on vector addition; the description vector for line features is:

The delta is converted to a special Euclidean group using an exponential map, and updated using a logarithmic map that converts back to the orthogonal description coordinates.

Step S42, construct a multi-vertex edge according to the visual relationship of the feature in the frame to be solved: point feature-line feature-camera. This setting is for the convenience of solving. Since the pose relationship between the features is independent of each other, the empty feature is used to fill in the actual operation process, that is, the empty point-line-camera description line feature-camera correspondence is used, and the point-empty line is used. - Camera description point feature-camera correspondence. The edge structure needs to include the reprojection error and the solution method of the Jacobian matrix. The specific steps are as follows:

Step S42.1, in the kth frame, set the world coordinate of the point feature i as P _i , the corresponding image coordinate as p _k,i , the camera built-in matrix as K, and the transformation matrix from the world coordinate system to the camera coordinate system as T _{k , cw} , then the reprojection error ep _k,i =p _k,i -KT _k,cw P _i .

The Jacobian matrix of the point feature reprojection error for the camera pose δξ is:

所示，[]_1-3表示向量的前三维，d(.)表示距离函数：

l′(l₁，l₂，l₃)为使用方程系数表示的直线从世界坐标系到影像的投影，x₁(u₁，v₁，1)和x₂(u₂，v₂，1)为使用齐次坐标表示的实际量测线段端点坐标。根据链式法则，线特征重投影误差对相机姿态求解雅克比矩阵为：Step S42.2, in the kth frame, the world Plück coordinate of the line feature j is L _j , and the corresponding measurement line segment l _k,j in this frame is represented by the homogeneous coordinates of its endpoints, then the characteristic line j is reprojection error

shown, [] _1-3 represent the first three dimensions of the vector, and d(.) represents the distance function:

l'(l ₁ , l ₂ , l ₃ ) is the projection of the straight line represented by the equation coefficients from the world coordinate system to the image, x ₁ (u ₁ , v ₁ , 1) and x ₂ (u ₂ , v ₂ , 1 ) is the end point coordinates of the actual measured line segment represented by homogeneous coordinates. According to the chain rule, the Jacobian matrix of the line feature reprojection error for the camera pose is:

In step S42.3, the robustness of the optimization process is enhanced through two steps: (1) a robust kernel function is set for each edge to reduce the influence of outliers; (2) each optimization grouping is repeated. After each set of iterations is completed, the features corresponding to the edges with the reprojection error greater than the threshold th are marked as outliers, and these edges are eliminated. In the next set of iterations, only the edges (Inliers) whose errors are within the threshold range are retained. Obtain a more stable pose estimation. According to the different degrees of freedom of the chi-square distribution, the point feature th _p =5.991th _p =5.991, and the line feature th _l =7.815. If the number of non-outlier features after the initial pose calculation is sufficient, and the distance from the last local optimization reaches a certain distance, local graph optimization is required to further stabilize the relationship between pose and feature and reduce the cumulative error.

When local optimization is in progress, through map search, find past keyframes that have a common view field with the current keyframe, use the poses and features of these common view keyframes to construct vertices, and construct corresponding edges according to the visual relationship to join the optimization graph. optimize at the same time. After the closed loop is detected, global optimization is required.

Step S43, in the Ladybug multi-camera panoramic shooting system, when estimating the pose of a single frame, the pose of each frame relative to the initialized local coordinate system needs to be calculated separately, and finally unified into the Ladybug coordinate system, and the coordinates of each local camera are converted to Center and solve the mean to get the coordinates of the Ladybug device in the scene.

In step S44, except that the initial frame must be used as a key frame, the subsequent conditions for judging whether a frame of image is a key frame are as follows: (1) N ordinary frames have passed since the previous key frame was added to the map (implementation; In the process, take N=9); (2) The number of interior points/line features (Inliers) obtained after the solution is within a certain range, such as less than nf _max (nf _max =90) and greater than nf _min (nf _min =70) ); (3) the ratio of the number of internal point/line features to the number of matching features in the previous frame is less than _{rkf (r kf =} 0.9) and greater than 50. Condition (2)(3) indicates that the matching features are decreasing, but the solution result for this frame is still stable. If, in the above three conditions, the solution result of a certain frame satisfies the condition (1), and satisfies any one of (2) and (3), the frame will be added to the map as a key frame. In order to reduce the situation of repeatedly providing feature information between keyframes, it is also necessary to remove the too dense keyframes. Obtain the remaining keyframes that share the line-of-sight range with the current keyframe and the feature information in them through the common vision relationship. If 90% or more of the features of the current keyframe can be observed in other keyframes, the frame belongs to the redundant keyframe. removed from the map.

Further, this step also includes: using a dictionary generated by DBoW2 to perform bag-of-words closed-loop detection to check whether the camera has reached a location that it has passed before, and perform closed-loop correction, so as to impose constraints on the trajectory to eliminate or reduce accumulated errors . Then the optimized line segments are clipped, and finally the scene sparse feature map is generated.

Step S44.1, based on the common view relationship and the similarity between adjacent frames, filter adjacent frames in the multi-angle image data to obtain a closed-loop spare group.

Through the common view relationship screening, the adjacent frames that have a common field of view with the current frame are eliminated, and the frames with the same word as the current frame are found in the remaining key frames in the map as candidate frames. Count the number of common words between all candidate frames and the current frame, and take 80% of the maximum number of common words as the threshold for screening. The number of common words exceeds the threshold, and the similarity with the current frame exceeds the minimum similarity between the current frame and the aforementioned adjacent frames. all keyframes. Group these keyframes and their previous 10 adjacent keyframes, calculate the sum of the similarity between each frame in each group and the current frame as the group score, take 75% of the highest group score as the threshold, and find out all groups with a score higher than The key frame with the highest similarity with the current frame in the threshold group is used as the candidate group. All frames in the candidate group are checked for consistency (check whether they have a common adjacent frame with the current frame), so as to filter out the final closed-loop candidate group.

Step S44.2, using the bag-of-words method closed-loop detection method to detect the matching area corresponding to the closed-loop standby group, and correcting the common view relationship of the current frame according to the detection result, and updating the coordinate values of the feature points in the matching area, and Correct the position of the closed loop in the world coordinate system.

The candidate group is screened according to the number of matching points, and the remaining frames in the candidate group and the three pairs of matching points in the current frame are used for sim3 calculation, that is, the rotation matrix, translation vector and scale factor between matching points are obtained through the similarity transformation group. The obtained sim3 relationship uses the number of matching inliers as a condition to filter out the approximate matching area. After correcting the camera pose and features of the matching area using the sim3 relationship, search in the nearby area for matching. If there are more than 40 matching points, it is determined that a closed loop is detected and the solution thread is notified to stop inserting new key frames and enter the next step, otherwise the candidate group is cleared and the next detection is awaited.

Step S44.3, update the common time domain and connection relationship between past frames in the graph structure according to the detected closed loop.

Update the common view relationship of the current frame according to the closed-loop detection result, solve the sim3 relationship with the adjacent frames of the current frame, update the coordinate values of each feature point in the area according to the closed-loop sim3, and convert the sim3 relationship into T∈SE(3)T ∈SE(3);

Correct the position of the closed loop in the world coordinate system, and then, according to the newly discovered closed loop, update the common viewshed and connection relationship between past keyframes in the map. Finally, the graph structure is reconstructed according to the updated relationship. First, the key part (Essential Graph) of the new connection relationship in the graph is optimized due to the addition of closed loops. Finally, the global optimization of all key frames and feature point lines is performed.

Step S44.4, combining the updated graph structure and using the pre-saved end point coordinates of the line feature on the multi-angle image to crop the extracted straight line, and use the cropped line segment to determine the three-dimensional map.

When using line features for graph optimization, in order to reduce the description parameters to further reduce the amount of calculation, and reduce the influence of the line segment endpoints between different frames drifting along the straight line caused by image frame truncation or line segment detection operators, the orthogonal parameters used Describes the position of the line where the line segment is projected to the world coordinate system. When constructing a sparse map of scene features, use the pre-saved endpoint coordinates of line features on the image to first crop the straight line, and use the cropped line segments to construct a sparse scene map. A schematic diagram of the structure of the local feature sparse map is shown in Figure 4.

Exemplary Equipment

This embodiment discloses an information processing device, as shown in FIG. 5 , including a processor and a storage medium communicatively connected to the processor, where the storage medium is suitable for storing a plurality of instructions; the processor is suitable for calling the The instructions in the storage medium are used to execute the steps of implementing the above-mentioned multi-camera vision SLAM method combined with dots and lines.

In addition, the above-mentioned logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

As a computer-readable storage medium, the memory 22 may be configured to store software programs and computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 30 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, ie, implements the methods in the above embodiments.

The memory 22 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal device, and the like. Additionally, memory 22 may include high-speed random access memory, and may also include non-volatile memory. For example, U disk, removable hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes, or temporary state storage medium.

On the other hand, this embodiment also provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors Execute the steps to realize the multi-camera vision SLAM method of point-line combination.

Compared with the existing feature-based visual SLAM, the method embodiment disclosed in the present invention comprehensively uses point-line features and multi-angle data captured by multiple cameras:

1) Use the six cameras on the Ladybug panoramic device to collect 360° scene data at the same time. Compared with a system that uses a single ordinary camera to collect data, using multiple wide-angle cameras to collect data can obtain more scene information in the same period of time; the trajectory calculation results integrate information from various angles, improving the stability of the system.

2) On the basis of the LBD line feature matching method, optimization strategies such as multi-channel matching, two-way verification, and Knn conditional constraints are added to improve the matching accuracy.

3) The joint solution method of point and line features is adopted. Line features have higher dimensions than point features, and contain more scene structure information. Adding point and line features to the optimization graph at the same time for joint calculation can improve the stability and accuracy of tracking. The sparse feature map constructed using line features can abstract the scene more clearly and intuitively.

Through the above three points, the embodiment of the present invention can obtain better trajectory tracking and map construction results.

In the following, the experiment of single-camera sequence images using the point-line combined feature algorithm, the comparison of the results of the experiment with the ORB algorithm using only point features, the camera parameter settings of the multi-camera panoramic data experiment d, and the multi-camera panoramic data experiment are given below. Comparison of the accuracy results of each single-camera trajectory solution in .

Table 1

Table 2

table 3

Table 1 compares the results of experiments on single-camera sequence images using the point-line combined feature algorithm and the ORB algorithm using only point features. In the table, the unit of trajectory length is unit length, RPE RMSE is the root mean square error value of Relative Pose Error, and the percentage is the ratio of error to trajectory length, which is used to measure the accuracy of trajectory tracking.

Table 2 shows the results of experiments using multi-camera panoramic data. The size of the images captured by each camera is 1232*1024, and the sequence images are captured by a special backpack equipped with Ladybug5+ equipment. Table 3 shows the accuracy results of each single-camera trajectory calculation in the aforementioned multi-camera panoramic data experiments. Combining Table 2 and Table 3, it can be seen that the tracking accuracy of multi-angle shooting has been significantly improved compared with that of a single camera.

To sum up, when experimenting with monocular fisheye camera data, the number of tracking frames and the track length after adding line features are higher than the ORB algorithm implementation results using only point features, and the accuracy is also improved.

The present invention proposes a multi-camera visual SLAM method, system and device combining point and line, by collecting multi-angle image data of a target scene, extracting and matching point features and line features in the multi-angle image data, and obtaining the result. Describe the position information of point features and line features in three-dimensional space, carry out preliminary estimation of camera pose for each frame of image in the multi-angle image data, and combine the extracted and matched point features, line features and the camera pose The initial pose estimation result is used to construct a graph structure, and a three-dimensional map is determined according to the extracted point features, line features and the graph structure. This embodiment adopts the joint solution method of point and line features. Since the line features contain more information, the stability and accuracy of the tracking can be improved, and the sparse feature map constructed by using the line features can make the scene more clearly and intuitively. abstract description.

It can be understood that for those of ordinary skill in the art, equivalent replacements or changes can be made according to the technical solutions of the present invention and the inventive concept thereof, and all these changes or replacements should belong to the protection scope of the appended claims of the present invention.

Claims (10)

1. a multi-camera visual SLAM method of point-line combination, is characterized in that, comprises: Collect multi-angle image data of the target scene area; Extracting and matching point features and line features in the multi-angle image data, and obtaining position information of the point features and line features in three-dimensional space; Perform a preliminary estimation of the camera pose for each frame of the multi-angle image data, and combine the extracted and matched point features, line features and the preliminary estimation result of the camera pose to construct a graph structure; A three-dimensional map is determined according to the extracted point features, line features and the graph structure. 2. The multi-camera visual SLAM method combining point and line according to claim 1, wherein before the step of extracting point features and line features in the multi-angle image data, further comprising: Performing distortion correction processing on the acquired multi-angle image data to obtain intermediate image data after distortion correction; The intermediate image data is masked by using a preset camera mask to obtain preprocessed multi-angle image data. 3. The multi-camera visual SLAM method combining point and line according to claim 1 or 2, wherein the step of extracting point features and line features in the multi-angle image data comprises: Extracting the corner features in the multi-angle image data using a feature point extraction algorithm, and using the rBRIEF descriptor to describe the corner features, and performing feature matching based on the corner descriptions to obtain the extracted point features; The line feature extraction algorithm is used to extract the line segment features in the image, and the LBD description operator is used to describe the extracted line segment features, and the line feature matching is performed based on the line segment feature description to obtain the extracted line features. 4. The multi-camera visual SLAM method of point-line combination according to claim 3, characterized in that, a line feature extraction algorithm is used to extract line segment features in the image, and an LBD description operator is used to describe the extracted line segment features, And the steps of performing line feature matching based on the line segment feature description include: Extracting structural line segments from the multi-angle image data by using a line feature extraction algorithm; Selecting a preset number of structural line segments, and using descriptors to describe the image features of the selected structural line segments, and performing structural line segment matching according to the image feature description; The straight line matched by the structural line segment is described using the four-dimensional orthogonal description of the Plück coordinate, and the extracted multiple line features are obtained. 5. The multi-camera visual SLAM method for point-line combination according to claim 4, wherein the step of performing structural line segment matching according to the image feature description comprises: Perform structural line segment matching in multiple channels of the image respectively, and mark structural line segments that meet preset matching conditions in at least one channel as valid line segments to obtain a first valid line segment set; And, using the k-nearest neighbor method to match each line segment marked as the effective line segment again to obtain the effective line segment obtained by matching again, and obtain the second effective line segment set; Bidirectional matching is performed on each valid line segment in the second valid line segment set to obtain the best matching pair. 6 . The multi-camera visual SLAM method for point-line combination according to claim 1 , wherein the preliminary estimation of the camera pose is performed on each frame of the multi-angle image data, and combined extraction and matching to obtain the SLAM method. 7 . The point feature, line feature and the preliminary estimation result of the camera pose, the steps of constructing the graph structure include: Use epipolar geometry to determine the relative pose relationship of the initial frame; Estimate the pose relationship of the frame to be solved relative to the known frame through camera motion state estimation and EPnP method, and obtain the relative pose relationship of each frame to be solved; Taking point features, line features and camera positions as vertices, a graph structure is constructed based on the projection relationship of the point features and line features. 7. The multi-camera visual SLAM method of point-line combination according to claim 6, wherein the point feature, line feature and camera position are used as vertices, and a graph structure is constructed with the point feature and line feature projection relationship The steps include: The line features and point features associated with the camera pose to be solved are added to the graph structure as vertices; According to the visual relationship between point features and line features in the solution frame, multi-vertex edges are respectively constructed in the G2O open source library; wherein, the multi-vertex edges are the corresponding relative positions between point features, line features and cameras posture relationship; According to the relative pose relationship between the point feature, the line feature and the camera corresponding to the multi-vertex edge, calculate the point feature reprojection error in each frame of image to solve the Jacobian matrix for the camera pose, and in each frame of image Calculate the reprojection error of the line feature to solve the Jacobian matrix for the camera pose; According to the solved reprojection error, the optimization algorithm is used to iteratively solve the camera position and feature coordinates. 8. The multi-camera visual SLAM method of point-line combination according to claim 7, wherein the step of determining a three-dimensional map according to the extracted point features, line features and the graph structure further comprises: Based on the common view relationship and the similarity between adjacent frames, the adjacent frames in the multi-angle image data are screened to obtain a closed-loop spare group; The bag-of-words closed-loop detection method is used to detect the matching area corresponding to the closed-loop spare group, and the common view relationship of the current frame is corrected according to the detection result, and the coordinate values of the feature points in the matching area are updated, and the closed-loop is corrected in the world coordinates. position in the system; Update the common time domain and connection relationship between past frames in the graph structure according to the detected closed loop; Combined with the updated map structure and the pre-saved end point coordinates of the line feature on the multi-angle image, the extracted straight line is clipped, and the clipped line segment is used to determine the three-dimensional map. 9. An information processing device, comprising a processor and a storage medium communicatively connected to the processor, wherein the storage medium is adapted to store a plurality of instructions; the processor is adapted to call the instructions in the storage medium , to perform the steps of implementing the multi-camera vision SLAM method of any one of the above claims 1-8. 10. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the claim Steps of the multi-camera vision SLAM method described in any one of 1-8 are required.

Images