CN109960402A - A virtual-real registration method based on fusion of point cloud and visual features - Google Patents

Abstract

The present invention claims to protect a virtual-real registration method based on point cloud and visual feature fusion. The whole process is completed in two stages, offline and online, and specifically includes the following steps: in the offline stage, the three-dimensional CAD model of the reference object part is stored in the CAD environment. Generate a 3D point cloud model to prepare for the establishment of the absolute coordinate system of the assembly and the subsequent virtual-real registration. In order to speed up the process of point cloud registration and the establishment of the assembly coordinate system, it is necessary to simplify the point cloud model of the reference object part at this stage; In the online stage, the video stream of the assembly site is collected and a point cloud is generated, and the registration between the point cloud of the reference object and the point cloud of the assembly site is used to complete the definition of the absolute coordinate system of the assembly. The virtual-real registration based on the point cloud requires a In the initial registration stage, the relative position of the initial point cloud is set, and the accurate tracking registration matrix in the absolute coordinate system is obtained. The present invention not only avoids the "frame loss" situation caused when the camera moves too fast, but also improves the accuracy of virtual and real registration.

Description

A virtual-real registration method based on fusion of point cloud and visual features

technical field

The invention belongs to the technical field of augmented reality, and in particular relates to a virtual-real registration method based on fusion of point clouds and visual features.

Background technique

The virtual-real registration (ie three-dimensional tracking registration) technology is to correctly match or organically combine the virtual objects or other information generated by the computer with the real scene, and the key is the tracking and positioning of the camera. Many scholars have proposed a large number of methods for camera tracking and positioning, mainly including tracking and positioning based on artificial identification and natural features. However, due to strict assembly requirements for mechanical products, manual marking is strictly prohibited. In addition, the assembly environment is mostly based on smooth and texture-less components, and the tracking registration based on natural features is also difficult to apply effectively. The tracking registration method based on dense point cloud uses scene depth image to generate point cloud, and realizes camera tracking and positioning through point cloud registration. It can perform well in mechanical product assembly environment with low light intensity and lack of surface texture. Therefore, it has attracted the interest and attention of the majority of scientific researchers.

The virtual-real registration method based on point cloud can show good robustness in the mechanical product assembly environment with low light intensity and lack of surface texture. However, the current virtual-real registration method based on dense point cloud can only estimate the relative pose of the sensor, which is not directly applicable to the mechanical product assembly environment. At the same time, the virtual-real registration method based on dense point cloud mainly uses the iterative closest point method to estimate the camera pose. However, when the camera moves rapidly, the iterative closest point (ICP) registration algorithm cannot obtain a sufficient number of correct initial corresponding data points, and the iterative process is easy to fall into a local optimum, which is difficult to ensure the convergence speed and registration accuracy, and even causes "lost" frame", resulting in the interruption of the virtual and real registration process. Therefore, how to use the redundant color image information of the depth sensor at the same time, through feature point matching, to obtain a sufficient number of correct initial corresponding data point sets, and improve the convergence speed of the iterative process and the calculation accuracy of the camera pose, become difficult. Moreover, since the surface of objects on the assembly site is mostly smooth and without texture, how to obtain a sufficient number of matching feature point pairs is also a problem that needs to be solved.

Virtual-real registration is the primary condition for virtual-real integration of AR assembly assistance systems. How to register the virtual assembly guidance information into the assembly environment stably and with high precision has become a research difficulty. The virtual-real registration method based on dense point cloud can show better robustness in the assembly environment of mechanical products with low illumination intensity and lack of surface texture. This type of method uses all or most points in the point cloud to participate in the calculation process, and uses the iterative closest point (ICP) registration algorithm to iteratively calculate all corresponding point pairs to obtain the camera pose, which greatly improves the accuracy of virtual and real registration. However, the iterative closest point strategy adopted by the current virtual-real registration method based on dense point cloud can only estimate the pose of the sensor relative to its initial position (not fixed), which is not directly applicable to the assembly scene that requires absolute assembly position information. At the same time, when the depth sensor moves too fast to obtain the correct initial corresponding data point set, the iterative process of the ICP algorithm is easy to fall into a local optimum, and it is difficult to ensure the convergence speed and registration accuracy. Moreover, when it exceeds the working range of the depth sensor (0.15-4m), it is easy to cause a "dropped frame" situation, resulting in the interruption of the virtual and real registration process. The continuous inter-frame registration used in the tracking process will also greatly reduce the tracking accuracy with the increase of the moving distance of the sensor; the virtual-real registration method based on feature point matching extracts feature points in the color image, and then maps the feature points to the three-dimensional space. , and then calculate the transformation between the corresponding feature point pairs through the RANSAC algorithm, so as to complete the estimation of the camera pose. The virtual-real registration method based on feature point matching has a fast response speed, does not have high requirements on the surface of the parts, and has a wide working range, and can still complete the camera pose estimation well when the depth sensor exceeds the working range. However, this method requires high ambient lighting conditions. Therefore, the combination of feature point matching and point cloud registration algorithm is one of the effective methods to solve the failure of virtual and real registration process in mechanical product assembly environment. However, due to the lack of sufficient texture features on the mechanical product assembly site or tooling surface, using traditional SIFT or SURF as a feature detector or descriptor for transformation matrix estimation, it is easy to cause virtual and real registration failure due to lack of sufficient matching feature points.

SUMMARY OF THE INVENTION

The present invention aims to solve the above problems of the prior art. A virtual-real registration method based on the fusion of point cloud and visual features is proposed, which not only avoids the "dropped frame" situation caused by the camera moving too fast, but also improves the accuracy of virtual-real registration. The technical scheme of the present invention is as follows:

A virtual-real registration method based on the fusion of point clouds and visual features. The whole process is completed in two stages, offline and online, including the following steps:

In the offline stage, the 3D CAD model of the reference object part is generated in the CAD environment of the computer-aided design to generate a 3D point cloud model to prepare for the establishment of the absolute coordinate system of the assembly and the subsequent virtual-real registration. In order to speed up the point cloud registration and the establishment of the assembly coordinate system At this stage, the point cloud model of the reference object part needs to be simplified;

In the online stage, the video stream of the assembly site is collected and a point cloud is generated. The registration between the reference object point cloud and the assembly site point cloud is used to complete the definition of the absolute coordinate system of the assembly. The virtual and real registration based on the point cloud needs to be in the iterative registration process. In an initial registration stage, the relative position of the initial point cloud is set, and the accurate tracking registration matrix in the absolute coordinate system is obtained.

Further, in the online stage, the point cloud generation and preprocessing of the assembly environment specifically include:

1) Point cloud generation step: perform parallel back-projection of each pixel point u _D in the depth image to the camera coordinate space to obtain vertices, the three-dimensional vertex V _t (u) and the normal vector N _t (u) corresponding to the vertex form a three-dimensional Assembly environment point cloud;

2) Steps of establishing the point cloud spatial index: using the spatial index Kd-tree to establish the point cloud data structure;

3) The nearest neighbor search step of Kd-Tree.

Further, in the step 1) point cloud generation step, after the depth image of the assembly environment is repaired by noise reduction, each pixel point u _D = (x _d , y _d ) at time i in the image is reversed by formula (1). Projected to the depth camera space coordinate system. In order to speed up this process, each pixel point u _D in the depth image is back-projected to the camera coordinate space in parallel to obtain vertices. The back-projection result is:

V _t (u)=D _i (u)M _{int_D} ^-1 [u _D ,1] (1)

In the formula: D _i (u)——the depth image at moment i; M _{int_D} ——internal parameters of the depth camera.

At the same time, the normal vector N _t (u) corresponding to each vertex is obtained by using the adjacent projection points through the following operations:

N _t (u)=(V _t (x+1,y)-V _t (x,y))×(V _t (x,y+1)-V _t (x,y)) (2)

V _t (x+1, y) corresponds to a vertex. x, y represent the horizontal and vertical coordinates of the pixel;

Normalize (2) to unit length N _t /||N _t || ₂ , ||N _t || ₂ , N _t represent the corresponding normal vector three-dimensional vertex V _t (u) and the corresponding normal vector N _t ( u) Form a 3D assembly environment point cloud.

Further, the step 2) point cloud spatial index establishment step: using the spatial index Kd-tree to establish the point cloud data structure, specifically including:

Step1: Determine the Split field value of the split point;

Step2: Determine the value of the Node-data field of the root node;

Step3: Determine the left subspace and the right subspace;

Step4: Recursive step.

Further, the basic idea of the step 3) nearest neighbor search is:

Step1: Perform a binary search to generate a search path. The point to be queried is quickly searched along the search path to find the nearest approximate point, that is, the leaf node;

Step2: Backtracking search, the found leaf nodes are not necessarily the nearest neighbors. The nearest neighbor points should be located in the circle domain with the query point as the center and passing through the leaf nodes. In order to find the real nearest neighbor, it is necessary to search backwards along the search path. Are there any data points that are closer to the query point.

Further, the process of establishing the assembly coordinate system specifically includes the steps:

Step1: Take the pump shell as the object to generate a point cloud

Step2: Collect the depth image R _t (u) of the assembly site at time t online (where u=(x, y)), and perform noise reduction and repair on the depth image R _t (u) to obtain the depth image after noise reduction and repair, and then Generate a 3D environment point cloud, where X={x ₀ ,x ₁ ,x ₂ ... x _n ,N ₀ ,N ₁ ,N ₂ ...N _n }

Step3: Use the point cloud P={p ₀ ,p ₁ ,p ₂ ...p _m ,n ₀ ,n ₁ ,n ₂ ...n _m } to use the iterative nearest point (ICP) for registration, and multi-threading of graphics processing unit (GPU) is used for parallel operation to improve registration speed. In order to obtain the corresponding registration points, the Kd-tree is used to speed up the matching search, and the Euclidian distance is used as the matching metric (see Equation 3). At the same time, in order to minimize the number of mismatched points, the normal vector feature is added as an additional constraint. The dot product d _ij =<m _i , m _j > is used to measure the normal vector of the candidate point. If d _ij is greater than a certain threshold ψ, it is considered to be the corresponding point, and a weight of 1 is assigned, which is expressed as formula 4

Step4: After adopting characteristic measurement methods such as Euclidean distance and normal vector, the number N of corresponding points and the weight _wi of each pair of corresponding points can be obtained. Use the least squares method to solve the positional relationship error matrix to determine the transformation matrix between the point cloud X and the point cloud P, see formula 5.

Step5: Through the solution of the rotation matrix R and the translation vector t, the definition of the initial absolute coordinate system of the assembly can be completed, preparing for the solution of the virtual and real registration matrix M in the subsequent absolute coordinate system

Further, the tracking registration matrix of the fusion of the point cloud and the visual feature specifically includes the steps:

First collect the color image and depth image of the assembly environment, and then use formulas (1) and (2) to transform the depth measurement value from the two-dimensional image coordinates to the global coordinate space to generate the environment point cloud; at the same time, the improved feature description operator ORB is used to extract The feature points in the color image are matched between consecutive frames; the matched ORB feature point pairs are mapped to the three-dimensional coordinate space using depth information, and outliers are eliminated by random sampling consistency. When the distance between the two points is less than the error value, Retain this feature point pair, otherwise remove outliers to obtain the initial transformation matrix; use the calculated transformation matrix and the correct matching point pair as the initial data set for ICP point cloud registration, and use loopback detection and pose optimization strategy to pair The transformation matrix obtained in the point cloud registration stage is optimized, and finally the accurate tracking registration matrix in the absolute coordinate system is obtained by combining with the matrix M _init .

The advantages and beneficial effects of the present invention are as follows:

Aiming at the characteristics of the mechanical product assembly environment, the invention first uses the reference model point cloud to define the absolute coordinate system of the virtual and real registration, and then designs a visual feature matching strategy based on the consistency of the direction vector, and designs the dense point cloud and The RGBD-ICP virtual-real registration method combined with visual features, on this basis, adds loop detection and global optimization strategies, which not only avoids the "drop frame" situation caused by the camera moving too fast, but also improves the virtual-real registration accuracy.

The present invention establishes a point cloud of the assembly environment based on the depth image. According to the characteristics of reflective and lack of texture in the assembly environment, a RGBD-ICP virtual-real registration method combining point cloud and visual features is designed, which provides the first condition for the generation of virtual-real fusion scenes.

Description of drawings

1 is a flowchart of a virtual-real registration method based on the fusion of point clouds and visual features according to a preferred embodiment of the present invention;

Figure 2 shows the plane cut corresponding to the two-dimensional Kd-tree;

Figure 3 shows a schematic diagram of the solution of the assembly environment tracking registration matrix;

Figure 4 represents the error metric: (a) Euclidean distance between point-to-point (b) point-to-tangent plane (Point-to-Tangent Plane);

Figure 5 shows the camera pose estimation loop closure problem in the tracking registration process;

Figure 6 shows the pose graph structure for inter-frame constraints.

Detailed ways

The technical solutions in the embodiments of the present invention will be described clearly and in detail below with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are only some of the embodiments of the invention.

The technical scheme that the present invention solves the above-mentioned technical problems is:

The present invention establishes a point cloud of the assembly environment based on the depth image. According to the characteristics of reflective and lack of texture in the assembly environment, a RGBD-ICP virtual-real registration method combining point cloud and visual features is designed, which provides the first condition for the generation of virtual-real fusion scenes.

The basic idea and workflow of the algorithm

The basic idea and workflow of the algorithm are shown in Figure 1. The whole process is completed in two stages, offline and online. In the offline stage, a 3D point cloud model is generated from the 3D CAD model of the reference object part in the CAD environment to prepare for the establishment of the absolute coordinate system of the assembly and the subsequent virtual-real registration. In order to speed up the process of point cloud registration and the establishment of the assembly coordinate system, it is necessary to simplify the point cloud model of the reference object part at this stage. In the online stage, the video stream of the assembly site is collected and the point cloud is generated, and the definition of the absolute coordinate system of the assembly is completed by the registration between the point cloud of the reference object and the point cloud of the assembly site. The virtual-real registration based on point cloud needs an initial registration stage in the iterative registration process. A reasonable relative position of the initial point cloud can reduce the number of iterations, reduce the registration time, and avoid the algorithm from falling into local optimum. The present invention uses feature point-based matching to initialize the RGBD-ICP registration process, thereby completing the virtual-real registration matrix calculation. According to the virtual and real registration matrix, the 3D model of the parts is superimposed into the assembly environment, and the registration of virtual guidance information in the assembly environment is realized.

The invention proposes a virtual-real registration technology based on point cloud. The invention mainly includes the following steps:

1. Assembly environment point cloud generation and preprocessing:

1) Point cloud generation:

After the depth image of the assembly environment is repaired by noise reduction, each pixel point u _D = (x _d , y _d ) at time i in the image can be back-projected to the depth camera space coordinate system by formula (1). In order to accelerate this process, the CUDA parallel computing platform of NVIDIA is used to perform parallel back-projection of each pixel point u _D in the depth image to the camera coordinate space to obtain a vertex map. The back-projection result is:

V _t (u)=D _i (u)M _{int_D} ^-1 [u _D ,1] (3)

where:

D _i (u)——the depth image at time i;

M _{int_D} - Depth camera internal parameters.

At the same time, the normal vector N _t (u) corresponding to each vertex is obtained by using the adjacent projection points through the following operations:

N _t (u)=(V _t (x+1,y)-V _t (x,y))×(V _t (x,y+1)-V _t (x,y)) (4)

Normalize (2) to unit length N _t /||N _t || ₂ . The three-dimensional vertex V _t (u) and the normal vector N _t (u) corresponding to the vertex form a three-dimensional assembly environment point cloud.

2) Point cloud spatial index establishment:

The above point cloud is only a collection of surface points of the target object, and does not contain the geometric topology relationship between points. In order to more efficiently realize the operation operations such as point cloud data registration in the following text, it is necessary to establish the spatial relationship between point clouds. Commonly used point cloud spatial organization relationships include octree, regular grid, K-dimensional tree (Kd-tree), etc. The present invention uses the spatial index Kd-tree to establish the point cloud data structure.

Kd-Tree is a binary tree that stores some K-dimensional data. Constructing a Kd-Tree on a K-dimensional data set represents the division of the K-dimensional space formed by the K-dimensional data set. To illustrate the construction process of Kd-Tree, take two-dimensional data points as an example, suppose there are six two-dimensional data points {(2,3), (5,4), (9,6), (4,7), (8,1), (7,2)}, the data points are located in two-dimensional space. In order to effectively find the nearest point, the Kd-tree adopts the idea of divide and conquer, so we can simply number the two axes of X and Y as 0,1, that is, Split={0,1}.

Step1: Determine the splitting point (Split) domain value. Calculate the maximum variance of the data in the X and Y directions of the above six two-dimensional data points respectively, and the corresponding dimension is the value of the Split field, so the value of the Split field first takes 0, that is, the X-axis direction;

Step2: Determine the value of the root node (Node-data) domain. Sort according to the data 2, 5, 9, 4, 8, 7 on the X dimension, where the value is 7, so the Node-data field value is (7, 2). Therefore, the split hyperplane of this node is a straight line passing through the (7,2) point and perpendicular to Split=0 (X axis), that is, X=7;

Step3: Determine the left subspace and the right subspace. The split hyperplane X=7 divides the entire space into two parts, the part with X≤7 is the left subspace, including 3 nodes (2,3), (5,4) and (4,7); X>7 The part is the right subspace, containing 2 nodes (9,6) and (8,1).

Step4: Recursion. The construction of Kd-tree is a recursive process. Repeating the process of the root node for the data in the left subspace and the right subspace can get the next level child nodes (5,4) and (9,6) (that is, the left and right subspaces). the root node of the subspace), while further subdividing the space and dataset. And so on until the space contains only one data point. The Kd-tree generated by six 2D data points is shown in Figure 2:

The three-dimensional space point cloud Kd-Tree construction is an extension of the two-dimensional situation. For the three-dimensional Kd-Tree, the root node selects the X axis, the child nodes of the root node (Node-data) select the Y axis, and the child nodes of the child nodes select the Z axis. axis, and so on. Each time, the median of the corresponding instance is selected as the split point (Split), the split point is used as the parent node, and the left and right sides are divided into two subtrees.

3) Nearest neighbor search of Kd-Tree:

Searching for data in Kd-Tree is also an important part of point cloud registration, and its purpose is to retrieve the point cloud data that is closest to the query point in Kd-Tree. In three-dimensional space, the Euclidian distance between two data points p(x ₁ ,x ₂ ,x ₃ ) and P(p ₁ ,p ₂ ,p ₃ ) can be expressed by equation (3):

The basic idea of nearest neighbor search is:

Step1: Perform a binary search to generate a search path. The point to be queried is quickly searched along the search path to find the nearest approximate point, that is, the leaf node.

Step2: Backtracking search. The found leaf nodes are not necessarily the closest neighbors, and the closest neighbors should be located in the circle with the query point as the center and passing through the leaf nodes. In order to find the true nearest neighbors, it is also necessary to search backwards along the search path to see if there are data points that are closer to the query point. For the specific process, see the literature.

2. The assembly absolute coordinate system is established.

Generate a point cloud model P={p ₀ , p ₁ , p ₂ ... p _m , n ₀ , n ₁ , n ₂ ... n _m } under the absolute coordinate system from the reference object model, where p ₀ ,p ₁ ,...p _m are the vertex coordinates of the model surface, and n ₀ ,n ₁ ,...n _m are the normal vectors corresponding to the vertices. Taking the pump casing as the object,

During the assembly process, the depth image R _t (u) (where u=(x, y)) of the assembly site is collected online at time t, and the depth image R _t (u) is denoised and repaired by the method in Chapter 2, and the reduced Noise-repaired depth image D _t (u), and then generate a three-dimensional environment point cloud X according to the above point cloud generation method, where X = {x ₀ , x ₁ , x ₂ ... x _n , N ₀ , N ₁ , N ₂ ...N _n }. The point cloud P={p ₀ , p ₁ , p ₂ ... p _m , n ₀ , n ₁ , n ₂ ... n _m } is generated by combining the 3D environment point cloud with the reference object model using the iterative closest point (ICP ) for registration, and multi-threading of a graphics processing unit (GPU) is used to perform parallel operations to improve the registration speed. In order to obtain the corresponding registration points, the Kd-tree is used to speed up the matching search, and the Euclidian distance is used as the matching metric (see Equation 3). At the same time, in order to minimize the number of mismatched points, the normal vector feature is added as an additional constraint. The dot product d _ij =<m _i ,m _j > is used to measure the normal vector of the candidate point. If d _ij is greater than a certain threshold ψ, it is considered to be the corresponding point, and a weight of 1 is assigned, which is expressed as:

After adopting characteristic measurement methods such as Euclidean distance and normal vector, the number N of corresponding points and the weight _wi of each pair of corresponding points can be obtained. Use the least squares method to solve the positional relationship error matrix to determine the transformation matrix between the point cloud X and the point cloud P

Through the solution of the rotation matrix R and the translation vector t, the definition of the initial absolute coordinate system of the assembly can be completed, and preparations are made for the solution of the virtual and real registration matrix M in the subsequent absolute coordinate system.

3. Tracking registration method for fusion of point cloud and visual features.

A schematic diagram of the tracking registration of point cloud and visual feature fusion is shown in Figure 3. First, the color image and depth image of the assembly environment are collected, and then the depth measurement value is transformed from the two-dimensional image coordinates to the global coordinate space using formulas (1) and (2) to generate the environment point cloud. At the same time, the improved ORB (Oriented FAST and Rotated BRIEF) operator is used to extract the feature points in the color image and perform continuous inter-frame matching. The matched ORB feature point pairs are mapped to the three-dimensional coordinate space using depth information, and outliers are eliminated by random sampling consistency (RANSAC). When the distance between the two points is less than the error value, the feature point pair is retained, otherwise the outliers are eliminated to obtain the initial transformation matrix. The transformation matrix obtained by calculation and the correct matching point pair are used as the initial data set for ICP point cloud registration, thus avoiding the ICP iteration process falling into local optimum, while improving the convergence speed and registration accuracy. The transformation matrix obtained in the point cloud registration stage is optimized by using the loop closure detection and pose optimization strategy, and finally the accurate tracking registration matrix in the absolute coordinate system is obtained by combining the matrix M _init .

1) Visual feature extraction and matching:

The key step in the tracking registration of point cloud and visual feature fusion is to solve the transformation relationship between the environmental point cloud data under different poses of the depth sensor. Among them, iterative closest point algorithm (ICP) is one of the stable and efficient methods, which can transform the point cloud data matching problem into an exact solution problem that minimizes the distance function between data sets. The ICP algorithm is mainly divided into two steps. The first step is to determine the corresponding data point set, and the second step is to solve the transformation relationship between the point cloud data according to the corresponding point set. Obtaining the correct initial set of corresponding data points is the primary condition of the ICP algorithm. It is the key to avoid the algorithm iteration process falling into local optimum and ensure the convergence speed and registration accuracy. However, when the camera moves rapidly, the ICP registration algorithm is prone to fall into local optimum in the iterative process because it cannot obtain the initial correspondence between the point cloud data of the two frames before and after. Moreover, when it exceeds the working range of the depth sensor (0.15-4m), it is easy to cause a "drop frame" situation, which leads to the interruption of the tracking registration process. Considering that there is a one-to-one correspondence between the color image data of the depth sensor and the depth image data, the initial matching problem of point clouds can be transformed into the matching problem of color image feature points. At present, many local features such as SIFT, SURF, ORB, BRISK, FREAK, etc. are widely used in image matching, object recognition and other fields. Based on the consideration of fastness and stability of feature point matching, the present invention adopts the ORB feature point pair as the initial matching data point set. However, due to the lack of sufficient texture features in the mechanical product assembly environment or tooling surface, it is difficult to obtain a sufficient number of matching point pairs in the original ORB matching process, and the feature points are relatively concentrated, which is not conducive to ensuring the reliability of the rotation-translation matrix calculation.

In order to obtain a sufficient number of correct ORB feature point pairs as the ICP initial matching data point set, the present invention needs to improve the original ORB matching process. The original ORB feature matching algorithm uses FAST as the feature point detector, the improved BRIEF as the feature descriptor, and uses the BF (Brute Force) pattern matching algorithm for feature descriptor matching. The result of using the original ORB for feature extraction and BF for descriptor matching. It can be seen that the original ORB can obtain a sufficient number of feature points (9466), but there are fewer matching point pairs (252), and there are more mismatches.

In order to eliminate the mismatch, the random sampling consistency detection (RANSAC) algorithm is often used to eliminate the mismatch. After using RANSAC, only 87 matching point pairs can finally be obtained, and the number of matching point pairs is further reduced, which is difficult to meet the initial registration requirements of ICP.

In order to obtain a sufficient number of correct matching point pairs, the present invention is inspired by the idea of grid-based motion statistics (GMS) method—the matching points with smooth motion consistency are unlikely to appear randomly, and the matching point pairs can be judged true or false. This is achieved by a simple count of surrounding matching points. Based on the above idea, the OBR feature matching process is improved, the present invention is called OGMS, and the algorithm idea is based on the following assumptions:

Assumption: Matched point pairs in the surrounding neighborhood of correctly matched point pairs should have the same unit direction vector.

Let M be a matrix composed of unit direction vectors of matching point pairs in the neighborhood of a correct matching point pair:

M=[m ₁ m ₂ m ₃ ... m _n ] ^T (8)

The similarity between the direction vectors m _i and m _j is represented by the inner product d _ij :

d _ij =<m _i ,m _j > (9)

Direction vector m _i(1×2) and matching point pair matrix The similarity of each direction vector is represented by G:

G=|m _i M ^T |/(n-1) (10)

Arrange all the elements in G in descending order. If an element in G is less than the threshold δ, it means that the connection direction of the corresponding matching point pair deviates from the connection direction of most matching point pairs, and the matching point pair is regarded as a mismatch. are culled, and the meshing in GMS is used to speed up the matching process.

The above improved ORB is used for feature matching results. The improved ORB is used to extract and match the features of two adjacent video frames in the assembly environment. The number of correct matching points is changed from 87 to 3478, and the process takes less time. The visual feature matching pseudocode is shown in Table 1.

Table 1 Pseudo code of feature matching algorithm

2) ICP point cloud registration fused with visual features:

In order to accurately superimpose the virtual assembly guidance information in the assembly environment, it is necessary to obtain the tracking registration matrix in the absolute coordinate system in real time. In this part, the ICP registration of point cloud data between images of different depths, that is, the rotation and translation transformation matrix between the corresponding point cloud data, will be solved to obtain the position and attitude of the current perspective. In order to avoid the ICP iterative process falling into local optimum, improve the convergence speed and registration accuracy, and avoid the problem of tracking failure caused by the rapid movement of the camera, by fusing the visual features in the color image, the existence of a difference between the color image data and the depth image data is used. A corresponding relationship, the initial matching problem of the point cloud is converted into the matching of color image feature points, and the initial transformation matrix obtained by calculation and the correct matching point pair are used as the initial data set for ICP point cloud registration. It is called RGBD-ICP point cloud registration, and the pseudo code of RGBD-ICP point cloud registration algorithm is shown in Table 2.

Table 2 Pseudo code of RGBD-ICP point cloud registration algorithm

First, extract ORB (Oriented FAST and Rotated BRIEF) feature point sets F _a , F _b from two adjacent video frame color images of the assembly environment respectively, and obtain feature point matching point pair sets C _s , C through the improved ORB feature point matching algorithm _t . Because the ORB feature is extracted on the image, it shows that there is similarity in the feature vector in the two-dimensional space, but this does not mean that it is still a matching point pair in the three-dimensional space, because the change of the viewing angle will cause the change of the spatial position. If the mismatched point pair is not eliminated in advance, but it participates in the calculation of the transformation matrix, it will inevitably result in the iterative update of the point cloud transformation matrix using the initial transformation matrix calculated from the mismatched point pair, which will form an error accumulation. Therefore, the present invention firstly uses the depth images corresponding to the two frames of color images, and uses the internal parameters of the depth sensor to map the matching point pairs to the three-dimensional space to form a spatial point cloud set P _s , P _t . Then, the 3D mismatched point pairs are eliminated by random sampling consistency (RANSAC) method to form interior point sets P _{s_inliner} , P _{t_inliner respectively} , and the initial point cloud transformation matrix M'=[R'|t'] is obtained at the same time.

Enter the ICP main loop from step 5. In the first iteration, the initial transformation matrix M'=[R'|t'] calculated by ORB feature point pair matching makes the point cloud between two adjacent frames without any prior knowledge of pose realize the nearest neighbor Initial registration. In order to achieve further accurate registration of point clouds, the point cloud registration problem is transformed into an exact solution problem that minimizes the error function between point cloud sets. The matching point pair that satisfies the minimum error function between the assembly environment data point clouds is used as the input of the algorithm. The data between images from different perspectives can be regarded as the rotation and translation motion between multiple coordinate systems, so that each time we do a rotation and translation motion, the data will be updated, and the input data will be updated through repeated iterations of the transformation matrix to minimize the difference between the environmental point cloud data. The offset error function is continuously updated and iteratively updated until the error meets a certain accuracy requirement. The present invention divides the error function into two parts, the first part calculates the average Euclidean distance between pairs of ORB visual feature matching points, and the second part calculates the tangent plane (Point-to-Tangent Plane) between the current point in the point cloud and the target matching point )the distance. The point-to-tangent plane point cloud matching method can produce more accurate registration than the classical point-to-point matching method, and the iterative convergence speed is significantly faster. The error function of the present invention is defined as:

where:

α—— is the weight value;

—— is the target point in the dense point cloud The normal vector of , which is obtained by the cross product of its surrounding neighborhood points.

In order to speed up the ICP registration process, Kd-tree is used to accelerate the nearest point search, and a graphics processing unit (GPU) is used for parallel computing. When the error change of the point cloud transformation matrix is less than the threshold δ or the maximum number of iterations is reached, the ICP main loop stops iterating, otherwise the point cloud will be re-registered using the most recently calculated transformation matrix.

3) Loop detection and global optimization of camera pose:

The point cloud registration between consecutive adjacent frames used above is a better short-distance tracking registration method. However, the registration error between two adjacent frames can cause the estimation of the camera pose to drift with the moving distance, resulting in inaccurate tracking registration results. This drift phenomenon is most pronounced especially when the camera moves a long distance, eventually returning to a previously visited location, which is known as the loop closure problem. To address the above issues, loop closure detection is required to identify if the camera has returned to a previously visited location.

The above embodiments should be understood as only for illustrating the present invention and not for limiting the protection scope of the present invention. After reading the contents of the description of the present invention, the skilled person can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims (7)

1. a virtual and real registration method based on point cloud and visual feature fusion, is characterized in that, the whole process is divided into two stages offline and online to complete, specifically comprises the following steps: In the offline stage, the 3D CAD model of the reference object part is generated in the CAD environment of the computer-aided design to generate a 3D point cloud model to prepare for the establishment of the absolute coordinate system of the assembly and the subsequent virtual-real registration. In order to speed up the point cloud registration and the establishment of the assembly coordinate system At this stage, the point cloud model of the reference object part needs to be simplified; In the online stage, the video stream of the assembly site is collected and a point cloud is generated. The registration between the reference object point cloud and the assembly site point cloud is used to complete the definition of the absolute coordinate system of the assembly. The virtual and real registration based on the point cloud needs to be in the iterative registration process. In an initial registration stage, the relative position of the initial point cloud is set, and the accurate tracking registration matrix in the absolute coordinate system is obtained. 2. a kind of virtual-real registration method based on point cloud and visual feature fusion according to claim 1, it is characterized in that, in described online stage, assembling environment point cloud generation and preprocessing specifically comprise: 1) Point cloud generation step: perform parallel back-projection of each pixel point u _D in the depth image to the camera coordinate space to obtain vertices, the three-dimensional vertex V _t (u) and the normal vector N _t (u) corresponding to the vertex form a three-dimensional Assembly environment point cloud; 2) Steps of establishing the point cloud spatial index: using the spatial index Kd-tree to establish the point cloud data structure; 3) The nearest neighbor search step of Kd-Tree. 3. a kind of virtual and real registration method based on point cloud and visual feature fusion according to claim 2, is characterized in that, in described step 1) point cloud generation step, after the depth image of the assembly environment is repaired by noise reduction, the image Each pixel point u _D = (x _d , y _d ) at time i is back-projected to the depth camera space coordinate system by formula (1). In order to speed up this process, each pixel point u _D in the depth image is Perform parallel back-projection to the camera coordinate space to obtain vertices. The back-projection result is: V _t (u)=D _i (u)M _{int_D} ^-1 [u _D ,1] (1) In the formula: D _i (u)——the depth image at moment i; M _{int_D} ——internal parameters of the depth camera. At the same time, the normal vector N _t (u) corresponding to each vertex is obtained by using the adjacent projection points through the following operations: N _t (u)=(V _t (x+1,y)-V _t (x,y))×(V _t (x,y+1)-V _t (x,y)) (2) V _t (x+1, y) is the vertex corresponding to the pixel, and x and y represent the horizontal and vertical coordinates of the pixel; Normalize (2) to unit length N _t /||N _t || ₂ , ||N _t || ₂ , N _t represent the corresponding normal vector three-dimensional vertex V _t (u) and the corresponding normal vector N _t ( u) Form a 3D assembly environment point cloud. 4. a kind of virtual and real registration method based on point cloud and visual feature fusion according to claim 2, is characterized in that, described step 2) point cloud spatial index establishment step: adopt spatial index Kd-tree to carry out point cloud data structure establishment, including: Step1: Determine the Split field value of the split point; Step2: Determine the value of the Node-data field of the root node; Step3: Determine the left subspace and the right subspace; Step4: Recursive step. 5. a kind of virtual and real registration method based on point cloud and visual feature fusion according to claim 2, is characterized in that, the basic idea of described step 3) nearest neighbor search is: Step1: Perform a binary search to generate a search path. The point to be queried is quickly searched along the search path to find the nearest approximate point, that is, the leaf node; Step2: Backtracking search, the found leaf nodes are not necessarily the nearest neighbors. The nearest neighbor points should be located in the circle domain with the query point as the center and passing through the leaf nodes. In order to find the real nearest neighbor, it is necessary to search backwards along the search path. Are there any data points that are closer to the query point. 6. a kind of virtual and real registration method based on point cloud and visual feature fusion according to claim 1, is characterized in that, described assembly coordinate system establishment process specifically comprises steps: Step1: Take the water pump shell as the object to generate a point cloud; Step2: Collect the depth image R _t (u) of the assembly site at time t online (where u=(x, y)), and perform noise reduction and repair on the depth image R _t (u) to obtain the depth image after noise reduction and repair, and then Generate a 3D environment point cloud, where X={x ₀ ,x ₁ ,x ₂ ... x _n ,N ₀ ,N ₁ ,N ₂ ...N _n } Step3: The point cloud generated by the 3D environment point cloud and the reference object model P={p ₀ , p ₁ , p ₂ ... p _m , n ₀ , n ₁ , n ₂ ... n _m } Use the iterative closest point for registration, and use the multi-threading of the graphics processor to perform parallel operations to improve Registration speed. In order to obtain the corresponding registration points, Kd-tree is used to speed up the matching search, and the Euclidean distance is used as the matching metric. In order to minimize the number of mismatched points, the normal vector feature is added as an additional constraint. The point product d _ij =<m _i , m _j > is used to measure the normal vector of the candidate point. If d _ij is greater than a certain threshold ψ, it is considered to be a corresponding point, and a weight of 1 is given; Step4: After adopting characteristic measurement methods such as Euclidean distance and normal vector, the number N of corresponding points and the weight _wi of each pair of corresponding points can be obtained. The position relationship error matrix is solved by the least square method, and the transformation matrix between the point cloud X and the point cloud P is determined. Step5: Through the solution of the rotation matrix R and the translation vector t, the definition of the initial absolute coordinate system of the assembly can be completed, and the solution of the virtual and real registration matrix M in the subsequent absolute coordinate system can be prepared. 7. a kind of virtual and real registration method based on point cloud and visual feature fusion according to claim 1, is characterized in that, the tracking registration matrix of described point cloud and visual feature fusion specifically comprises steps: First collect the color image and depth image of the assembly environment, and then use formulas (1) and (2) to transform the depth measurement value from the two-dimensional image coordinates to the global coordinate space to generate the environment point cloud; at the same time, the improved feature description operator ORB is used to extract The feature points in the color image are matched between consecutive frames; the matched ORB feature point pairs are mapped to the three-dimensional coordinate space using depth information, and outliers are eliminated by random sampling consistency. When the distance between the two points is less than the error value, Retain this feature point pair, otherwise remove outliers to obtain the initial transformation matrix; use the calculated transformation matrix and the correct matching point pair as the initial data set for ICP point cloud registration, and use loop detection and pose optimization strategies to pair The transformation matrix obtained in the point cloud registration stage is optimized, and finally the accurate tracking registration matrix in the absolute coordinate system is obtained by combining with the matrix M _init .