CN113362467B - Point cloud preprocessing and ShuffleNet-based mobile terminal three-dimensional pose estimation method - Google Patents

Abstract

The invention discloses a mobile terminal three-dimensional pose estimation method based on point cloud preprocessing and ShuffleNet. First, preprocessing is performed on the PC terminal, and the target point cloud data is subjected to three-dimensional reconstruction and imported into a three-dimensional rendering engine; in the three-dimensional engine, rotation The camera algorithm obtains the two-dimensional photos of the target under different viewing angles, and the key voxel block extraction algorithm proposed by the present invention is used to mark the photos and establish a training data set; the ShuffleNetv2-YOLOv3 which has the advantages of light weight and high performance and is suitable for mobile terminal computing is adopted Train the key voxel block detection model of the target; read the video stream from the mobile camera, detect the key voxel block of the target through the ShuffleNetv2‑YOLOv3 model, and calculate the 2D‑3D point pair corresponding to the center point of the key voxel block through RANSAC and EPNP algorithms Get the relative pose of the target. Finally, using the advantages of the mobile terminal, the pose of the target in the actual 3D world is calculated through the data provided by the built-in IMU and GPS.

Description

3D Pose Estimation Method for Mobile Terminal Based on Point Cloud Preprocessing and ShuffleNet

technical field

The invention belongs to the field of computer technology, and relates to a mobile terminal three-dimensional pose estimation method based on point cloud preprocessing and ShuffleNet, which can be widely used in the fields of robot grasping, vehicle intelligent navigation, augmented reality, medical diagnosis and the like.

Background technique

3D pose estimation plays a key role in the fields of robot grasping, vehicle intelligent navigation, augmented reality and medical diagnosis. At present, the mainstream pose estimation methods are divided into two categories. One is the recognition method based on two-dimensional images. This method predicts one center point and eight corner points of the object for the input RGB or RGB-D image, and then passes The PNP or EPNP algorithm obtains the 6D pose of the object. This type of algorithm has good real-time performance but low accuracy. The other is positioning based on point cloud data. This method first uses a deep network to establish a correspondence between 3D point cloud data and 2D images, and then obtains the 6D pose of the object through the PNP or EPNP algorithm. Due to the use of point cloud data, the accuracy is higher than the first class, but the speed is comparatively lower.

Mobile phones have the advantages of high penetration and portability, but because the hardware configuration is far lower than that of PCs, it is difficult to meet the requirements for recognition speed using conventional algorithms. The configuration requires an external lidar and depth camera, which will weaken its portability advantage. The mobile terminal can only use the RGB video stream recognition solution, resulting in low accuracy of pose analysis.

Contents of the invention

The present invention mainly aims at the demand of the mobile terminal for target pose estimation in the auxiliary industrial application field, and provides a mobile terminal three-dimensional pose estimation method based on point cloud preprocessing and ShuffleNet.

The design scheme adopted in the present invention is: a mobile terminal three-dimensional pose estimation method based on point cloud preprocessing and ShuffleNet, comprising the following steps:

Step 1: Perform 3D reconstruction on the target point cloud data obtained by laser scanning; import the 3D model obtained from 3D reconstruction into the rendering engine for taking pictures;

Step 2: Use the positioning and rotation camera algorithm to obtain the two-dimensional photos and camera poses of the target under different viewing angles; extract the feature points of the two-dimensional photos through SIFT and calculate the corresponding three-dimensional feature points, and divide the target model into voxel blocks of equal size , screen key voxel blocks of the target according to the number of three-dimensional feature points; generate two-dimensional projections of key voxel blocks on the photo set and establish a training data set; train the ShuffleNet feature detection model for the target through the ShuffleNetv2-YOLOv3 lightweight network;

Step 3: Input the video stream into the trained ShuffleNetv2-YOLOv3 target key voxel block detection model, identify the key voxel blocks to obtain 2D-3D matching point pairs, and calculate the relative pose of the target by combining RANSAC and EPNP algorithms;

Step 4: Combining the mobile terminal GPS and IMU information to calculate the absolute pose of the target in the 3D world.

The present invention combines the advantages of two types of three-dimensional pose estimation algorithms, and first performs preprocessing on the PC side. The 3D model is reconstructed for the target point cloud data through the Delaulay algorithm. The present invention adopts positioning, rotation, photographing and key voxel block extraction algorithms to automatically generate a target voxel feature detection data set, and adopts the ShuffleNetv2-YOLOv3 training feature detection model that has the advantage of light weight and high performance and is suitable for mobile computing.

The present invention makes full use of the advantages of mobile terminal hardware equipment in the identification stage, and introduces GPS and IMU data to locate the pose of the mobile terminal. The key voxel blocks of the target are detected by the trained ShuffleNetv2-YOLOv3 model, and then the relative pose between the target and the mobile camera is calculated by using the RANSAC and EPNP algorithms. Finally, the absolute pose of the target in the 3D world is calculated. Today, when the penetration rate of mobile phones exceeds 90%, the present invention can provide pose estimation on mobile terminals that does not rely on depth cameras and laser equipment, and whose real-time and accuracy meet the requirements of industrial auxiliary applications. It has the advantages of being portable, practical and easy to promote .

Description of drawings

Fig. 1 is a flowchart of an embodiment of the present invention.

Fig. 2 is a functional block diagram of an embodiment of the present invention.

Detailed ways

In order to facilitate those of ordinary skill in the art to understand and implement the present invention, the present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described here are only used to illustrate and explain the present invention, and are not intended to limit this invention.

Please see Fig. 1 and Fig. 2, a kind of mobile terminal 3D pose estimation method based on point cloud preprocessing and ShuffleNet provided by the present invention, comprises the following steps:

Step 1: Perform 3D reconstruction on the target point cloud data obtained by laser scanning; import the 3D model obtained from 3D reconstruction into the rendering engine for taking pictures;

The specific realization of this embodiment includes the following sub-steps:

Step 1.1: Perform 3D reconstruction on the target point cloud data obtained by laser scanning through the Delaulay algorithm to obtain the 3D model of the target;

Step 1.2: Import the target 3D model into the rendering engine, calculate the bounding box and center point of the target 3D model, and move the target 3D model to bring the center point to the origin.

Step 2: Use the positioning and rotation camera algorithm to obtain the two-dimensional photos and camera poses of the target under different viewing angles; extract the feature points of the two-dimensional photos through SIFT and calculate the corresponding three-dimensional feature points, and divide the target model into voxel blocks of equal size , screen key voxel blocks of the target according to the number of three-dimensional feature points; generate two-dimensional projections of key voxel blocks on the photo set and establish a training data set; train the ShuffleNet feature detection model for the target through the ShuffleNetv2-YOLOv3 lightweight network;

The specific realization of this embodiment includes the following sub-steps:

Step 2.1: Take pictures of the target in the 3D engine to obtain the target photo;

Step 2.2: Detect the two-dimensional feature points in the target photo through the SIFT algorithm, and obtain the feature point set K={k ₁ ,...,k _n };

Step 2.3: Calculate the three-dimensional coordinate point p _i corresponding to each feature point k _i through the screen ray projection algorithm, and record the three-dimensional feature point set corresponding to the two-dimensional feature point as P={p ₁ ,...,p _n };

Step 2.4: The camera rotates around the target and continues to take pictures of the target, repeat steps 2.2 and 2.3 until the multi-view photos of the target are obtained and the three-dimensional feature point set PS={P ₁ ,...,P _N } is calculated, where, N is the number of photos, and P is the three-dimensional feature point set of each photo;

Step 2.5: Divide the target 3D voxel (Volume Pixel) into M blocks of the same voxel size B={b ₁ ,...,b _M }; set the frequency of 3D feature points appearing in each voxel block is the voxel block weight q, select m blocks with the largest weight KB={b ₁ ,...,b _m } as key voxel blocks, where m<M;

Step 2.6: Take the key voxel block as a category, calculate its area on the two-dimensional photo set according to the projection transformation formula, generate label information, and obtain a training data set;

Step 2.7: Train ShuffleNetv2-YOLOv3 through the generated data set to obtain a detection model for key voxel blocks of the target.

Step 3: Input the video stream into the trained ShuffleNetv2-YOLOv3 target key voxel block detection model, identify the key voxel blocks to obtain 2D-3D matching point pairs, and calculate the relative pose of the target in combination with the EPNP algorithm;

The specific realization of this embodiment includes the following sub-steps:

Step 3.1: Read the video stream, input the trained ShuffleNetv2-YOLOv3 detection model of key voxel blocks, and output the two-dimensional areas corresponding to several key voxel blocks;

Step 3.2: Calculate the center point of the detected two-dimensional area, and form a 2D-3D matching point pair with the center point of the corresponding key voxel block;

Step 3.3: Calculate the relative pose between the target and the mobile camera through RANSAC and EPNP algorithms.

Step 4: Calculate the pose of the mobile camera in the three-dimensional world through the mobile GPS and IMU information, and calculate the absolute pose of the target in the three-dimensional world by combining the relative position between the target and the mobile camera;

The specific realization of this embodiment includes the following sub-steps:

Step 4.1: Read mobile GPS and IMU data;

Step 4.2: Calculate the positioning of the mobile terminal in the three-dimensional world through the data obtained in step 4.1;

Step 4.3: Calculate the absolute pose of the target in the three-dimensional world through the pose of the mobile terminal calculated in step 4.2 and the relative pose of the target calculated in step 3.

The real-time pose estimation of the present invention is performed on the mobile end, reads the video stream from the mobile end camera, uses the ShuffleNet model to obtain the feature points of the target in the RGB image, and then obtains the relative pose through the RANSAC and EPNP algorithms. The present invention makes full use of the advantages of the mobile terminal, and calculates the pose of the target in the actual three-dimensional world through the positioning information provided by the GPS and IMU of the mobile terminal, combined with the relative pose.

It should be understood that the parts not described in detail in this specification belong to the prior art.

It should be understood that the above-mentioned descriptions for the preferred embodiments are relatively detailed, and should not therefore be considered as limiting the scope of the patent protection of the present invention. Within the scope of protection, replacements or modifications can also be made, all of which fall within the protection scope of the present invention, and the scope of protection of the present invention should be based on the appended claims.

Claims (4)

1. A mobile terminal three-dimensional pose estimation method based on point cloud preprocessing and ShuffleNet comprises the following steps:

step 1: performing three-dimensional reconstruction on target point cloud data obtained by laser scanning; importing a three-dimensional model obtained by three-dimensional reconstruction into a rendering engine for photographing;

step 2: respectively acquiring two-dimensional photos and camera poses of the target under different visual angles by adopting a positioning rotation photographing algorithm; extracting two-dimensional photo feature points through SIFT, calculating corresponding three-dimensional feature points, dividing a target model into equal-size voxel blocks, and screening target key voxel blocks according to the number of the three-dimensional feature points; generating two-dimensional projection of key voxel blocks on the photo set and establishing a training data set; training a ShuffLeNet characteristic detection model aiming at a target through a ShufflLeNet v2-YOLOv3 lightweight network;

the specific implementation of the step 2 comprises the following substeps:

step 2.1: shooting a target in a three-dimensional engine to obtain a target photo;

step 2.2: detecting two-dimensional feature points in the target picture through an SIFT algorithm to obtain a feature point set K = { K = ₁ ,...,k _n }；

Step 2.3: calculating each characteristic point k by a screen ray projection algorithm _i Corresponding three-dimensional coordinate point p _i And recording a three-dimensional feature point set corresponding to the two-dimensional feature points as P = { P = ₁ ,...,p _n }；

Step 2.4: the camera rotates around the target and continues to take a picture of the target, and the step 2.2 and the step 2.3 are repeated until a multi-view picture of the target is obtained and a three-dimensional feature point set PS = { P = { is calculated ₁ ,...,P _N N is the number of photos, and P is a three-dimensional feature point set of each photo;

step 2.5: dividing a target three-dimensional voxel into M blocks of the same voxel size B = { B = { (B) } ₁ ,...,b _M }; setting the frequency of the three-dimensional feature points appearing in each voxel block as a voxel block weight q, and screening m blocks KB = { b } with the maximum weight ₁ ,...,b _m As key voxel block, where m<M；

Step 2.6: calculating the region of the key voxel block on the two-dimensional photo set according to a projection transformation formula by taking the key voxel block as a category, and generating labeling information to obtain a training data set;

step 2.7: training ShuffleNet v2-YOLOv3 through the generated data set to obtain a detection model aiming at the target key voxel block;

and 3, step 3: inputting the video stream into a trained detection model of ShuffleNetv2-YOLOv3 target key voxel blocks, identifying the key voxel blocks to obtain 2D-3D matching point pairs, and calculating the relative pose of the target through RANSAC and EPNP algorithms;

and 4, step 4: and calculating the absolute pose of the target in the three-dimensional world by combining the GPS and IMU information of the mobile terminal.

2. The point cloud preprocessing and ShuffleNet-based mobile terminal three-dimensional pose estimation method according to claim 1, characterized in that: in the step 1, performing three-dimensional reconstruction on target point cloud data obtained by laser scanning through a Delauay algorithm to obtain a target three-dimensional model; and importing the target three-dimensional model into a rendering engine, calculating a target three-dimensional model bounding box and a central point, and moving the target three-dimensional model to enable the central point to reach the origin.

3. The method for estimating the three-dimensional pose of the moving end based on the point cloud preprocessing and the ShuffleNet according to claim 1, wherein the concrete implementation of the step 3 comprises the following substeps:

step 3.1: reading a video stream, inputting a trained detection model of ShuffleNetv2-YOLOv3 key voxel blocks, and outputting two-dimensional regions corresponding to a plurality of key voxel blocks;

step 3.2: calculating the detected central point of the two-dimensional region, and forming a 2D-3D matching point pair with the central point of the corresponding key voxel block;

step 3.3: and calculating the relative pose between the target and the mobile terminal camera through RANSAC and EPNP algorithms.

4. The method for estimating the three-dimensional pose of the moving end based on the point cloud preprocessing and the ShuffleNet according to any one of claims 1 to 3, wherein the specific implementation of the step 4 comprises the following sub-steps:

step 4.1: reading GPS and IMU data of a mobile terminal;

step 4.2: calculating the positioning of the mobile terminal in the three-dimensional world through the data acquired in the step 4.1;

step 4.3: and (3) calculating the absolute pose of the target in the three-dimensional world according to the moving end pose calculated in the step 4.2 and the relative pose of the target calculated in the step 3.

Images