CN112580496B - Face Relative Pose Estimation Method Combined with Face Keypoint Detection - Google Patents

Abstract

The invention discloses a method for estimating the relative pose of a human face by combining key points of the human face and RGB-D data. This method recognizes the key points of the face on the RGB image and calculates the initial pose angle difference, and combines the depth data to match the RGB-D point cloud to obtain a relatively high-precision relative pose of the face between different angles. estimate. Different from the previous recognition method, this method uses RGB-D data, and has lower requirements on the size of the data set, and can achieve high-precision relative pose estimation on small sample data. Moreover, it also combines face key point information to enhance the robustness of pose estimation.

Description

Face Relative Pose Estimation Method Combined with Face Keypoint Detection

technical field

The invention belongs to image recognition, human face key point recognition, and human face pose estimation, and particularly relates to a method for estimating relative face pose based on RGB-D data and human face key points.

Background technique

The relative pose of faces has important research value. Its goal is to accurately estimate the relative pose between a pair of face RGB-D data from different angles. Face relative pose estimation has the following main application scenarios:

(1) Attention detection: By judging the head posture, the human attention can be judged. For example, it can detect whether the long-distance driver is looking ahead. If he does not look ahead for a long time, he can beat in advance to ensure safety and reduce accidents; another example is to monitor whether students are concentrating in class.

(2) Behavior analysis. Similar to (1), video surveillance analysis and other algorithms can be used to judge whether a person has misbehavior, so as to achieve early warning and prevent problems before they happen.

(3) Human-computer interaction. Human head movements can sometimes express meaning and convey information. Shaking the head means denial in most people's eyes, nodding means agreeing, and bowing the head for a long time means that you may be thinking about a problem. If robots can understand such behaviors, it will improve the quality and effectiveness of human-robot interactions.

(4) Gaze tracking, also known as eye tracking. Accurate head pose estimation can improve the accuracy of gaze tracking. Eye tracking can be used in the field of games. For example, the movement of characters in the game can be controlled with the eyes, so that the somatosensory operation can be improved to a higher level.

For face relative pose estimation, it can be roughly divided into two genres. The first is a feature-based approach. In this method, the traditional method is used to analyze the features of the human face and compare them with the features of the standard frontal face to obtain the difference between the two features, and the relative pose estimation can be obtained by analyzing the difference between the features.

The other is a method mainly based on deep learning. By designing deep learning networks with different structures, the features of the current face are extracted and the angle is returned to obtain the pose of the current face. With the development of computing resources, such methods have gradually become popular, but they are greatly limited by the size of the model and the requirements for accuracy.

Point cloud can accurately reflect the real size and shape structure of the object surface in three-dimensional space. It is a relatively primitive three-dimensional structural representation of objects. As data acquisition becomes easier, it has gradually become a comparison in the field of computer vision. important data structures.

Contents of the invention

The object of the present invention is to provide a method for estimating the relative pose of a human face based on RGB-D data combined with key point detection of the human face. This method implements the process of relative pose estimation for a group of faces from different angles by means of key point detection and the introduction of point pair features to iterate the closest point. In the present invention, using the point cloud to estimate is more accurate than simply using RGB images to estimate the result, and fusing the process of relative attitude estimation into the point cloud matching process can maximize the accuracy of the result.

In order to achieve the above object, the technical solution of the present invention is: a method for estimating the relative pose of a human face combined with key point detection of a human face. The method is as follows: firstly, 68 key points of the human face are extracted by using the key point detection technology of the human face, and the first relative attitude estimation is carried out by using the matching between the three-dimensional coordinates of two groups of key points. On the basis of the first time, the relative pose estimation of the second time has a better initial value, and at the same time, the face is cropped to optimize the preprocessing before iterating the nearest point algorithm, and finally the point pairs of six-dimensional coordinates The features are matched iteratively, and the relative pose is estimated during the matching process.

Specifically, the inventive method comprises the following steps:

Step 1: In the case of the existing RGB-D data, use the Dlib face key point detection CNN model to carry out two corresponding angle RGB images I _source and I _target , including the facial contour, mouth, eyes, nose and other parts. 68 face key points are detected, and a plurality of RGB-D three-dimensional coordinate points {KeyPoint _snum } and {KeyPoint _tnum } corresponding to the key points are selected, and two sets of three-dimensional coordinate points are matched to obtain the initial rotation matrix R of the relative estimation of the face pose _start .

Step 2: According to the two-dimensional key point coordinates in step 1, regions of different sizes are cropped in the RGB images I _source and I _target for matching. The purpose of the present invention is to calculate the relative pose of I _target relative to I _source , so the clipping area I' _source of I _source is larger than the clipping area I' _target of I _target .

Step 3: Down-sample the face regions I' _source and I' _target corresponding to the two postures obtained in step 2 to obtain the corresponding face regions I" _source and I' _target . For the points in I' _target , select Among them, the point set t∈I′ _target , and select the point set s∈I″ _source in the I″ _source at the same time. After obtaining the point set, calculate each point P _i ∈ t in the point set and all other points in the point set P _j ∈ t and i Point-to-feature FP _ij of ≠ j, get the matrix FP _mat composed of FP, the size is N _P *NP, where N _P is the number of points in the point set _. Calculate the point-to-feature FQ _ij composition of Q _i ∈ s in the same way The matrix FQ _mat with a size of N _Q *N _Q. Hash the features FP _ij and FQ _ij in the point pair feature matrix FP _mat and FQ _mat and save them in the hash table H. For the most matching point pair feature Assign the same hash value, thus find the corresponding point pairs in the two point sets t and s and form point sets, respectively PP _t and PP _s . Then use the method of singular value decomposition to calculate PP _t and PP _s Between the rotation matrix R and the space translation T. Iterate until ||PP _s -PP _t || is less than a certain threshold ε or the number of iterations exceeds a certain limit Iter _max to stop the iteration.

Step 4: Perform matrix multiplication calculation on the rotation matrix R _start obtained in step 2, the rotation matrix R obtained by multiple iterations in step 3, and the spatial translation T to obtain the rotation matrix R _final . Using the transformation relationship between the rotation matrix and the Euler angles, the estimated result of the relative pose {angle _x , angle _y , angle _z } is obtained.

Further, the Dlib key point detection used in step 1 uses the dlib.get_frontal_face_detection() function in the Python third-party library dlib for face detection, and uses the model shape_predictor_68_landmarks.dat for key point detection.

Further, in the step 2, select 5 groups of key points {KeyPoint _snum } and {KeyPoint _tnum } that are not on the same plane and are relatively stable and not easy to be blocked for matching. The number of matching points is fixed as snum, tnum=31, 37, 46, 49, 55, respectively 2 outer corners of the eyes, 2 outer corners of the mouth, and the tip of the nose.

Further, for the matching of two groups of key points (KeyPoint _snum } and (KeyPoint _tnum } in the step 2, the matching method used comes from the affine transformation function estimateAffine3D() in OpenCV.

Further, the clipping of the face area in the described step 3 is based on the two-dimensional coordinates of the key points of the face, and the method is as follows:

The clipping area of the face is a rectangle, and the four sides of the rectangle are determined by the maximum and minimum values in the two dimensions of the key point coordinates {X _snum , Y _snum } and {X _tnum , Y _tnum }, respectively (Rec _Left = min{X _snum }, Rec _Right ＝max{X _snum }, Rec _Up ＝min{Y _snum }, Rec _Bottom ＝max{Y _snum }}. Wherein, the boundary of I _source clipping area I' _source is in 4 most value On the basis of , expand 5 pixels outward, which are {Rec _Left -5, Rec _Right +5, Rec _Up -5, Rec _Bottom +5}

On the basis of the 4 most values, the boundary of the I _target clipping area I′ _target shrinks inward by 5 pixels, which are respectively {Rec _Left +5, Rec _Right -5, Rec _Up +5, Rec _Bottom -5}.

Further, in the step 3, the point pair features in the point set can be calculated using six-dimensional point cloud data (x, y, depth, R, G, B} to obtain ten-dimensional feature descriptors, or three-dimensional point cloud data can be used (x, y, depth} to obtain a four-dimensional feature descriptor, reducing the time required for feature generation and occupying space.

表示RGB向量，PPF(p_i，p_j，n_i，n_j)＝(||d||₂，∠(n_i，d)，∠(n_j，d)，∠(n_i，n_j))，其中d＝p_i-p_j，p_i，p_j表示物体表面选取点的坐标，n_i，n_j表示选取点的法向量，∠表示向量间的夹角，CPPF表示带有颜色的PPF。For a detailed description of point pair features, FP _ij = CPPF(PPF(p _i , p _j , n _i , n _j ), c _i , c _j ), where

Indicates RGB vector, PPF(p _i , p _j , n _i , n _j )=(||d|| ₂ , ∠(n _i , d), ∠(n _j , d), ∠(n _i , n _j )), where d=p _i -p _j , p _i , p _j represent the coordinates of the selected point on the surface of the object, n _i , n _j represent the normal vector of the selected point, ∠ represents the angle between the vectors, CPPF represents the color The PPF.

Further, before the iteration, the data is preprocessed for downsampling, using random sampling, which is selected from the sample() function in the Python module random. Using different ratios of downsampling can trade off between iteration speed and iteration accuracy, the general ratio is b=0.1, 0.3, 0.5, 0.8

Furthermore, the method of finding the nearest point is used in the iterative process. This method selects the Python third-party module sklearn module to select the nearest point. The function used is neighbors.NearestNeighbors()

Further, the method of finding the rotation matrix and spatial translation is used in the iterative process. This method chooses the method of singular value decomposition, which is selected from the function numpy.linalg.svd() of the Python language extension library numpy

Furthermore, in the iterative process, after each calculation of the rotation matrix R and the spatial translation T, it is necessary to transform the data (x, y, depth) of the I' _target so that it updates its position before the next iteration so that Find the new closest point.

The beneficial effects of the present invention are:

(1) Through the realization of point cloud matching, the relative pose estimation is integrated into the process of point cloud matching, so that the accuracy is greatly improved compared with the simple extraction of features and estimation

(2) Through the detection and initial matching of the key points, a good initial value is provided for the subsequent iterative closest point process, which avoids the direct use of the iterative closest point process to cause the algorithm to iterate in the wrong direction or fail to converge

(3) By clipping the face area, the number of target point cloud points is relatively small, and the number of source point cloud points is relatively large. This can enhance the robustness of the matching process and reduce the possibility of non-convergence

(4) Through the introduction of point pair features, the disadvantages of using Euclidean distance to find corresponding points in the original ICP algorithm are reduced. The introduction of point pair features can better and more accurately find corresponding points for point cloud matching.

Description of drawings

Fig. 1 is the flow chart of the steps of the method for estimating the relative pose of a human face combined with key point detection of a human face according to an embodiment of the present invention;

Fig. 2 is the loss statistics of the relative pose estimation according to the embodiment of the present invention.

specific implementation plan

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

On the contrary, the invention covers any alternatives, modifications, equivalent methods and schemes within the spirit and scope of the invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Without the description of these detailed parts, the present invention can be fully understood by those skilled in the art.

Referring to FIG. 1 , it is a flow chart of steps of a method for estimating a relative pose of a human face combined with key point detection of a human face according to an embodiment of the present invention.

For source data Data _source ={source image Image _source , source depth data Depth _source } and target data Data _source ={target image Image _target , target depth data Depth _target }, it is processed in the following steps:

1. Detect face key points. specifically:

(1.1) Import the third-party library Dlib, and import the face key point detection model shape_predictor_68_landmarks.dat

(1.2) Send the Image _source and Image _target into the detection model to identify key points, and obtain the key point coordinates {X _snum , Y _snum } and {X _tnum , Y _tnum } of the face in the image, where snum, tnum =1,...,68

2. Select relatively stable key points for preliminary estimation of relative positions. specifically:

(2.1) Select 5 relatively stable key points, namely 2 outer corners of the eyes, 2 outer corners of the mouth, and the tip of the nose. The corresponding key points are numbered snum, tnum=31, 37, 46, 49, 55, and two groups are obtained keypoint {KeyPoint _snum } and {KeyPoint _tnum }

(2.2) After obtaining two sets of key points {KeyPoint _snum } and {KeyPoint _tnum }, import the third-party library Opencv, use the estimateAffine3D() function to register the two sets of key points, and get the initial rotation matrix R _start

3. Crop the face area. Crop the face of the source image into a larger area, and the face of the target image into a smaller area, specifically:

(3.1) The cropped area of the face is a rectangle, and the cropped data includes not only RGB, but also Depth data. The four sides of the rectangle consist of the key point coordinates {X _snum , Y _snum } and the maximum and minimum values on the two dimensions of {X _tnum , Y _tnum }, a total of 4 values, respectively (Rec _Left ＝min{X _snum }, Rec _Right = max(X _snum }, Rec _Up = min(Y _snum }, Rec _Bottom = max{Y _snum }}.

(3.2) I _source crops a relatively large area, so the boundary of the cropped area I' _source is expanded by 5 pixels on the basis of the above-mentioned 4 most values, which are respectively {Rec _Left -5, Rec _Right +5, Rec _Up -5, Rec _Bottom +5}

(3.3) The I _target crops a relatively small area, so the boundary of the cropped area I' _target shrinks inward by 5 pixels on the basis of the above-mentioned 4 most values, respectively (Rec _Left +5, Rec _Right -5, Rec _Up +5, Rec _Bottom -5}.

4. Match the point cloud data of the clipping area, specifically:

(4.1) The cropped data contains (x, y, depth, R, G, B}. The matching process uses the method of iterating the closest point. Before the iteration, the data will be down-sampled and preprocessed, using random Sampling, selected from the sample() function in the Python module random.Using different proportions of downsampling can trade-off between iteration speed and iteration accuracy, the general proportion is b=0.1,0.3,0.5,0.8

(4.2) The point cloud after downsampling can be iteratively matched to the nearest point to get the final result. To obtain the corresponding face area I″ _source and I′ _target , first, for the points in I′ _target , select the point set t∈I′ _target , and select the point set s∈I″ _source in I″ _source at the same time. Get After the point set, calculate the point pair feature FP _ij of each point P _i ∈ t in the point set and all other points P _j ∈ t and i≠j in the point set, and obtain the matrix FP _mat composed of FP, the size of which is N _P * _NP , Where N _P is the number of points in the point set. In the same way, calculate the matrix FQ _mat composed of point pair features FQ _ij of Q _i ∈ s, and the size is N _Q *N _Q , where N _Q is the number of points in the point set.

(4.3) Hash the features FP _ij and FQ _ij in the point pair feature matrix FP _mat and FQ _mat and save them in the hash table H, and assign the same hash value to similar point pair features, thus To find the corresponding point pairs in the two point sets t and s, respectively PP _t and PP _s . Then use the method of singular value decomposition to calculate the rotation matrix R and space translation T between PP _t and PP _s . Iterate until ||PP _s -PP _t || is less than a certain threshold ε or the number of iterations exceeds a certain limit Iter _max , then stop iterating.

5. Synthesize the results of the two registrations. specifically:

The initial rotation matrix R _start is obtained in step (2.2), and a series of rotation matrices R _i in the iterative process are obtained in step (4.3), where i=1, . . . , N. Multiply the matrix to get the final matrix result R _final . In the case of the existing R _final , using the transformation relationship between the rotation matrix and the Euler angle, the estimation result of the relative attitude can be finally obtained (angle _x , angle _y , angle _z }

Figure 2(a) shows the loss distribution of the rotation angle pitch of a pair of faces with different poses relative to pose estimation. The horizontal axis is the loss size, the unit is °, and the scale is 0.1°. The vertical axis is the number of samples, and the total number of samples is about 200.

Figure 2(b) shows the loss distribution of the rotation angle yaw of the relative pose estimation of a pair of faces with different poses. The horizontal axis is the loss size, the unit is °, and the scale is 0.1°. The vertical axis is the number of samples, and the total number of samples is about 200.

Figure 2(c) shows the loss distribution of the rotation angle roll of the relative pose estimation of a pair of faces with different poses. The horizontal axis is the loss size, the unit is °, and the scale is 0.1°. The vertical axis is the number of samples, and the total number of samples is about 200.

The above results show that the accuracy of the pose estimation results using the method of the present invention is very high.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. within range.

Claims (5)

1. A human face relative attitude estimation method combining human face key point detection is characterized by comprising the following steps:

step 1: for two RGB images I with corresponding angles _source And I _target Face key point detection is carried out, and a plurality of RGB-D three-dimensional coordinate points { KeyPoint' corresponding to key points are selected from the face key point detection _snum And { KeyPoint }and { KeyPoint _tnum And matching two groups of three-dimensional coordinate points to obtain an initial rotation matrix R of face pose relative estimation _start ；

Step 2: according to the two-dimensional key point coordinates in the step 1, in the RGB image I _source And I _target Respectively cutting areas with different sizes for matching; wherein I _source Region I 'of cutting' _source Is greater than I _target Region I 'to be cut' _target ；

And step 3: obtaining face regions I 'corresponding to the two postures in the step 2' _source And I' _target Down sampling is carried out to obtain a corresponding face area I ″) _source And I ″) _target For I ″) _target Selecting the point in the point set t belongs to I ″) _target While at I ″) _source Wherein the selected point set s belongs to I ″) _source (ii) a Calculate each point P in the set of points _i E t and all other points P in the point set _j Point pair characteristics FP of e t and i ≠ j _ij To obtain FP _ij Composed matrix FP _mat Size is N _P *N _P In which N is _P The number of points in the point set; q is calculated in the same manner _i E s point pair characteristic FQ _ij Composed matrix FQ _mat Size is N _Q *N _Q (ii) a Point pair feature matrix FP _mat And FQ _mat Feature FP in _ij And FQ _ij Hashing and storing the hashed values into a hash table H, and distributing the same hash value to the characteristics of the most matched point pairs, thereby finding corresponding point pairs in two point sets t and s and forming point sets which are respectively PP _t And PP _s (ii) a Calculating PP by singular value decomposition _t And PP _s A rotation matrix R and a spatial translation T in between; iterating until | | PP _s -PP _t Stopping iteration when | | is less than a certain threshold value epsilon or the iteration times reach a certain number;

and 4, step 4: the rotation matrix R obtained in the step 1 is processed _start Performing matrix multiplication calculation with the rotation matrix R obtained by iteration for multiple times in the step 3 and the space translation T to obtain a rotation matrix R _final (ii) a Obtaining the estimation result of the relative attitude { angle using the transformation relation between the rotation matrix and the Euler angle _x ，angle _y ，angle _z }。

2. The method according to claim 1, wherein in step 1, two groups of key points { KeyPoint ™ _snum And { KeyPoint }and { KeyPoint _tnum Match, number of matching points is fixed to snum, tnum =31,37,46,49,55, 2 external canthus, 2 external mouth canthus and nose tip respectively.

3. The method according to claim 1, wherein the clipping of the face region in step 2 is based on two-dimensional coordinates of key points of the face, and the method specifically comprises the following steps:

the cutting area of the face is a rectangle, and the four sides of the rectangle are provided with key point coordinates { X _snum ，Y _snum And { X } _tnum ，Y _tnum Maximum and minimum decisions in two dimensions of the }; wherein, I _source Region I 'of cutting' _source Is expanded by 5 pixels on the basis of 4 maxima, and I _target Region I 'of cutting' _target The boundary of (2) is further shrunk by 5 pixels on the basis of 4 maxima.

4. The method according to claim 1, wherein in the step 3, the point pair feature in the point set is calculated by using six-dimensional point cloud data { x, y, depth, R, G, B } to obtain a ten-dimensional feature descriptor: FP _ij ＝CPPF(PPF(p _i ，p _j ，n _i ，n _j )，c _i ，c _j ) Or a four-dimensional feature descriptor obtained by using three-dimensional point cloud data { x, y, depth }: FP _ij ＝PPF(p _i ，p _j ，n _i ，n _j ) (ii) a Wherein c is _i ，

Representing an RGB vector, PPF (p) _i ，p _j ，n _i ，n _j )＝(||d|| ₂ ，∠(n _i ，d)，∠(n _j ，d)，∠(n _i ，n _j ) Wherein d = p) _i -p _j ，p _i ，p _j Coordinates representing selected points of the surface of the object, n _i ，n _j Normal vectors of selected points are represented, an included angle between vectors is represented, and CPPF represents colored PPF.

5. The method according to claim 1, wherein in step 3, in the iterative process, after each calculation to obtain the rotation matrix R and the spatial translation T, the pair I ″ "is required _target Is transformed so that it updates its position before the next iteration to form a new point pair feature.

Images