CN108257089B - A method of the big visual field video panorama splicing based on iteration closest approach - Google Patents

Abstract

The invention relates to a method for video panorama stitching with a large field of view based on iterative closest points. The three-dimensional information of the scene is used to transform each frame view into the same image plane to realize scene stitching. Specifically, for all views of two adjacent frames, perform the following operations: 1. Extract and match the features of the views of the two adjacent frames; 2. Calculate the relative pose of the views of the two adjacent frames; 3. Under the epipolar constraints, for Dense matching of two adjacent frames of views; 4. According to the dense matching results, calculate the 3D model of the overlapping area of the adjacent two frames of views; 5. Use the iterative nearest point method to convert the 3D model obtained in step 4 to the 0th frame view. Under the camera coordinate system; 6. Project the converted 3D model onto the view plane of the 0th frame, so as to establish the mapping relationship between each point on the 3D model and the position on the 0th frame view; 7. Map the same position of the 0th frame view point fusion to complete the stitching. Compared with the traditional splicing method based on homography, the present invention is more real and reliable.

Description

A method for video panorama stitching with large field of view based on iterative closest point

technical field

The invention relates to a method for stitching a large-view video panorama based on the iterative nearest point, by estimating the depth value of each pixel point, mapping each pixel to the same image plane, and then merging the points mapped to the same position. belongs to the field of computer vision.

Background technique

Video panorama stitching is to seamlessly stitch each frame of view with overlapping areas in a video to obtain a panoramic image that reflects the overall picture of the video scene. Video panorama stitching technology is aimed at videos obtained by common imaging devices. The popularity of digital cameras and smartphones has greatly reduced the cost of panorama acquisition, and high-quality video panorama stitching technology has also exploded in huge market demand. In addition, video panorama stitching technology is also widely used in virtual reality, augmented reality and other fields, and its development prospects are very broad and have extremely high research value.

Traditional video panorama stitching techniques are based on the assumption of plane homography. Specifically, firstly extract and match the features with strong robustness for the views of two adjacent frames, use the features obtained by these matching to estimate the homography mapping matrix, and then map the pixels of one of the views through the estimated homography matrix. Go to another view, and fuse the gray values (if it is a color image, fuse the three RGB channels respectively, the same below), and get the stitching result. Traditional methods assume that all scenes are on the same plane during the stitching process of two views. In reality, the assumption that the scenes are all on the same plane is obviously not valid. When the camera distance is much greater than the change of the depth of the scene itself, the change of the depth of the scene itself can be ignored, and it is approximately considered that the scene is located on the same plane. At this time, the effect of the traditional video panorama stitching method is ideal. When the depth change of the scene itself cannot be ignored, the stitching result of the traditional method will produce large distortion, which makes the traditional method limited in practical application.

SUMMARY OF THE INVENTION

The technology of the present invention solves the problem: overcomes the deficiencies of the prior art, provides a method for panorama stitching of a large field of view video based on the iterative closest point, and uses the depth information of the scene to accurately map the pixels of different views onto the same image plane, The panorama distortion due to the planar homography assumption is fundamentally solved.

Technical solution of the present invention: a method for panorama stitching of large field of view video based on iterative nearest point, the implementation steps are as follows:

(1) Extract and match the features of two adjacent frame views;

(2) Based on the features obtained in step (1), calculate the relative poses of the views of two adjacent frames;

(3) The limit constraint is obtained through the relative pose calculation, and under the epipolar constraint, dense matching is performed on the views of two adjacent frames to obtain dense matching point pairs;

(4) using the dense matching point pairs obtained in step (3) to calculate the three-dimensional model of the overlapping area of two adjacent frames of views;

(5) Using the iterative closest point method, the three-dimensional model obtained in step (4) is transformed into the camera coordinate system of the 0th frame view, that is, the world coordinate system;

(6) project the three-dimensional model converted in step (5) into the 0th frame view, and establish the mapping relationship between each point on the three-dimensional model and the position on the 0th frame view;

(7) Based on the mapping relationship obtained in step (6), the points at the same position of the view in the 0th frame are fused to complete the splicing.

The implementation of extracting and matching the features of two adjacent frame views in step (1) is as follows:

(1) Extract SIFT feature points for two adjacent frame views, and calculate the descriptor of each feature point;

(2) Based on the descriptors of each feature, the features extracted from the views of two adjacent frames are matched to obtain a number of matching feature point pairs between the views of the two adjacent frames.

In step (4), the three-dimensional model of the overlapping area of two adjacent frames of views is calculated, and the implementation is as follows:

According to the principle of photographic ranging, the three-dimensional points corresponding to all matching point pairs between the two frames of views obtained in step (3) are calculated, and these three-dimensional points together form the three-dimensional model of the overlapping area of the adjacent two frames of views, and the three-dimensional model is M ^k represents each point in M ^k , h is the number of points in M ^k , and the superscript k represents that M ^k is a dense matching point pair composed of the kth frame view and the k+1th frame view, and the color of each point in M ^k is When calculating 3D points, match the mean of the point pair color.

In step (5), the iterative closest point method is used to transform the 3D model obtained in step (4) into the camera coordinate system of the 0th frame view. The implementation is as follows:

(11) The 3D model corresponding to the overlapping area of the 0th frame view and the 1st frame view is M ^0. Through the iterative nearest point method, the optimal rigid body transformation from M ^k to ^{M 0} is calculated. The translation vector T ^k is described;

(12) The optimal rigid body transformation is applied to each point in M ^k , as shown in formula (1),

in, for the The point transformed to the world coordinate system, i is used to index ^Mk (or defined below) at points; note It is the result of transforming M ^k to the camera coordinate system of the 0th frame view.

In step (6), the mapping relationship between each point on the three-dimensional model and the position on the 0th frame view is established, and the specific implementation is as follows:

(21) will The projection of each point in the middle to the 0th frame view is shown in formula (2):

In formula (2), π( ) is the perspective projection mapping, and [u, v] ^T is The coordinates of the projected position in the view at frame 0, where u and v are The abscissa and ordinate of the projection in the 0th frame view;

(22) Combine formula (2) and formula (1) to obtain the mapping between each point in M ^k and the position of the 0th frame view, as shown in formula (3):

In step (7), the points that map the same position of the 0th frame view are fused, and the specific implementation is as follows:

For all positions p of the view at frame 0, do the following:

(31) find out all the points that establish a corresponding relationship with position p through step (6), calculate the mean value of the color of the point that establishes a corresponding relationship with position p, and denote it as C;

(32) Assign the color of position p to C.

Compared with the prior art, the present invention has the advantage that: the present invention considers the three-dimensional information of the scene during panoramic stitching. Calculate the 3D model of the overlapping area of two adjacent frame views, convert the 3D model to the camera coordinate system of the 0th frame view, and then establish an accurate mapping of each point on the 3D model to the position in the 0th frame view, so as to obtain a more realistic , natural panorama. However, the traditional method assumes that all the scenes are located on the same plane, and establishes the mapping relationship through the homography matrix, ignoring the three-dimensional information of the scene, thereby obtaining an inaccurate mapping relationship. When the depth of field changes sharply, the stitching process of the traditional method will generate a large number of defects, resulting in a poor quality panorama.

Description of drawings

Figure 1 shows the flow of a large-scale video panorama stitching method based on iterative closest points;

FIG. 2 shows the experimental result of panorama stitching of a certain video according to the present invention, (a) several frames are intercepted from the video used in the experiment; (b) the stitching result of the present invention.

specific implementation

The present invention will be described in detail below with reference to the accompanying drawings and embodiments. For the convenience of description, the present invention uses the symbol k to index each frame view of the video, and the kth frame view and the k+1th frame view are adjacent views.

As shown in Figure 1, the present invention is implemented as follows:

1. Extract and match the features of two adjacent frame views;

Image features refer to some pixels with a certain type of properties in a digital image. Each image feature often corresponds to a descriptor (feature vector), which is used to describe the feature. Common image features include FAST, HOG, SURF, SIFT, etc. Considering that the pose calculation has a high requirement on the robustness of the feature, the present invention selects the SIFT feature.

The basis of feature matching is the descriptor of the feature, specifically, record and are the features extracted from the kth and k+1th frame views, respectively, where n and m are the number of features in the kth and k+1th frame views, respectively. Denote D( ) as the descriptor operator, then and The descriptors of are and If the feature in the kth frame view (0≤l≤n-1) and features in the k+1th frame view is the matching feature, then and The conditions shown in formula (4) must be satisfied.

The ||·|| symbol in formula (4) represents the Euclidean distance operator, and min(·) represents the minimum value operator. Assuming that after matching, s groups of matching features can be obtained, which are uniformly recorded as (x ₀ , x′ ₀ ), (x ₁ , x′ ₁ ),...,(x _s-1 ,x′ _s-1 ).

2. Calculate the relative pose of two adjacent frame views

Denote the fundamental matrix of the kth frame view relative to the k+1th frame view as F, then the matching feature obtained in step 1 should satisfy the epipolar constraint equation Both x′ _t and x _t are homogeneous coordinates, t=0,1,...,s-1, which are used to index the matching features obtained in step 1. When s≥8, F can be estimated by singular value decomposition. Calculate this matrix E from the camera internal parameter matrix K and the estimated F, and decompose the singular value of E to obtain the pose of the k+1th frame view relative to the kth frame view, which is described by a rotation matrix and a translation vector.

3. Under the epipolar constraint, densely match the views of two adjacent frames

The purpose of dense matching is to match the pixels in the kth frame view as much as possible to obtain the corresponding pixels in the k+1th frame view under the condition of satisfying the epipolar constraints. The matching is based on the characteristics of the pixel points. remember and represent the homogeneous coordinates of the pixels in the kth frame and the k+1th frame view, respectively, where, and respectively The abscissa and ordinate on the kth frame view; and respectively The abscissa and ordinate on the view of the k+1th frame; i and j are the indices of the view pixels of the kth and k+1th frames, respectively. The pixel point feature operator is denoted as V(·). right In the pixels located in the k+1 frame view and satisfying the epipolar constraint, search for the The pixel with the closest feature As its matching point, as shown in formula (5):

In formula (5), argmin represents the operator to obtain the minimum parameter. The second row of formula (5) is the limit constraint, and its geometric meaning is The distance to the epipolar line is less than ε. Among them, F represents the fundamental matrix, which is recalculated from the pose obtained in step 2.

4. Use the pixels obtained by dense matching in step 3 to calculate the three-dimensional model of the overlapping area of two adjacent frames of views;

Through the pixels in the kth frame view coordinates and camera parameters, you can get a line starting from the kth frame view optical center, and ray; similarly, according to the pose obtained in step 2, you can also get a view optical center starting from the k+1th frame, pointing to A ray of matching points. By solving the intersection of these two rays, we can get Corresponding 3D position The above calculation The principle is called the principle of photometric ranging. The grayscale (or color) of ( will be used in step 7), defined as the average of its matching point grayscales (if the processing is color video, The RGB channels are to match the average of the dot RGB channels). Converting all the pixels with matching points in the kth frame view into the corresponding 3D position, the 3D model of the overlapping scene of the kth frame and the k+1th frame view can be obtained, denoted as where h is the number of points contained in ^Mk ).

5. Using the iterative closest point method, transform the 3D model obtained in step 4 into the same coordinate system;

For the convenience of description, in the following description, the camera coordinate system corresponding to the 0th frame view is referred to as the world coordinate system (that is, the coordinate system where M ⁰ is located). The purpose of this step is to convert M ^k to the world coordinate system. In this step and step 6, i represents M ^k (or The index of each point in the definition see below). Using the iterative closest point method, M ^k is registered to M ⁰ , resulting in a rotation matrix R ^k and a translation vector T ^k . R ^k and T ^k are rigid body transformations that transform each point in M ^k to the world coordinate system, as shown in equation (6),

in, for the The point after transformation to the world coordinate system; note It is the result of transforming M ^k to the world coordinate system.

6. Project the three-dimensional model transformed in step 5 into the 0th frame view, thereby establishing a mapping relationship between each point on the 3D model and the position on the 0th frame view.

right Will Projected into the first view, there are:

In formula (4), π( ) is the perspective projection mapping, and [u,v] ^T is The coordinates of the projected position in the view at frame 0, where u and v are The abscissa and ordinate of the projection in the view at frame 0. Formula (7) put It is mapped to the position of [u, v] ^T in the view of frame 0. Substituting equation (6) into (7), we get The mapping to the position where the view coordinates of the 0th frame are [u, v] ^T , as shown in formula (8):

According to formula (8), the mapping of each point in M ^k to the position in the 0th frame view is established.

7. Fuse all pixels mapped to the 0th frame view.

Denote the number of video frames as n _frames , and for k=1, 2, . . . , n _frames -1, perform steps 1 to 6. Denote the coordinates of a pixel in the 0th frame view as [u,v] ^T , where u is the abscissa and v is the ordinate. The set of all points mapped to the position of [u, v] ^T in the 0th frame is called the support set of [u, v] ^T , denoted as U(u, v). Write U(u,v)={X ₀ ,X ₁ ,...,X _r-1 } (r is the number of points in U(u,v)), if U(u,v) is not empty ( That is, r>0), then the grayscale (or color) at [u, v] ^T is:

Among them, C(X _i ) represents the color of Xi ₍ see step 4 for definition), is the color at the view [u, v] ^T of the 0th frame, according to the definition of formula (9), is the average value of C(X _i ). For all positions where the support set is not empty in the 0th frame view, the grayscale is recalculated to obtain the grayscale of the corresponding position, stitching is realized, and the panorama is obtained.

Figure 2 shows the experimental results of the present invention, (a) is a four-frame view intercepted from the experimental video; (b) is the result of panoramic stitching of the experimental video using the present invention. It can be seen that the panorama image obtained by splicing by the method of the present invention reflects the whole picture of the scene, and the transition is natural and the degree of authenticity is high.

The above embodiments are provided for the purpose of describing the present invention only, and are not intended to limit the scope of the present invention. The scope of the invention is defined by the appended claims. Various equivalent replacements and modifications made without departing from the spirit and principle of the present invention should be included within the scope of the present invention.

Claims (5)

1. a method for stitching a large field of view video panorama based on iterative nearest point, is characterized in that, comprises the following steps: (1) Extract and match the features of two adjacent frame views; (2) Based on the features obtained in step (1), calculate the relative poses of the views of two adjacent frames; (3) The epipolar constraint is obtained by calculating the relative pose, and under the epipolar constraint, dense matching is performed on the views of two adjacent frames to obtain dense matching point pairs; (4) using the dense matching point pairs obtained in step (3) to calculate the three-dimensional model of the overlapping area of two adjacent frames of views; (5) Using the iterative closest point method, the three-dimensional model obtained in step (4) is transformed into the camera coordinate system of the 0th frame view, that is, the world coordinate system; (6) project the three-dimensional model converted in step (5) into the 0th frame view, and establish the mapping relationship between each point on the three-dimensional model and the position on the 0th frame view; (7) Based on the mapping relationship obtained in step (6), the points at the same position of the 0th frame view are fused to complete the splicing; The implementation of extracting and matching the features of two adjacent frame views in step (1) is as follows: (1) Extract SIFT feature points for two adjacent frame views, and calculate the descriptor of each feature point; (2) Based on the descriptors of each feature, the features extracted from the views of two adjacent frames are matched to obtain a number of matching feature point pairs between the views of the two adjacent frames. 2. the method for the large field of view video panorama stitching based on iterative nearest point according to claim 1, is characterized in that: in step (4), calculate the three-dimensional model of adjacent two frame views overlapping area, realize as follows: According to the principle of photographic ranging, the three-dimensional points corresponding to all matching point pairs between the two frames of views obtained in step (3) are calculated, and these three-dimensional points together form a three-dimensional model of the overlapping area of two adjacent frames of views, and the three-dimensional model is M ^k , represents each point in M ^k , h is the number of points in M ^k , and the superscript k represents that M ^k is a dense matching point pair composed of the kth frame view and the k+1th frame view, and the color of each point in M ^k is When calculating 3D points, match the mean of the point pair color. 3. the method for the large field of view video panorama stitching based on the iterative closest point according to claim 2, is characterized in that: utilize the iterative closest point method in step (5), the three-dimensional model that step (4) obtains is transformed into . The implementation in the camera coordinate system of the 0th frame view is as follows: (11) The 3D model corresponding to the overlapping area of the 0th frame view and the 1st frame view is M ⁰ through the iterative nearest point method, and the optimal rigid body transformation from M ^k to M ⁰ is calculated, and the optimal rigid body transformation is obtained through the rotation matrix R ^k and The translation vector T ^k is described; (12) The optimal rigid body transformation is applied to each point in M ^k , as shown in formula (1), in, for the Point after transformation to world coordinate system, i is used to index M ^k or It is the result of transforming M ^k to the camera coordinate system of the 0th frame view. 4. the method for the large field of view video panorama stitching based on the iterative nearest point according to claim 3, is characterized in that: in step (6), establish the mapping relationship between each point on the three-dimensional model and the position on the 0th frame view, The specific implementation is as follows: (21) will The projection of each point in the middle to the 0th frame view is shown in formula (2): In formula (2), π( ) is the perspective projection mapping, and [u, v] ^T is The coordinates of the projected position in the view at frame 0, where u and v are The abscissa and ordinate of the projection in the 0th frame view; (22) Combine formula (2) and formula (1) to obtain the mapping between each point in M ^k and the position of the 0th frame view, as shown in formula (3): 5. the method for the large field of view video panorama stitching based on the iterative nearest point according to claim 1, is characterized in that: in step (7), the point fusion of the same position of the 0th frame view of the mapping, concrete realization is as follows: For all positions p of the view at frame 0, do the following: (31) find out all the points that establish a corresponding relationship with position p through step (6), calculate the mean value of the color of the point that establishes a corresponding relationship with position p, and denote it as C; (32) Assign the color of position p to C.