CN100448271C - Video Editing Method Based on Panorama Stitching - Google Patents

Abstract

The present invention discloses a video editing method based on concatenation of panoramagram, being used for editing or repairing section series of motion video. The method includes the steps as follows: matching with the characteristic point of the every frame image of the video series, calculating the projection matrix between the frames and appending the image to acquire the video panoragram; editing the video panoragram with the human interaction and computer according to the specific demand of video editing; restoring the video series from the edited panoragram based on the projection relationship. Applying the video panoragram as the medium, the present invention shifts the video editing into the image editing of vider panoragram, not only directly providing the whole scene message to the user, but also reducing the calculating quantity and the human interaction.

Description

Video Editing Method Based on Panorama Stitching

technical field

The invention relates to a video sequence processing method, in particular to a new method for replacing traditional video editing with image editing of the panorama by synthesizing video panoramas.

Background technique

In traditional video editing, videotapes are usually used as the storage medium for lens shooting and editing. Since the materials are stored sequentially on the videotapes, it is necessary to search and copy repeatedly to complete the editing, and rearrange these materials on another videotape. This editing method It is called the linear editing method. Since the development of digital technology, a dedicated non-linear editing machine has appeared, which can be more conveniently processed without following the linear position of the material on the tape. In fact, the PC can also be used as a non-linear editing machine, because all the material is captured to disk, and can be processed at any time in any timeline position.

Non-linear video editing is largely convenient for video editing, but it is still a very cumbersome task, because the existing non-linear video editing is to expand video files frame by frame and edit them with frame precision. Due to the huge amount of video data, frame-by-frame editing of videos requires a lot of human interaction and computer calculations.

Since the objects in the video usually appear in many frames, if you directly modify the video, you need to do it frame by frame, and there is a lot of repetitive work. If the whole segment of video information is represented by an image, the image is edited as required, and then the video is re-obtained based on the edited image, not only the workload of manual participation is greatly reduced (it can even be completely automated by the computer). completed), it also saves calculation time and improves work efficiency. Therefore, the present invention proposes a new video editing method based on panorama mosaic.

Contents of the invention

The purpose of the present invention is to provide a video editing method based on panorama stitching, which solves the disadvantages of editing video frame by frame in the existing video editing method, which is not intuitive, requires a large amount of calculation, and takes a long time, and provides a fast and intuitive method. A method for editing video content.

In order to achieve the above object, the technical scheme adopted in the present invention is:

1. The present invention provides a video content editing method based on panorama mosaic, which is used for editing a moving video sequence. The method includes:

1) Generate a video panorama describing the overall picture of the motion video with a plurality of video frames;

2) Editing the image content of the obtained video panorama;

3) Back-project the edited video panorama back to the coordinate system of each video frame to generate an edited video sequence.

2. The generation of the video panorama comprises the following steps:

1) Carry out global motion estimation to the relative global motion between a plurality of video frames, obtain the planar projection relation between each video frame image;

2) If there are moving objects in the moving video sequence, remove them first;

3) According to the planar projection relationship between each video frame image, the first frame image is used as a reference frame to establish a panoramic coordinate system, and each video frame image is projected into the panoramic coordinate system, and the size of the panoramic image is estimated ;

4) According to the planar projection relationship between each video frame, calculate the corresponding point of each image point in multiple video frame images on the panorama, sort the multiple corresponding points, and take the median value as the value on the panorama , forming a video panorama;

3. The global motion estimation includes:

1) Matching step: extract the corner points of each video frame image, and perform correlation matching to obtain an initial matching point set;

2) Parameter estimation step: Use Ransac to eliminate the wrong matches in the initial matching point set, and use least squares to estimate the transformation parameters under perspective projection.

4. The moving object removal method includes:

1) Use the frame difference method to determine the approximate range of the moving object;

2) Using color-based region segmentation to divide the image into regions of different colors;

3) Combine the two with the graph cutting method, and use the segmentation result of the previous frame as a constraint to optimize the solution.

5. It also includes color brightness correction for each video frame image to eliminate color differences caused by different exposure and white balance during shooting.

6. The method for editing panorama content includes:

1) Image transplantation: select an area manually, then put the information of this area into the area to be filled, and change the color of the original area information according to the information outside the filled area, so that this filling becomes natural;

2) Image editing based on information propagation: use the known information around the boundary of the area to be edited, and "propagate" the gray information on the boundary into the area to be edited along the direction of the smallest gradient;

3) Semi-automatic filling of texture images: Automatically fill textured areas.

7. The generation of the edited video sequence comprises the following steps:

1) Calculate the backprojection matrix from the video panorama to each video frame coordinate system;

2) Generate each video frame image from the video panorama according to the back-projection matrix, and complete the video editing process.

The beneficial effects that the present invention has are:

1. The present invention converts the frame-by-frame editing of the traditional video editing method into a one-time editing of the synthesized video panoramic image, which greatly reduces the manual interaction and calculation required for editing;

2. Since the user's editing work is completed on the obtained video panorama, it is more intuitive and accurate.

Description of drawings

Fig. 1 is the flowchart of the method of the present invention.

Figure 2 is a schematic diagram of image transplantation.

Fig. 3 is a schematic diagram of semi-automatic filling of texture images.

Fig. 4 is an example of the present invention being used for editing video scene content, wherein (a) is each frame of the original video sequence, (b) is the generated video panorama, (c) is the result after editing the video panorama, ( d) The result of back-projecting the edited video sequence back to the coordinate system of each video frame, that is, the final result.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

FIG. 1 shows a flowchart of a method for video editing according to the present invention.

During the shooting of motion video, the motion of the camera will cause the motion of the background of the video image, and this type of motion is called global motion. The local motion corresponds to the global motion, and the local motion refers to the foreground motion caused by the motion of the moving object. Considering that the moving object can be a rigid body or a non-rigid body, the frame difference method based on the time domain is used for the initial segmentation. The frame difference method is based on the assumption that the background is stationary or has a uniform global motion, while the moving objects have properties different from this global motion. In the case that the background has a uniform global motion, only the global motion parameters need to be obtained, that is, the moving object area that does not obey the parameters can be obtained.

As shown in FIG. 1 , in step 101 , global motion estimation is performed on relative global motion between frames of a video sequence to obtain global motion parameters. Based on the requirements of video editing, the global motion between every adjacent two frames in the video frame is estimated, so that the global motion of each frame of the video relative to the aforementioned reference frame can be calculated recursively, and the coordinates of each frame of the video can be obtained The transformation parameters of the system relative to the reference frame coordinate system. Typically, the reference frame is the first frame in the optional scene.

The law of global motion between adjacent image frames can be represented by global motion parameters, and different parameter models can be selected according to different scenes targeted by global motion estimation. In the present invention, in order to reflect the change of scene depth, what select for use is the perspective transformation model of eight degrees of freedom:

The matrix form of perspective transformation can be expressed as:

$\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}^{<span\; class="notranslate">\; \&prime;</span>}\\ {\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{<span\; class="notranslate">\; \&prime;</span>}\\ {\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}^{<span\; class="notranslate">\; \&prime;</span>}\end{array}\right]<span\; class="notranslate">\; =</span>\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 13</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; three</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 31</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 32</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 33</span>}}\end{array}\right]\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\\ {\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ {\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}\end{array}\right]$

It is a general form of non-singular linear transformations in a homogeneous coordinate system. The perspective transformation matrix has 9 parameters, but what is meaningful in the homogeneous coordinate system is its ratio, so this transformation actually has 8 parameters. For every two adjacent frames of images in the same video sequence, this parameter can be calculated only if there are four pairs of corresponding points.

Solve transformation parameter, the present invention has adopted feature point matching algorithm, and it comprises following three steps:

① Extract corner points from adjacent video frame images, such as using Harris corner points, SUSAN corner points, etc.

② Use the correlation matching of the extracted corner neighborhood information to get the rough matching result.

For a feature point x in sequence image n, the correlation window is set to (2n+1)×(2m+1). The search area in image n+1 is defined as (2d _u +1)×(2d _v +1). Calculate the correlation coefficient ρ(x,x') of each point x' and x in the search area:

$\mathrm{<span\; class="notranslate">\; \&rho;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; Cov</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>}$

where Cov(x, x′) is the covariance of x′ and x:

$\mathrm{<span\; class="notranslate">\; Cov</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span><span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; u</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; v</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span>}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}$

σ(x) is the standard deviation of the correlation window at point x=(u,v):

$\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\sqrt{\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}}$

E(x) is the mean of the correlation window at point x=(u,v):

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; m</span>}}\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\text{<span class="notranslate"> 2m+1</span>}<span\; class="notranslate">\; )</span>}$

Select the matching point with the largest correlation coefficient ρ(x, x′) as the best matching. In order to ensure the correctness of the matching point, a threshold T should also be set, and the correlation coefficient of the best matching should be greater than this threshold.

③Because of the matching in ②, there is a possibility of mismatching, and even if there is no error in the matching, if the matching points happen to be located on the moving object, the camera motion parameters obtained through these matching points are also wrong. Therefore, there must be a means of checking the matching results to ensure the robustness of the matching results. The method adopted in the present invention is to use the perspective transformation matrix as a constraint, use RANSAC to vote, and remove the point pairs that do not conform to the global motion parameters of the camera in the rough matching result.

For the RANSAC algorithm, please refer to literature 1: Fischler M.A.and Bolles R.C.RandomSample Consensus: A Paradigm for Model Fitting with Applications to ImageAnalysis and Automated Cartography. Communications of the ACM, 1981, Vol.24: 381-395.

A brief description of the RANSAC algorithm:

则投影后两个对应点之间的欧氏距离为Assuming that the homogeneous coordinates of point x are expressed as (x ₁ , x ₂ , 1) ^T , then the coordinate x′ after perspective matrix projection is

Then the Euclidean distance between two corresponding points after projection is

$\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; =</span>\sqrt{{<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

in:

x ₁ '=h ₁₁ x ₁ +h ₁₂ x ₂ +h ₁₃

x ₂ '=h ₂₁ x ₁ +h ₂₂ x ₂ +h ₂₃

x ₃ '=h ₃₁ x ₁ +h ₃₂ x ₂ +h ₃₃

Assuming that the number of corresponding point groups obtained by corner point matching is P, the initial value of the maximum matching point group P _max is 0, and the initial value of the number of iterations N is set to 200.

a) Randomly select 4 groups of point pairs from P to obtain the perspective matrix H _Pi ;

b) Calculate the distance between each matching point and the model, if the distance is less than the threshold d, mark this point as true, otherwise mark it as false, write down the group of point pairs marked as true in all point pairs under the current model Number P _G ;

c) If P _max <P _G , then set P _max =P _G , and save all point pairs currently marked as true, and go to step d; otherwise, go back to step a;

d) Calculate the number of iterations $k_P=\frac{\log (1-T)}{\log (1-{(P_G/P)}^4)},$ Where T is the prior probability of the proportion of correct matching results in the predicted rough matching results;

e) If N>k _P , then make N=k _P , compare it with the current total number of iterations k, if k<N, make k=k+1, return to step a; otherwise, go to step f;

f) If P _max ≥ 4, go to ④ use the least squares method to solve the optimal perspective matrix _HP for all point pairs that satisfy the current model, and the previously recorded matching point pairs are to use the RANSAC method to remove all gross points for the remaining results.

Using perspective transformation as the constraint condition of RANSAC can not only eliminate the error points in the matching, but also add the constraint of the parameter equation to the matching.

4. under normal circumstances, matching point number is more than asking 4 groups of points required by the perspective transformation matrix of 8 degrees of freedom. The present invention asks the solution of this overdetermined linear equation with least squares method, and its process is as follows:

Assuming that there are n groups of corresponding matching points x, x′, and their corresponding perspective matrix is H _P , then the solution H _P of the least squares method should have the minimum value of the following formula:

$\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; P</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

make $\frac{\mathrm{\&sigma;E}}{\mathrm{\&sigma;}{h}_P}=0,$ The derivative of _HP can be deduced by deriving each element of it.

For example, $\mathrm{\&Sigma;}{\left|\right|\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{\mathrm{twenty\; one}}& h_{\mathrm{twenty\; two}}& h_{\mathrm{twenty\; three}}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]-\left[\begin{array}{c}{x}_1^{\&prime;}\\ x_2^{\&prime;}\\ x_3^{\&prime;}\end{array}\right]\left|\right|}^2$ Take the partial derivative for h ₁₁ , and expand to get:

∑2(h ₁₁ x ₁ +h ₁₂ x ₂ +h ₁₃ x ₃ -x′ ₁ )x ₁ =0

Each element in _HP is expanded as the partial derivative of h ₁₁ in the above formula, and then the results can be combined to get:

$\mathrm{<span\; class="notranslate">\; 2</span>}\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; P</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}<span\; class="notranslate">\; =</span>}\mathrm{<span\; class="notranslate">\; 0</span>}$

Right now $h_P\underset{i}{\mathrm{\&Sigma;}}{x}_i{x_i}^T=\underset{i}{\mathrm{\&Sigma;}}{x}_i^{\&prime;}{x_i}^T\&CenterDot;$

The matrices obtained by multiplying the two 3×1 and 1×3 column vectors and row vectors on the left and right sides are respectively: $A=\underset{i}{\mathrm{\&Sigma;}}{x}_i{x_i}^T$ and $B=\underset{i}{\mathrm{\&Sigma;}}{x}_i^{\&prime;}{x_i}^T,$ Then _HP A=B, the solution of _HP can be obtained as:

H _P =BA ^-1

As shown in FIG. 1, in step 102, when there is a moving object in the video sequence, it should be removed. The present invention adopts the frame difference method to determine the approximate range of the moving object, uses this as the initial value, combines the correlation between each pixel in the frame and between frames, defines the energy equation, and uses the graph cutting method to solve the energy equation to obtain the final Split results.

Step 101 has already introduced the method of obtaining the global motion parameters of the camera. Using this parameter, one frame of image I _i+1 can be projected into the coordinate system of the adjacent frame I _i to obtain a new image I _i+1 ′. This image is stationary relative to the background of I _i . Assuming that images I _i+1 and I _i satisfy the projection relationship P, that is, x ⁱ⁺¹ = ^Pxi , where x ⁱ and x ⁱ⁺¹ represent the coordinates of corresponding points on images I _i and I _i+1 respectively, then the new image The calculation process of I _i+1 ′ is:

① Generate a blank image I _i+1 ′ with the same size as the original image;

② for $\&ForAll;x\&Element;I_{i+1}^{\&prime;},$ Find its corresponding point coordinates x'=P ^-1 x (P ^-1 is the back projection of P) in the original image;

③ The obtained corresponding coordinate x', which is usually not an integer, is interpolated by bilinear interpolation.

It should be noted that since the ranges of the two frames of images are not exactly the same, when projecting one frame of image to the coordinate system of another frame of image, there will be some "blind areas" (the calculated coordinate value exceeds the boundary of the image), The pixel value of the point in the blind area is set to 0.

Ideally, when there is no moving object in the sequence, I _i+1 ′ and I _i are the same image. If there is a moving object in the sequence, I _i+1 ′ and I _i will be different in the area where the moving object exists. Therefore, the moving object area in _I can be defined as:

{x _i |x _i ∈I _i ，|f(x _i )-f(x _i+1 ′)|＞T}

In the above formula, f(xi ₎ represents the pixel value of the point in I _i whose position is x _i , and f(x′ _i+1 ) is the new image I _i generated by projecting I _i+1 to I _i The pixel value whose coordinates are x'i ₊₁ in ₊₁ ', T is the set threshold. When the subtraction result of the corresponding point is greater than the threshold T, the point is considered to be a point on a moving object, otherwise it is regarded as a static background point.

In order to retain as much information as possible of the static background image in each frame image, the information of the third frame can be added, that is, the image of the previous and subsequent frames is used to estimate the moving object in the current frame, so the above formula is modified,

{x _i |x _i ∈I _i ，|f(x _i )-f(x _i+1 ′)|＞T&|f(x _i )-f(x _i-1 ′)|＞T}

Affected by noise and other factors, the results obtained directly from the above formula often have some error isolated points or small areas, which can be easily processed by morphological operators.

If the graph cutting method uses pixels as nodes, the amount of calculation will be large, which will affect the efficiency of the algorithm. Therefore, before using the graph cutting algorithm, the present invention uses the mean value shift method to pre-segment the image, and uses the regions obtained after the segmentation as the nodes of the graph. This can not only reduce the amount of calculation, but also ensure the accuracy of the segmentation result because the mean shift can more accurately locate the edge of each color region.

For the mean shift algorithm, please refer to literature 2: Fukunaga K. and Hostetler L.D. The estimation of the gradient of a density function, with applications in pattern recognition, IEEE Transactions on Information Theory, 1975, 21(1): 32-40.

The color segmentation algorithm based on the mean shift is mainly divided into two steps: first, the image is subjected to mean shift filtering in the joint domain, which has discontinuous retention, and each pixel is divided into the nearest mode in the joint domain, and Replace the original value of each pixel with the 3D color component in the corresponding mode. Then, an iterative method is used to fuse the pattern attraction domains in the range of h _r /2 in the color space until convergence, and finally the segmented image is obtained.

The color segmentation algorithm based on mean shift uses color information and spatial position to divide the image into small regions. However, the result of segmentation does not have semantic knowledge, and it is impossible to distinguish between moving objects and backgrounds based on this result. Therefore, we use the method of graph cutting to establish the relationship between the moving object area and the result of the color mean shift algorithm, so that the temporal domain and spatial domain information can be combined to obtain more accurate segmentation results.

For graph cutting algorithm, please refer to literature 3: Yuri Boykov, Olga Veksler, Ramin Zabih. Efficient Approximate Energy Minimization via Graph Cuts. IEEE transactions on Pattern Analysis and Machine Intelligence, 2001, 20(12): 1222-1239.

As shown in Figure 1, in step 103, the panorama stitching adopts the plane stitching method, selects the plane where one frame is located as the reference frame, and uses the plane projection model to project all other frames onto the plane where this frame is located to construct a panorama . Step 101 has introduced in detail the method of estimating global motion parameters by matching. Suppose x _i represents the coordinates of the point on the i-th frame image, P _{i, j} represents the perspective projection matrix between two frames i and j, and the projection relationship formula between frames is expressed as:

x _i =P _i,i+1 x _i+1

Using the transitivity between adjacent frames, the projection relationship between each frame image and the first frame image can be obtained, namely:

x ₁ =P _1,2 x ₂

x ₂ =P _2,3 x ₃

.

.

.

x _n-1 = P _{n-1, n} x _n

From this, the projection relationship between each frame image and the first frame image can be calculated:

x ₁ =P _1,2 x ₂ =P _1,2 P _2,3 x ₃ =P _1,3 x ₃ (wherein, P _1,3 =P _1,2 P _2,3 )

                 

                 

                 

x ₁ =P _1,2 P _2,3 ...P _n-1,n x _n =P _1,n x _n (wherein, P _1,n =P _1,2 P _2,3 ...P _n-1,n )

Select the first frame of image as the reference frame, and then the projection relationship between each frame of image and the panorama can be obtained.

After determining the projection relationship between each frame and the coordinate system of the panorama, the size of the panorama needs to be calculated next: calculate the projection position of the four vertices of each frame image in the coordinate system of the panorama, and record its coordinates as (x, y). Compare the sizes of their coordinates to get x _max , x _max and y _max , y _min , then the size of the panorama can be determined as W×H, where W=x _max -x _min , H=y _max -y _min .

Since the perspective transformation is a linear reversible transformation, the projection relationship from the point on the panorama to each frame can be expressed as: $x_{\mathrm{no}}=P_{1,\mathrm{no}}^{-1}x,$ where matrix P _1,n ^-1 is the inverse matrix of P _1,n . Under this projection relationship, the coordinates {x _n , y _n } of the pixel point {x, y} on the panorama on the image of the nth frame can be obtained. Since the obtained {x _n , y _n } may not be an integer, the pixel value of this point can be calculated by bilinear interpolation.

When the camera motion between adjacent frames in a video sequence is relatively small, the points on the panorama usually correspond to the points on the image of multiple video frames. Assuming that the number of video frames corresponding thereto is M, the median value of the pixel values of the corresponding M points can be taken as the pixel value of the points on the panorama.

As shown in Figure 1, in step 104, according to the concrete requirement of video editing, carry out image content editing to the panorama generated by step 103, to the method for image editing a lot, in the present invention, adopted following three kinds of methods:

① Image transplantation: Figure 2 shows a schematic diagram of image transplantation. The selected image g in the left region is transplanted into the right region Ω, and Ω′ represents the internal boundary of Ω. Let the desired composite image after transplantation be u, the transplanted image be g, and the original image of the region Ω to be transplanted be f.

In order to introduce the information of the transplanted image, let the first differential of the composite image be the same as that of the transplanted image $\&dtri;u=\&dtri;g,$ Right now:

$\left\{\begin{array}{cc}\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; u</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; g</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}& \mathrm{<span\; class="notranslate">\; foru</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}\\ \frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; u</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; g</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}& \mathrm{<span\; class="notranslate">\; foru</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}\end{array}\right.$

At the same time, the transplanted synthetic image is also subject to constraints on the boundaries of the transplanted region. Usually, in order to maintain the continuity of the image, u=f for u∈ Ω′ should be satisfied.

The above two constraints can be expressed by the same energy function J(u):

$\mathrm{<span\; class="notranslate">\; J</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; \&Omega;</span>}}{<span\; class="notranslate">\; \&Integral;</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; (</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; u</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; -</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; g</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; u</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; -</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; g</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}\underset{{\mathrm{<span\; class="notranslate">\; \&Omega;</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{<span\; class="notranslate">\; \&Integral;</span>}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>$

The u that can minimize the value of this energy function J(u) is the synthesized image. Among them, λ (λ>0) is the Lagrange multiplier, which adjusts the proportion of the above two conditions in the whole constraint.

②Image editing based on information propagation: use the known information around the boundary of the area to be edited, and "propagate" the gray information on the boundary into the area to be edited along the direction of the smallest gradient.

Let I _o (i,j):[0,M]×[0,N]→R represent an image with a size of M×N. The image inpainting algorithm will obtain a series of images I(i, j, n) through iteration: [0, M]×[0, N]×N→R, satisfying I(i, j, 0)=I _o (i , j) and $\underset{\mathrm{no}\&Right\; Arrow;\&infin;}{\lim }I(i,j,\mathrm{no})=I_R(i,j)$ (I _R (i, j) is the output image), its mathematical expression is written as:

${\mathrm{<span\; class="notranslate">\; I</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; I</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&Delta;tI</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&ForAll;</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}$

In the above formula, n represents the repair time, that is, the number of iterations, (i, j) represents the coordinates of the pixel, and Δt is the step size of each iteration, _It ⁿ (i, j) represents the image I ⁿ (i, j), and I ⁿ⁺ⁱ (i, j) is the result obtained by I ⁿ (i, j) after one iteration under the constraint of ^{It n} ₍ i, j). The valid area of this equation is the interior of the manually specified area Ω to be modified. A repaired image can be obtained after n iterations.

是扩充的方向，可得到I_t ⁿ(i，j)的表达式为：The key of the algorithm is to find a suitable _It ⁿ (i, j). In artificial repair technology, people usually slowly expand the information outside the damaged area into the damaged area along the outer boundary of the damaged area. Therefore, when imitating manual restoration with a computer, this idea can be borrowed to smoothly expand the external information of Ω to the internal of Ω. Suppose L ⁿ (i, j) is the information to be expanded, and

is the direction of expansion, and the expression of ^{It n} ₍ i, j) can be obtained as:

${\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; \&delta;</span>}{\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; N</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>$

是信息Lⁿ(i，j)的变化量。在此等式中可以估计出图像的信息Lⁿ(i，j)并可计算出其在方向

上的变化量。在稳定后的状态也就是算法收敛时，满足Iⁿ⁺¹(i，j)＝Iⁿ(i，j)，也就是 $\stackrel{\&RightArrow;}{{\mathrm{\&delta;L}}^n}(i,j)\&CenterDot;\stackrel{\&RightArrow;}{N^n}(i,j)=0,$ 意味着信息量L已经完全扩充到方向中。in

is the change amount of information L ⁿ (i, j). In this equation, the information L ⁿ (i, j) of the image can be estimated and its direction in

The amount of change on . In the stable state, that is, when the algorithm converges, it satisfies I ⁿ⁺¹ (i, j)=I ⁿ (i, j), that is $\stackrel{\&Right\; Arrow;}{{\mathrm{\&delta;\; L}}^{\mathrm{no}}}(i,j)\&CenterDot;\stackrel{\&Right\; Arrow;}{N^{\mathrm{no}}}(i,j)=0,$ It means that the amount of information L has been fully expanded to direction.

Because it is hoped that the information is smoothly diffused into the image, L ⁿ (i, j) is a smoothing operator, and the Laplacian operator can be selected, which is expressed as $L^{\mathrm{no}}(i,j)=I_{\mathrm{zz}}^{\mathrm{no}}(i,j)+I_{\mathrm{yy}}^{\mathrm{no}}(i,j).$ Of course, other smoothing operators are also applicable.

的法线方向为平滑信息变化的方向

。对于Ω内的每个点(i，j)，的方向垂直此点所在的边界。修复区域是任意的，故的方向与原图像本身无关。如果等照度线的方向和

一致，则选取

时，最好的方向就是等照度线的方向。对任意点(i，j)，梯度

是变化最大的方向，因此与梯度垂直的方向

是变化最小的方向。定义

为等照度线的方向，从而方向矢量

的表达式为：Since the continuity of the isoluminescence line is always along the normal direction of the boundary, the boundary is chosen

The normal direction of is the direction of smooth information change

. For each point (i, j) within Ω, The direction of the point is perpendicular to the boundary where this point is located . The repair area is arbitrary, so The orientation of is independent of the original image itself. If the direction of the isolux line and

agree, select

, the best direction is the direction of the isolux line. For any point (i, j), the gradient

is the direction of greatest change, so the direction perpendicular to the gradient

is the direction of least change. definition

is the direction of the isoluminescence line, so the direction vector

The expression is:

$\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; \&dtri;</span>}^{<span\; class="notranslate">\; \&perp;</span>}{\mathrm{<span\; class="notranslate">\; I</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>$

③ Semi-automatic filling of texture images: The two methods provided above are mainly aimed at smooth image areas, and semi-automatic filling of texture images should be used for areas with rich textures. Use the undamaged area in the image as a sampling sample, and repair the image in units of blocks. Define the size of each "block" as w×w, divide the damaged area into n blocks based on w, {B ₁ , B ₂ ,..., B _n }, and then repair each small block in turn.

，如图3中未损坏区域内用阴影所表示的部分。计算两个带状区域的距离，可以得到与当前损坏块B_k距离小于某一给定阈值的块的集合ψ_B。定义集合ψ_B为：As shown in Figure 3, for the damaged block Bk currently to be repaired, take a band of width w _B in the known area around it , the shaded part in Figure 3. For the current damaged block B _k shown in the figure, the image on the right is still a damaged part, so the information of the right strip area is unknown, so the strip areas taken are the left, upper and lower sides. Similarly, for other damaged blocks, only the strip-shaped areas with known information in the four sides are considered. Take the same position for any sampling specimen B _{(x, y)} in the sampling area (here is the undamaged part of the image) (B _{(x, y)} represents the block whose point in the lower left corner is (x, y)) and a banded region of size

, as shown in the shaded area in the undamaged area in Figure 3. By calculating the distance between the two banded areas, a set ψ _B of blocks whose distance to the current damaged block B _k is smaller than a given threshold can be obtained. Define the set ψ _B as:

${\mathrm{<span\; class="notranslate">\; \&psi;</span>}}_{\mathrm{<span\; class="notranslate">\; B</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; \{</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{{\mathrm{<span\; class="notranslate">\; B</span>}}_{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>}{\mathrm{<span\; class="notranslate">\; E.</span>}}_{{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; <</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \}</span>$

Among them, d _max is a given threshold. Randomly select a block in the set ψ _B , and copy the gray value of each point in this block to the current damaged block B _k in turn. The rest of the damaged blocks are processed in the same way (areas that have been repaired may provide boundary constraints for the next block to be repaired). Until the value of the last block is determined, the entire image is patched.

As shown in FIG. 1 , in step 105 , according to the specific requirements of video editing, a video sequence is recovered from the panorama edited in step 104 .

The projection relationship between each video frame image and the panorama is expressed as:

x=P _1,n x _n

Since the perspective transformation is a linear reversible transformation, it can be obtained:

$x_{\mathrm{no}}=P_{1,\mathrm{no}}^{-1}x,$ where P _1,n ^-1 is the inverse matrix of P _1,n .

Under this projection relationship, that is, the coordinates of the pixel points on each video frame on the panorama. If the coordinate points are changed to non-integer numbers, the pixel value of the point can be calculated by bilinear interpolation.

Fig. 4 is an example of the present invention being used for editing video scene content, wherein (a) is each frame of the original video sequence, (b) is the generated video panorama, (c) is the result after editing the video panorama, ( d) The result of back-projecting the edited video sequence back to the coordinate system of each video frame, that is, the final result.

Claims (4)

1, a kind of video editing method based on panorama mosaic, it is characterized in that, the steps of this method are as follows: 1) Generate a video panorama describing the overall picture of the motion video with a plurality of video frames; 2) Editing the image content of the obtained video panorama; 3) Back-projection of the edited video panorama back to each video frame coordinate system to generate an edited video sequence; Described video panorama generation comprises the following steps: (1) Carry out global motion estimation to the relative global motion between a plurality of video frames, obtain the plane projection relation between each video frame image; (2) If there is a moving object in the motion video sequence, it will be removed first; (3) According to the planar projection relationship between each video frame image, the first frame image is used as a reference frame to establish a panoramic coordinate system, and each video frame image is projected into the panoramic coordinate system, and the panoramic image is estimated. size; (4) According to the planar projection relationship between each video frame, calculate the corresponding point of each image point in multiple video frame images on the panorama, sort these multiple corresponding points, and take the median as the panorama value to form a video panorama; The described panorama content editing method comprises: (1) Image transplantation: select an area manually, then put the information of this area into the area to be filled, and change the color of the original area information according to the information outside the filled area, so that this filling becomes natural; (2) Image editing based on information propagation: use the known information around the boundary of the area to be edited, and "propagate" the gray information on the boundary into the area to be edited along the direction of the smallest gradient; (3) Semi-automatic filling of texture images: automatic filling of textured areas; The video sequence generation after described editing comprises the following steps: (1) Calculate the back projection matrix from the video panorama to each video frame coordinate system; (2) Generate each video frame image from the video panorama according to the back-projection matrix, and complete the video editing process. 2. a kind of video editing method based on panorama stitching according to claim 1, is characterized in that, described global motion estimation comprises: 1) Matching step: extract the corner points of each video frame image, and perform correlation matching to obtain an initial matching point set; 2) Parameter estimation step: Use Ransac to eliminate the wrong matches in the initial matching point set, and use least squares to estimate the transformation parameters under perspective projection. 3. A kind of video editing method based on panorama mosaic according to claim 1, is characterized in that, described moving object removal method comprises: 1) Use the frame difference method to determine the approximate range of the moving object; 2) Using color-based region segmentation to divide the image into regions of different colors; 3) Combine the two with the graph cutting method, and use the segmentation result of the previous frame as a constraint to optimize the solution. 4. A kind of video editing method based on panorama mosaic according to claim 1, is characterized in that, also comprises carrying out color brightness correction to each video frame image, to eliminate the color that is caused by different exposure and white balance when shooting. difference.

Images