CN109087250B - Image stitching method based on regular boundary constraints - Google Patents

Abstract

The invention discloses an image splicing method based on regular boundary constraint, which comprises the following steps: s1, performing initial splicing on the images; s2, extracting irregular boundaries of the panorama; s3 is established based on the rule boundary of the segment rectangle; s4 image stitching based on the segmented rectangular boundary constraint. The method defines a rule boundary according to the content and the shape characteristics of the panoramic image and realizes image splicing based on the rule boundary through global energy optimization.

Description

Image stitching method based on regular boundary constraints

technical field

The invention belongs to the technical field of image processing, and in particular relates to an image stitching method based on regular boundary constraints.

Background technique

With the rapid development of image acquisition and editing processing technologies, panoramic images have been widely used in portable mobile devices such as smartphones, and their wider viewing angles bring better visual experience to users. The problem of irregular borders (irregular borders) in image stitching in the prior art makes the panoramic image need to be cut a lot during the display process, and thus part of the image information is easily lost.

Although the current image editing technology has made a lot of progress and is widely used in smart devices, however, due to the free movement of the camera of mobile terminals such as mobile phones during the shooting process, most of the panoramic images after feature registration have irregular boundaries. In order to display the panorama in a common display device, it usually needs to be cut. Generally speaking, the panorama captured by the mobile phone camera is the result obtained after being cut by a rectangular window. At this time, part of the image information has been lost, thus reducing the wide viewing angle experience of the panorama.

In order to generate a panorama with regular boundaries and avoid the loss of image content caused by cutting, there are currently two main research methods:

(1) Image content completion method. The method first obtains the bounding rectangle of the panorama, and then complements the image content between the rectangular and irregular boundaries. However, the performance of image completion methods is unstable, and it is difficult to complete content with certain semantics with high quality.

(2) Kaiming He et al. proposed a deformation-based panorama image rectangularization method in 2013. The method takes the panorama with irregular boundary as input, takes the circumscribed rectangle as the boundary constraint, combines the local shape and line preservation constraints, and obtains the panorama with the rectangular boundary through deformation-based optimization.

Although the above two methods can improve the problem of the irregular boundary of the panorama, there are still the following problems:

1) Taking image stitching and boundary regularization as two independent processes, it is difficult to guarantee the optimal results.

2) The grid on the irregular panorama has the problem of voids in the boundary.

3) When there is a large amount of missing content in the panorama, the result after deformation optimization has a large distortion, which corresponds to the error of the feature.

Therefore, when the gap between the irregular boundary and the circumscribed rectangle is large, it is necessary to redefine the regular boundary constraint to reduce the distortion caused by the deformation while ensuring the boundary regularity.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide an image stitching method based on regular boundary constraints, which defines regular boundaries according to the content and shape features of the panorama, and realizes image stitching and boundary regularization through global energy optimization.

In order to solve the above-mentioned technical problems, the present invention adopts the following technical solutions:

An image stitching method based on regular boundary constraints, the image stitching method comprising:

S1, using multiple partially overlapping images as input, establish energy optimization constrained by feature correspondence between images, local grid shape preservation, and global similarity preservation, and obtain the initial image stitching result through grid deformation;

S2, take the mesh vertices of the initial image splicing as input, extract the boundaries of multiple image meshes after deformation, and construct them into polygons, and obtain the irregular boundaries of multiple image meshes through the union operation of polygon Boolean operations. It consists of some mesh vertices and intersections between meshes;

S3, take the irregular boundary as the input, segment the boundary according to the upper grid intersection and vertices of the irregular boundary, perform adjacent boundary clustering according to the direction of the segmented boundary and the number of vertices, and classify the boundary vertices of each segment after the clustering. The coordinate mean of the panorama is used as the target boundary condition of the panorama, thereby generating a segmented rectangle boundary constraint;

S4, according to the irregular boundary and segmental boundary constraints of step S2 and step S3, realize seamless panorama stitching and regular boundary preservation in the following manner: linearly combine image initialization stitching constraints, regular boundary constraints, and straight line preservation constraints , then solve the energy optimization and obtain the panorama stitching result through deformation.

In a preferred embodiment, step S4 includes: initializing the image splicing constraints, regular boundary constraints, and line retention constraints according to the formula

E(v)=w _a (E _a )+w _s (E _s )+w _g (E _g )+w _r (E _r )+w _l (E _l )

Perform linear combination, solve energy optimization and obtain panorama stitching results through deformation.

In a preferred embodiment, step S4 further includes: optimizing the segment boundary in an iterative manner, and solving a new deformed mesh based on this constraint to obtain a panorama with a more regular boundary, so that the segment regular boundary is close to a rectangle and avoids distortion.

Adopting the present invention has the following beneficial effects:

1. The image stitching method based on regular boundary constraints described in the present invention can generate high-quality panoramic images, and ensure visually acceptable distortion while regularizing the boundaries.

2. The image stitching method based on regular boundary constraints according to the present invention has high efficiency and practicability, can quickly calculate and render a panoramic image, and effectively ensure the content integrity and boundary regularity of the panoramic shooting result.

3. The image stitching method based on regular boundary constraints of the present invention proposes to add regular boundary constraints to the optimization framework of image stitching, and realize image stitching based on regular boundaries through global optimization, thereby solving the following problems: when irregular The gap between the boundary and the circumscribed rectangle is large. By defining a reasonable segmented rectangle boundary, the content integrity of the panoramic stitching can be guaranteed to the greatest extent, and the distortion caused by the deformation can be effectively controlled.

Description of drawings

1 is a schematic diagram of a processing flow of an image stitching method based on rule boundary constraints according to an embodiment of the present invention;

FIG. 2 is an image stitching system based on the processing method shown in FIG. 1 .

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Referring to Fig. 1, the present invention discloses an image stitching method based on regular boundary constraints. Fig. 1 shows a schematic processing flow diagram of an embodiment of the image stitching method of the present invention, and the method includes:

S1, initial stitching of images

Taking multiple partially overlapping images as input, establish an energy optimization constrained by the feature correspondence between images {E _a }, local grid shape preservation {E _s }, and global similarity preservation {E _g }. Generate the initial stitching result of the image, where:

1) Feature point correspondence The APAP method proposed by Zaragoza et al. in 2014 is used to achieve accurate correspondence of feature points.

2) Local shape preservation is achieved by constraining the deformation of each mesh, that is, dividing each quadrilateral mesh into two triangles, and using the method proposed by Igarashi et al. in 2009 to achieve the shape feature preservation of each right triangle.

3) Global feature preservation The method proposed by Chen et al. in 2016 is adopted to achieve global similarity transformation by maintaining the scale and rotation consistency of the stitched images.

S2, Extraction of Irregular Boundaries of Panorama

According to S1, the initial spliced grid of the image is used as the input, and the boundaries of each grid are extracted and constructed into corresponding polygons, and then the Boolean union operation is performed on multiple polygons to obtain the irregular boundaries of the panorama. At this point, the boundary contains some mesh vertices and intersections between meshes. Further, the corner points of the grid contained in the irregular boundary (that is, the four corner points of the upper, lower, left, and right of the grid) are extracted, and the corner points that are closest to the four vertices of the circumscribed rectangle of the irregular boundary are determined. As the corner points of the irregular boundary, the corner points extracted at this time divide the vertices on the irregular boundary into four directions.

S3, regular boundary establishment based on segmented rectangles

The boundary is segmented based on mesh intersections and corners in each direction of the irregular boundary. Then, further, according to the length and direction of each segment boundary, segment boundary clustering analysis is performed, the same or too short segment boundaries are merged, and finally the average coordinate value of each segment boundary after clustering is used as the target boundary. value. For the mesh intersection on the boundary, firstly extract the two pairs of mesh vertices that form the intersection, and use their linear interpolation to represent the intersection, and finally transform the boundary constraints of the intersection into constraints on the interpolated mesh vertices.

S4, Image Stitching Based on Piecewise Rectangle Boundary Constraints

According to the irregular boundary and segmental boundary constraints of steps S2 and S3, the seamless stitching of the panorama and the maintenance of the regular boundary are realized in the following manner. According to the irregular boundary constraints of S3, combined with the initial image stitching constraints of S1, a global image is established. Energy optimization realizes image stitching based on segmented rectangular boundary constraints. The energy constraints include feature consistent correspondence {E _a }, local shape preservation {E _s }, global similarity preservation {E _g }, line preservation {E _l }, rules Boundary {E _r }, the above energy is linearly combined according to the following formula to obtain the total energy equation E(v), and the deformed grid is obtained by minimizing E(v), wherein, in a specific embodiment, each The weights of the energy terms are w _a =1.5, _{ws =10, w g =} 0.75, _wr =1000, w _l =15. Finally, texture mapping is performed according to the deformed grid position to obtain the result of image stitching, and the seams at the stitching are further eliminated through seamless fusion.

The formula is:

E(v)=w _a (E _a )+w _s (E _s )+w _g (E _g )+w _r (E _r )+w _l (E _l )

In an embodiment, in order to make the segmented regular boundary as close to a rectangle as possible and avoid unnecessary distortion, the segmented boundary is continuously optimized in an iterative manner, and a new deformed mesh is solved based on this constraint, so as to obtain a more regular boundary panorama. In each iteration process, the adjacent segment boundaries in each direction are first traversed, and the deformed mesh with the smallest energy cost E(v) after merging the adjacent boundaries is used as the result of this iteration. If the energy cost difference between two adjacent iterations is too large or there is no segment boundary for merging, the iteration ends.

Referring to FIG. 2 , the image stitching method based on regular boundary constraints of the present invention can effectively solve the boundary irregularity problem of the traditional stitching method. Through the method based on iterative optimization, the optimal segmented rectangular boundary can be obtained, thereby effectively improving the performance of panoramic stitching. Visual effects.

The present invention can perform high-quality splicing of images captured by a user using a handheld mobile device by moving, and ensure the normativeness of the panorama boundary through regular boundary constraints, thereby effectively avoiding the loss of image content due to cutting. The present invention can be applied to deformation-based image and video editing, such as video splicing, video de-shake and other operations.

It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those of ordinary skill in the art will appreciate that various changes can be made without departing from the spirit and scope of the invention as defined by the appended claims. Changes in form and detail.

Claims (3)

1. an image splicing method based on regular boundary constraints, is characterized in that, described image splicing method comprises: S1, take multiple images containing partial overlap as input, establish feature correspondence between images, local grid shape preservation Constrained energy optimization, the global similarity is maintained as the constraint, and the initial stitching result of the image is obtained through mesh deformation; S2, taking the mesh vertices of the initial image stitching as input, extracting the boundaries of multiple image meshes after deformation, and constructing them into multiple meshes. Edge, the irregular boundary of multiple image meshes obtained by the union operation of polygon Boolean operations, the boundary is composed of partial meshes It consists of grid vertices and intersections between grids; S3, take the irregular boundary as input, segment the boundary according to the mesh intersections and vertices on the irregular boundary, according to The direction and length of the boundary after segmentation are used to cluster adjacent boundaries, and the mean coordinates of the boundary vertices of each segment after clustering are used as Panorama target boundary conditions, resulting in segmented rectangle boundary constraints; S4, according to the irregular boundary and segmental boundary constraints of step S2 and step S3, the panorama is realized in the following manner Graph seamless stitching and regular boundary preservation: integrate image initialization stitching constraints, regular boundary constraints, and line preservation constraints into Line linear combination, then solve the energy optimization and obtain the panorama stitching result through deformation. 2. the image stitching method based on regular boundary constraints as claimed in claim 1, is characterized in that, step S4 comprises: Initialize the image stitching constraints, regular boundary constraints, and line retention constraints according to the formula E(v)=wa(Ea)+ws(Es)+wg(Eg)+wr(Er)+wl(E _l ) Perform linear combination, solve energy optimization and obtain panorama stitching results through deformation; where E(v) represents the total energy, Ea represents the corresponding energy of feature consistency, Es represents the local shape preservation energy, Eg represents the global similarity preservation energy, Er represents the regular boundary energy, El represents the line preservation energy, and the weights of each energy term are respectively is wa=1.5, ws=10, wg=0.75, wr=1000, wl=15. 3. the image stitching method based on regular boundary constraints as claimed in claim 2, is characterized in that, step S4 also comprises: optimize segmental boundary by iterative mode, and take this as constraint to solve new deformed grid, obtain boundary more regular Panorama to make segmented regular boundaries close to rectangles and avoid distortion.

Images