CN104506828B - A kind of fixed point orientation video real-time joining method of nothing effectively overlapping structure changes - Google Patents

Abstract

The invention discloses a fixed-point directional video real-time mosaic method without effective overlapping and variable structure. The method includes: separately collecting video stream information at different positions; dividing the compressed video stream information into a plurality of first static Video frame group; the static image corresponding to each road video stream information in the first static video frame group is converted into a top view of the object to be photographed to form a second static video frame group; for the second static video frame group Each video frame is positioned, panoramic rough stitching, compensation fusion, and panoramic fine stitching to obtain a real-time panoramic video stream. According to the invention, images without effective overlap and variable structure collected in the same scene are spliced according to different viewing angles and different directions, and then video streams are generated in real time. The stitching method of the invention not only improves the stitching precision of images, but also ensures the stitching efficiency of images and meets the real-time requirement of stitching video streams.

Description

A real-time mosaic method for fixed-point directional video without effective overlapping variable structure

technical field

The invention relates to the technical field of real-time video image splicing, and more particularly relates to a fixed-point directional video real-time splicing method without effective overlapping variable structure.

Background technique

In recent years, with the rapid development of video stitching technology, it has been widely used in the fields of high-resolution images with large viewing angles, virtual reality, medical images, remote sensing technology, and military fields. Video stitching technology mainly includes image stitching technology and video real-time synthesis technology. First, image stitching technology is the core of video stitching technology, mainly including image registration and image fusion. Image fusion is to synthesize high-quality images by eliminating the intensity or color discontinuity between adjacent images caused by geometric correction, dynamic scenes or lighting changes. Second, real-time video synthesis technology is realized by using parallel computing architectures such as FPGA, Intel's IPP, and Nvidia's CUDA to improve the execution efficiency of image stitching algorithms.

From the perspective of image acquisition, it can be roughly divided into four categories. 1) A single camera with a fixed fulcrum, and the lens rotates to collect images of the same scene; 2) A single camera is fixed on a slide rail and moves in parallel to collect images of the same scene; 3) Multiple cameras capture the same scene at different viewing angles and directions For image acquisition, there is an effective overlapping area between images that can be used for image stitching; 4) Multiple cameras collect images of the same scene at different viewing angles and directions, and there is no effective overlapping area between images that can be used for image stitching, or even between images. There are minor gaps and holes. According to the actual problems studied by the present invention, the fourth type of situation is mainly concerned, that is, video collection and splicing of the same scene are performed by using multiple cameras in fixed-point orientation.

From the perspective of image registration, the core technology of image stitching, there are currently two image registration technologies, namely phase correlation method and geometric feature method. The phase correlation method is scene-independent and can precisely align images with pure two-dimensional translation. However, the phase correlation method is only suitable for solving image registration problems with pure translation or rotation. In the affine and perspective transformation models, this method cannot register images, and it is impossible to achieve the camera position and imaging plane in the actual process. The absolute parallel, so its actual use value is low. The geometric feature method is a method of splicing images based on low-level geometric features in the image, such as edge models, corner models, and vertex models. However, the premise of the image registration method based on geometric features is that the two images must have a certain amount of overlapping areas, and the matched images must have coherent features in time. Image stitching in overlapping areas is impotent.

From the perspective of image fusion, its purpose is to eliminate the stitching seams between images from the perspective of image color brightness and structure. There are many methods of image fusion. For the fusion algorithm of image color and brightness, there are simple light intensity weighted average and weighted average fusion, and complex ones include image Voronoi weight method and Gaussian spline interpolation method. Its core idea is to segment the image first, use the overlapping area as the matching standard, and then fuse the stitching seam of the image by means of image correction, color transformation and pixel interpolation. For the excessive difference in the image structure, a simple feathering method is generally used, and the weights are averaged according to the Euclidean distance, and then the filtering technology is used to eliminate the blurring of the image caused by the feathering and the ghosting phenomenon in the spliced video. Obviously, the existing methods cannot handle the fusion of image stitching seams in non-overlapping areas collected by multiple cameras from different perspectives and directions in the same scene, and for real-time video stream fusion, the existing technology cannot meet real-time performance. requirements.

Patent Publication No. CN103501415A invention patent is a real-time video stitching method based on structural deformation of overlapping parts. Its working principle is to first calculate the stitching seams of two images, and then extract one-dimensional feature points on the two stitching seams And matching, move the matched feature points to the coincident position and record the displacement, carry out the diffusion of structural deformation within the set range of influence of deformation diffusion, and finally calculate the gradient map after structural deformation, and use the fusion method on the gradient domain to complete the image Fusion to get the final stitched image. In order to meet the real-time requirements of video stitching, the patent implements the image stitching algorithm on FPGA, so as to achieve fast and efficient video stitching effect. Apparently, the method adopted in this patent cannot splicing images of non-overlapping areas collected by multiple cameras at different angles of view and in different directions on the same scene, and cannot meet the actual needs of the actual problems studied by the present invention.

Patent Publication No. CN101593350A invention patent is a method, device and system for depth adaptive video splicing. Its video stitching system includes camera array, calibration device, video stitching device, and pixel interpolator and mixer. First, the camera array generates multiple source videos; then, the calibration device performs epipolar calibration and camera position calibration, determines the seam area of each spatially adjacent image pair in multiple source videos, and generates a pixel index table; again, the video The splicing device uses the average offset value to form a compensation item of the pixel index table to update the pixel index table; finally, the pixel interpolator and the mixer use the updated pixel index table to combine the multi-source video to generate a panoramic video. Obviously, this patent hopes to seek a balance between stitching quality and stitching efficiency by simplifying the image stitching algorithm, so as to realize the stitching of video streams. However, with the diversity and complexity of captured videos, a single serial graphic stitching algorithm It can't meet the real-time requirements of video stream splicing with large amount of data and complicated calculation, and its practicability is poor.

Contents of the invention

Aiming at the deficiencies in this field, the object of the present invention is to provide a method for fixed-point directional real-time splicing of images without effective overlap and variable structure collected by multiple cameras from different angles of view and directions in the same scene.

The technical scheme that realizes the above-mentioned purpose of the present invention is:

The invention discloses a fixed-point directional video real-time mosaic method without effective overlapping variable structure, and the method includes the following steps:

Step 1, installing a multi-camera video acquisition array, respectively collecting video stream information at different locations, and performing analog-to-digital conversion, synchronization and compression processing on the video stream information;

Step 2, convert the compressed video stream information into the same video format, and divide it into a plurality of first static video frame groups according to time sequence; wherein, each of the first static video frame groups includes the multiple video frames at the same time n-way video stream information collected by the camera video collection array;

Step 3, converting the static image corresponding to each video stream information in the first static video frame group into a top view of the object to be photographed according to the side view to top view geometric model, forming a second static video frame group;

Step 4. According to the positioning model, each video frame in the second static video frame group is positioned, and the panoramic rough mosaic is performed to obtain the panoramic rough mosaic image;

Step 5. According to the positioning model, determine the stitching seam position in the overlapping region, the stitching seam position in the seamless non-hole and non-overlapping region, and the stitching seam position in the region with holes or cracks in the rough panorama mosaic image;

Step 6. For the stitching seams with overlapping regions or the stitching seams of seamless, hole-free and non-overlapping regions, use brightness and color interpolation algorithms to fuse and stitch the stitching seams;

Step 7. The splicing process of the splicing seam in the region with holes or cracks is as follows:

Determine the bordering relationship between subimages corresponding to each video frame in the second static video frame group and between the subimages and holes or cracks according to the positioning module, and determine the hole or crack subimages according to the bordering relationship;

Extracting line features of subimages adjacent to leak or crack subimages;

matching the extracted line features to obtain a line feature pair;

Compensate for cracks or holes by extrapolating boundary points using features of adjacent area lines;

For the rough panorama mosaic image after compensation for leaks or cracks, use brightness and color interpolation algorithms to fuse and stitch seams to obtain panorama video frames after fine panorama mosaic;

Step 8: Process the first static video frame group at different times according to the steps 3 to 7 to obtain panoramic video frames at different times, and synthesize the panoramic video frames according to time sequence to obtain a real-time panoramic video stream.

Wherein, the second step is performed after receiving the synchronous video segmentation instruction, and after the operation of the second step is completed, the first static video frame is stored in chronological order.

Wherein, the geometric model of the side view variable top view in the step 3 is:

Among them, s is the scale factor, f _x , f _y is the focal length of the camera, c _x , _cy are the image correction parameters, R _x , R _y , R _z are the three column vectors of the rotation matrix, t is the translation vector, ( x, y, z) are the element coordinates of the side view of the static image, and (X, Y, Z) are the top view coordinates of the corresponding element.

Wherein, the positioning model in the steps 4 and 5 is:

Among them, x ₀ , y ₀ , z ₀ are the coordinates of the center point of the camera lens, x ₁ , y ₁ , z ₁ are the coordinates of the intersection point between the object to be photographed and the camera plane xoy, (α, β, γ) are the coordinates of the scene corresponding to the camera The direction angle of the generatrix of the cone, x ₂ , y ₂ , z ₂ are the intersection coordinates of the latitudinal circle corresponding to the scene of the camera and the generatrix of the cone, and x, y, z are the coordinates of the intersection of the scene of the camera and the camera plane xoy.

Wherein, the panoramic rough stitching in the step 4 is specifically:

First, generate a blank image that is as large as the size of the panorama of the subject;

Secondly, use the positioning model to perform positioning processing on the sub-image corresponding to each video frame in the second static video frame group, and determine the position, size and direction of each sub-image in the blank image;

Thirdly, the sub-images are filled to the corresponding places in the blank image one by one according to the predetermined numbering sequence of each camera in the multi-camera array and the positioning information of the sub-images taken by them, so as to realize the coarse mosaic of the panorama.

Wherein, the step 7, extracting the line feature of the sub-image adjacent to the leak or crack sub-image is specifically:

Assume that C(x,y) is the center pixel point, and let L(x,y) and R(x,y) be respectively the left and right adjacent areas of C(x,y) along a certain direction The average gray value, then the mean ratio is estimated as shown in formula (3);

RoA: C(x,y)=max{R(x,y)/L(x,y), L(x,y)/R(x,y)} (3)

Then, compare is compared with a predetermined threshold T ₀ when When it is greater than the threshold T ₀ , point C is considered as a boundary point;

The line feature fragments in the sub-images adjacent to the hole or crack sub-image will be extracted by the above algorithm, and reorganized into line features.

Wherein, in the step seven, matching the extracted line features is specifically:

The line feature is described by the corresponding line segment function, assuming that there are n sub-images surrounding holes or cracks, first, the slope of the line segment function of each sub-image extracted, the set I formed is expressed by formula (4) as follows,

Among them, m, n, l all represent the total number of line features extracted in the corresponding sub-image;

Use the following formula (5) to realize line feature matching between sub-images,

in, are any element in the set _I , and T1 is the matching threshold; satisfying the formula wherein, in the step 7, using the adjacent area line features to extrapolate the boundary points to compensate the cracks or loopholes is specifically:

First, according to the corresponding first line segment function that has matched the line feature pair, a second line segment function that can satisfy all the line features in the corresponding feature pair is constructed, and at the same time, the second line segment function is considered to be a pair of loopholes or Reasonable fitting of line features of fractures;

Then, using the second line segment function to extrapolate the hole or crack, thereby matching the position of the line feature determined by the line feature pair;

Finally, the line features of the leaks or cracks are extrapolated, using the color and brightness of the corresponding matching line feature pairs in the sub-images adjacent to the leaks or cracks sub-images, using color and brightness interpolation calculations, the splicing seams are processed. Fusion stitching.

Wherein, in the steps six and seven, the color and brightness interpolation algorithm is used to carry out fusion stitching specifically as follows:

Assuming that there are m sub-images adjacent to the hole or crack sub-image, the grayscale, color and brightness value of a point P in the crack or hole can be calculated according to the grayscale, color of the point closest to point P in the m subimages and luminance value, calculated by formula (6)

Among them, g(p) represents any one of the gray value, color value and brightness value of point P, and g _i ( _xi , y _i ) represents the point corresponding to g(p ), the gray value, color value or brightness value of ), the function ξ(x) is a linear weight function;

The complete panoramic video frame is obtained by performing the fusion operation shown above on each pixel in the crack or hole one by one.

Wherein, in the eighth step, the panoramic video stream is compressed, stored and displayed.

Corresponding to the method of the present invention, the device used includes multi-camera video acquisition array U3, multi-channel video synchronous acquisition unit U4, multi-channel video synchronous segmentation unit U5, multi-channel video side view variable top view unit U6, GPGPU video frame positioning, splicing , compensation and fusion unit U7 and real-time panoramic video stream generation unit U9, as shown in Figure 2;

The multi-camera video acquisition array collects the video stream information of different positions of the object to be photographed, and transmits it to the multi-channel video synchronous acquisition unit; the multi-channel video synchronous acquisition unit performs analog-to-digital conversion and synchronization on the received video stream information and compression processing, and delivered to the multi-channel video synchronous segmentation unit; the multi-channel video synchronous segmentation unit receives the synchronous video segmentation instruction of the multi-channel video synchronous acquisition unit, converts the information it receives into the same video format, Divide into a plurality of first static video frame groups according to time sequence, and transfer the first static video frame groups to the multi-channel video side view changing top view unit; wherein, each of the first static video frame groups is Including the n-channel video stream information collected by the multi-camera video acquisition array at the same moment; the static image corresponding to each channel of video stream information in the first static video frame group to be received by the multi-channel video side view changing top view unit , converted into a top view of the object to be photographed, forming a second static video frame group, and passing the second static video frame group to the GPGPU video frame positioning, splicing, compensation and fusion unit; the GPGPU video frame positioning, The splicing, compensation and fusion unit performs positioning, splicing, compensation and fusion on each video frame in the second static video frame group to obtain a panoramic video frame of the object to be photographed, and transfers the panoramic video frame to the A real-time panoramic video stream generating unit; the real-time panoramic video stream generating unit synthesizes the received panoramic video frames at different times into a real-time panoramic video stream according to time sequence. The above-mentioned GPGPU video frame positioning, splicing, compensation and fusion units are based on the CUDA parallel computing framework.

Multi-Camera Video Acquisition Array

The multi-camera video acquisition array is a camera array composed of n cameras installed according to fixed installation parameters. As shown in Figure 2, each camera in the array uses lenses with different viewing angles and different shooting angles. Realize the basic coverage of the acquisition scene U1. However, there is no effective splicing overlapping area between the camera images U2 of the same scene U1 collected, and there are even small gaps and loopholes.

Multi-channel video synchronous acquisition unit

As shown in Figure 2, this unit is composed of multiple video capture cards with multi-channel video synchronous capture function. The working process is that the multi-channel video synchronous acquisition unit U4 converts the n-channel analog signals of the multi-camera video acquisition array U3 video sources into digital signals through the A/D conversion module on the acquisition card, and then transmits them to the board itself. In the storage belt, the built-in video compression chip and video synchronization chip on the video capture card execute synchronization and compression algorithms for each channel of video, so that the huge video signal is synchronized, compressed and reduced to form n channels of video streams, Then pass it to the multi-channel video synchronous segmentation unit U5 to complete the whole workflow.

Multi-channel Video Synchronous Segmentation Unit

This unit is an FPGA programmable hardware platform, and a parallel image processing algorithm hardware logic circuit is preloaded in the platform. Sequential time sequence is divided into several static sub-image groups (in particular, "video frame" and "sub-image" are not distinguished in the present invention), and each group is composed of n static images at the same moment of n road video streams, and at the same time Then the image groups at each moment are sequentially transmitted to the multi-channel video frame side view changing top view unit U6 to complete the entire working process of this unit.

Multi-channel video frame side view variable top view unit

In order to save the cost of the device and improve the integration of the device, as shown in Figure 2, this unit and the multi-channel video synchronization division unit U5 are implemented on the same FPGA programmable hardware platform. The hardware logic circuit of the core image conversion algorithm of this unit has also been preloaded in the platform as a follow-up algorithm of the multi-channel video synchronization segmentation unit U5. The bird's-eye view of the subject being photographed; the present invention has built a geometric transformation model based on multi-camera installation parameters in the image transformation algorithm for changing the bird's-eye view from the side view, and its specific steps are as follows:

(1) According to the principle of camera imaging, the coordinate transformation equation from the camera’s physical side-view plane coordinate system (x, y, z) to the camera’s virtual top-view coordinate system (X, Y, Z) can be established as shown in the following formula,

The specific meaning of each parameter is as follows:

s scale factor

f _x ,f _y the focal length of the camera

c _x , c _y image correction parameters

Three column vectors of R _x , R _y , R _z rotation matrix

t translation vector

(2) Calibrate the imaging parameters of the camera according to the installation parameters of the multi-camera and the standard images taken by the camera, and obtain the imaging parameters required for the side view to the top view. The process is shown in Figure 3. Taking the black and white checkerboard U61 as an example, the calibration point U63 required for the calculation of parameters is calibrated through the imaging calibration program, and the equations required for calculating the parameters are established by using the position changes of the calibration points, so as to solve the side The view-to-top view transforms the values of the parameters in the model to complete the calibration of camera imaging parameters.

(3) By using the calibrated camera parameters and the image geometric transformation model of changing the side view to the top view established on this basis, the black and white checkerboard side view U61 as shown in Figure 3 can be transformed into the black and white checkerboard top view U62 .

To sum up, the workflow of the multi-channel video frame side view changing top view unit U6 is as follows: first, it receives the sub-image group (the first static video frame group) at a certain moment delivered by the multi-channel video synchronous segmentation unit U5, and then according to the camera The numbering sequence uses the image geometric transformation model based on multi-camera installation parameters to convert the side view of the atomic image group into a top view, and transfers the converted sub-image group as a new sub-image group to the GPGPU video frame positioning, splicing, compensation and fusion unit U7, and is ready to accept the next sub-image group, and so on to complete the entire workflow.

GPGPU video frame positioning, splicing, compensation and fusion unit

This unit, as the key unit of the present invention, is an image processing software system using NVIDIA's high-performance GPU as the hardware platform. It is developed based on the CUDA parallel computing framework, and consists of the sub-image positioning function module U71, the sub-image group fixed-point orientation panoramic rough mosaic function module U72, the sub-image group seam compensation function module U73, and the sub-image group seam fusion function module U74 , composed of four functional sub-modules. Each of the above-mentioned functional submodules is a kind of image processing algorithm proposed by the present invention, and its principle is described as follows respectively:

(1) Sub-image positioning function module

According to the installation method of a single camera of the multi-camera video acquisition array is fixed, as shown in Figure 4, combined with the principle of the formation of the camera scene, the positioning model of the spliced sub-image can be established to determine the specific area shot by each camera. The steps are as follows,

1) First, according to the fixed-point installation parameters of the single camera U711 shown in Figure 4, the camera lens center point p ₀ U712 and the intersection point p ₁ U713 of the camera center line l ₀ U714 and the xoy plane where the subject is located can be determined, and established in Figure 4 The coordinates in the (x, y, z) coordinate system are p ₀ (x ₀ , y ₀ , z ₀ ) and p ₁ (x ₁ , y ₁ , z ₁ ), then the spatial straight line equation of the camera centerline l ₀ As shown in the following formula,

2) Secondly, according to the directional installation parameters of the single camera U711 shown in Figure 4, the spatial orientation angle of the centerline U714 of the camera lens can be determined, and combined with the camera's field of view angle, the generatrix l ₁ U712 of the cone forming the scene area can be determined by the camera U711 direction angle (α, β, γ), and because the bus line l ₁ U712 passes through the midpoint p ₀ (x ₀ , y ₀ , z ₀ ) of the camera lens, the equation of the space straight line l ₁ U712 is shown in the following formula,

3) Finally, from the latitude circle method, it can be seen that the scene curve Γ ₂ U717 formed by the camera U711 on the xoy plane can be obtained by eliminating the parameters x ₂ , y ₂ , and z ₂ by the following formula,

Wherein, the point M ₁ is the intersection point on any latitude circle Γ ₁ U715 and the bus line l ₁ , and the coordinates are (x ₂ , y ₂ , z ₂ ).

To sum up, the sub-image positioning function module only needs to establish the scene curve equation of each camera according to the fixed-point orientation installation parameters of each camera in the multi-camera video acquisition array U3 in Figure 2, so as to predict each splicing The area and size of the sub-image in the panorama to realize the positioning of the spliced sub-image in the panorama.

(2) Sub-image group fixed-point orientation panoramic rough mosaic function module

The function of this functional sub-module is to convert the multi-channel video frame side view to the top view sub-image group obtained by the top-view sub-image group under the action of the sub-image positioning function module to perform rough splicing of the panorama. The specific workflow is as follows: firstly, generate a blank image with the same size as the panorama area according to preset settings; secondly, use the sub-image positioning function in Fig. 5 for each sub-image in the received overlooking sub-image group Module U71 performs positioning processing in turn to determine the position, size and direction of each sub-image in the blank image; again, according to the predetermined label sequence of each camera in the multi-camera array and the positioning information of its captured sub-images, the sub-images are divided one by one. Fill in the corresponding places in the blank image to realize the rough stitching of the panorama; finally, calibrate the stitching seam of the rough stitched panorama, and mark the overlapping area, seam or hole area and the seamless stitching area in the stitching seam. The specific position, shape and size of the area complete the entire workflow of the sub-image group fixed-point directional panoramic rough mosaic function module U72.

(3) Sub-image group seam compensation function module

As shown in Figure 6, this functional sub-module consists of sub-image line feature extraction sub-module U731, line feature matching sub-module U732 between adjacent sub-images and adjacent area line feature extrapolation boundary point compensation seam and loophole sub-module U733 three sub-modules Composition, and its working principle is described as follows,

1) sub-image line feature extraction sub-module

In this sub-module, to extract the 2D line features of each sub-image, the step boundary in the image must be obtained first. In the present invention, the RoA algorithm is used to detect the boundary of the sub-image, and the algorithm determines whether the target pixel is an edge point by calculating the mean ratio of adjacent areas. Because this method uses the intensity mean value of adjacent regions, it greatly reduces the strong fluctuation of a single pixel caused by speckle noise, making the line feature of the sub-image obtained by this method more reliable. In order to reduce the amount of calculation, for each sub-image that needs to extract line features, only the line features in a certain area adjacent to the stitching seam are extracted. The algorithm is done by comparing adjacent areas along a certain direction. The steps are as follows. First, assume that C(x,y) is the center pixel point, and set L(x,y) and R(x,y) respectively as the left side of C(x,y) along a certain direction. , the average gray value of the right adjacent area, the mean ratio is estimated as follows;

RoA: C(x,y)=max{R(x,y)/L(x,y), L(x,y)/R(x,y)}

Then, compare RoA:C(x,y) with a predetermined threshold T ₀ , when RoA:C(x,y) is greater than the threshold, point C is considered as a boundary point; finally, the sub-image will be extracted by the above algorithm The line feature fragments in are reorganized into meaningful line features by certain means to complete the function of the entire functional sub-module.

2) Line feature matching sub-module between adjacent sub-images

In this sub-module, to match the two-dimensional line features extracted between adjacent sub-images, the line features must be mathematicalized first. The present invention uses a mathematical fitting method to describe the line features extracted from adjacent sub-images with corresponding line segment functions. Taking stitching holes as an example, it is assumed that there are n sub-images surrounding the stitching holes. First, the set I composed of the slope of the line segment function of each sub-image extracted is represented by the following formula,

Among them, the subscripts such as m, n, and l all represent the total number of line features extracted in the corresponding sub-image; then use the following formula to achieve line feature matching between sub-images,

in, are any elements in the set I, but Not representing the same element at the same time, T ₁ is a small positive number for the matching threshold; finally, the perfectly matched line features are recombined into line feature pairs to complete the line feature matching process of the entire sub-image.

Taking the stitching vulnerability T734 shown in Figure 7 as an example, the above-mentioned line feature matching process can be described as the following steps: First, extract the line features of the left picture T731, the right picture T732, and the bottom picture T733 as shown in the figure, respectively {T7311 , T7312, T7313, T7314}, {T7321, T7322}, {T7331, T7332, T7333}; then, use the line feature matching algorithm of the line feature matching formula to match the line features extracted from the three images; finally, the matched line The features are recombined into line feature pairs as {(T7311, T7331), (T7312, T7332), (T7313, T7322, T7333), (T7314, T7321)}, thus completing the adjacent splicing loopholes shown in Figure 7 Line feature matching process for three images.

3) Line feature extrapolation of adjacent areas Compensation seam and loophole sub-module of boundary point

In this sub-module, due to the splicing loophole T734 shown in Figure 7, to compensate the image of the loophole T734, it is necessary to use the existing image information for extrapolation. In the present invention, the compensation is performed by using the matched line feature pairs in FIG. 7 obtained in the line feature matching sub-module U732 between adjacent sub-images. Specifically, firstly, according to the line segment function of the existing matching line feature pair, a line segment function that can satisfy all the line features in the matching feature pair is constructed, and at the same time, this function is considered to be a reasonable fit for the line feature of the splicing loophole; then Use the newly constructed line segment function to extrapolate the splicing loopholes, and then match the location of the line features determined by the matching line feature pair; finally, use the corresponding matching line features in the original stitching sub-image to extract the loophole line features Blend and fill the gray value, color and brightness of the image. After performing similar processing on all matching line feature pairs as above, the line feature images compensated for stitching holes in Figure 7 can be realized, as shown in T7341, T342, T343, and T344. By comparing with the original image T735 at stitching holes, it is obvious that the hole image compensated by extrapolating boundary point compensation seams and hole sub-modules through adjacent area line features is basically consistent with the original image.

To sum up, the workflow of the sub-image group stitching seam compensation function module is shown in Figure 6. First, through the sub-image line feature extraction function sub-module U731, the line features of all sub-images in the rough stitching video frame panorama are extracted and extracted. The slope of the line segment function of each sub-image forms a set; secondly, the line feature matching function sub-module U732 between adjacent sub-images completes the line feature matching of all sub-images, and obtains all matching line feature pairs between sub-images; again, adjacent sub-images Regional line feature extrapolation boundary point compensation seam and hole function sub-module U733 reconstructs a line segment function that can satisfy all the line features in the matching feature pair according to the line segment function of the existing matching line feature pair to perform hole and seam line Feature extrapolation and fitting, so as to determine the position of the hole and the centerline feature of the seam; finally, use the gray value, color and brightness of the original spliced sub-image to fuse and fill the centerline feature and image of the hole, so as to obtain no seam and hole The panorama of the video frame to complete the entire workflow of this functional module.

(4) Sub-image group seam fusion function module

According to the previous analysis, in order to obtain a natural and complete video frame panorama without stitching seams and loopholes by utilizing the sub-image group seam compensation function module U73, it is necessary to eliminate the gap between adjacent sub-images and between adjacent sub-images and compensation. The difference in grayscale, color and brightness between stitching seams and stitching holes must be blended with grayscale, color and brightness. In the present invention, the sub-image group splicing seam fusion function module U74 adopts the method proposed by Szeliski to carry out the fusion operation. Or the grayscale, color and brightness value of a certain point P in the splicing hole can be obtained from the grayscale, color and brightness value of the point closest to point P in the adjacent area of this splicing seam or splicing leak in m sub-images , calculated by the following formula,

Among them, g(p) represents any one of the gray value, color value and brightness value of point P, and g _i ( _xi , y _i ) represents the point corresponding to g(p) of the i-th image closest to point P Gray value, color value or brightness value of . The function ξ(x) is a linear weight function, which is determined by the distance between the closest point of the i-th image to point P and point P. The greater the distance, the greater the weight. When the distance is the maximum distance, the weight is 1. Otherwise, when the distance is the minimum distance Weight is 0. By performing the above-mentioned fusion operation on each pixel in the stitching seam and the stitching hole one by one, a natural and complete video frame panorama without stitching seams and holes can be obtained, and the stitching seam compensation function module of the sub-image group can be realized Function of U73.

By explaining the working principle of each sub-function module of the GPGPU video frame positioning, splicing, compensation and fusion unit one by one above, the workflow of the GPGPU video frame positioning, splicing, compensation and fusion unit U7 can be described as shown in Figure 5 : Firstly, the overlooking sub-image group output by the multi-channel video frame side view changing top view unit U6 is used as the original image group of panoramic image mosaic, under the processing of sub-image positioning function sub-module U71, it is predicted that each mosaic sub-image is in the panorama secondly, in the sub-image group fixed-point orientation panorama rough stitching function submodule U72, utilize the sub-image positioning information of the sub-image positioning function sub-module U71 to carry out the rough stitching of the panorama, and obtain stitching seams and stitching The video frame panorama of loophole; Again, will have the video frame panorama of splicing seam and splicing loophole to send in sub-image group stitching seam compensation function submodule U73 and carry out the compensation of splicing seam and splicing loophole; Finally, through sub-image group The image group seam fusion function sub-module U74 performs fusion processing of seams and stitching holes to obtain a natural and complete panorama of video frames without seams and holes, and completes positioning, stitching, compensation and fusion of GPGPU video frames The entire workflow of unit U7.

Real-time panoramic video stream generation unit

This unit is used as the output unit of the present invention as shown in FIG. 2 , and the real-time panoramic video stream generation unit U9 is also a video stream generation software system based on the high-performance GPU of NVIDIA Corporation as the hardware platform. In this unit, the multi-thread scheduling mechanism and the CUDA parallel computing architecture are used to position, splice, compensate and fuse the GPGPU video frame into a natural and complete video frame panorama without seams and holes (panoramic video frame ), form a video stream at 24 frames per second in the order of time. At the same time, through a simple video compression algorithm, the video stream is compressed into a commonly used video format for storage. Complete the entire workflow of this unit.

The beneficial effects of the present invention are:

The present invention provides a real-time video splicing method for splicing images without effective overlap and variable structure collected in the same scene from different angles of view and different directions, and then generating video streams in real time; the method is based on acquisition device installation parameters, performance parameters and The linear feature of the image is collected, and the image positioning model, transformation model and compensation fusion model are established. By using these efficient image processing models, the splicing method of the present invention not only improves the accuracy of image splicing, but also ensures the efficiency of image splicing. Real-time requirements for video stream splicing.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a kind of fixed-point directional video real-time mosaic method without effective overlapping variable structure of the present invention;

FIG. 2 is a schematic structural diagram of a video splicing device using a splicing method provided by an embodiment of the present invention;

Fig. 3 is a schematic diagram of a black and white Qige side view changed to a top view provided by an embodiment of the present invention

FIG. 4 is a schematic diagram of a sub-image positioning function module provided by an embodiment of the present invention;

Fig. 5 is the structural block diagram of GPGPU video frame positioning, splicing, compensation and fusion unit provided by the embodiment of the present invention;

Fig. 6 is a structural block diagram of a sub-image group stitching seam compensation function module provided by an embodiment of the present invention;

Fig. 7 is a schematic diagram of the principle of the sub-image group splicing seam compensation function module provided by the embodiment of the present invention;

FIG. 8 is a flow chart of a real-time mosaic method for fixed-point directional video without effective overlapping variable structure according to an embodiment of the present invention.

detailed description

The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. The following examples are used to illustrate the present invention, but should not be used to limit the scope of the present invention.

In conjunction with Fig. 8, the specific implementation of the present invention is further described. The present invention is applied to a 2650m3 blast furnace in China, and ^three cameras are installed from different directions of the blast furnace to take pictures of the material surface from the side, and are used to obtain a diameter of 8.2 meters. The material surface of the round surface. Due to the dark, high temperature and dusty environment in the blast furnace, it is impossible to use a single camera to capture the entire material surface of the blast furnace, and each camera can only be fixed and fixed at a fixed point. This satisfies the need for three cameras to collect images of the same material surface at different viewing angles and different directions, and there is no effective overlapping area for image splicing between the collected images, and even there are small gaps and loopholes between the images. premise. First, install the necessary equipment of the video splicing device according to Figure 2. After the equipment is installed, start video splicing of the collected video information according to the process in Figure 1. The working steps are as follows,

1. According to the multi-camera installation parameters S61, calibrate the camera imaging parameters S62 of the installed 3 cameras for shooting the material surface, and build the side view variable top view geometric transformation model S63 again, so that the side view variable top view S6 of the sub-image group can be used;

2. Utilize the fixed-point orientation parameters of the three cameras and the plane S711 where the photographed object is located to construct a positioning model S712 for splicing sub-images in the panorama, and use the positioning model S712 for splicing sub-images in the panorama to determine the position of the sub-images in the panoramic image Position S713, position S714 of sub-image splicing loopholes and breaks, sub-image splicing overlapping area S715, and bordering relationship between sub-images and between sub-images and loopholes and breaks S716, so as to facilitate later splicing and use;

3. The three camera arrays installed on the top of the blast furnace collect videos of different positions on the blast furnace material surface respectively, forming a multi-camera video sequence S4, and the ith moment obtained by synchronously segmenting the video stream S51 through the video stream synchronous segmentation unit The sub-image group S52 corresponding to the i-th frame to be spliced;

4. Using the image geometric transformation model S63 constructed in step 1, the side view of the sub-image group S52 to be stitched in the i-th frame is changed from a side view to a top view;

5. Using the position S713 of the sub-image determined in step 2 in the panoramic image, perform fixed-point directional panoramic stitching S721 of the sub-image group by converting it into a top view;

6. According to the rough panorama mosaic image obtained in step 5, by judging the seam S722, the seam area is divided into three situations: seams with overlapping areas, seams with no holes and no overlaps, and seams with holes seams of cracks;

7. For the stitching seams with overlapping areas determined in step 6, use the specific position of the sub-image stitching overlapped area S715 obtained in step 2, and use the traditional brightness and color similarity matching method to directly stitch and fuse S741;

8. For the seamless, hole-free and non-overlapping stitching seams determined in step 6, use brightness and color fusion at the seams to stitch S742;

9. For the splicing seams with holes and cracks determined in step 6, first use the border relationship S716 between the sub-images and the sub-images and holes and fractures determined in step 2 to determine the sub-images adjacent to the splicing holes and seams S731 ; Secondly, extract the line feature and match S732 to the adjacent area of the sub-image, and on this basis obtain the line feature S733 of the seam and the hole by extrapolating the boundary point of the line feature of the adjacent area, so as to realize the compensation of the line of the hole and the seam Features S734; Finally, use color and brightness interpolation fusion for holes and other parts S735;

10. Through steps 7, 8, and 9, the complete panorama of frame i at the i-th moment can be obtained, and at the same time, jump back to step 3 to perform panorama stitching on the sub-image groups collected by the 3 cameras at the i+1th moment, and so on. Obtain the image sequence of the entire blast furnace panoramic material surface distributed over time; use the acquired image sequence of the blast furnace panoramic material surface distributed over time to synthesize a real-time video stream, thereby obtaining real-time video information of the blast furnace panoramic material surface.

The above embodiments are only used to illustrate the present invention, but not to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications or equivalent replacements of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all should cover Within the scope of the claims of the present invention.

Claims (10)

1. A fixed point directional video real-time splicing method without an effective overlapping variable structure is characterized by comprising the following steps:

the method comprises the following steps that firstly, a multi-camera video acquisition array is installed, video stream information of different positions is acquired respectively, and analog-to-digital conversion, synchronization and compression processing are carried out on the video stream information;

converting the compressed video stream information into the same video format, and dividing the video stream information into a plurality of first static video frame groups according to a time sequence; each first static video frame group comprises n paths of video stream information collected by the multi-camera video collection array at the same moment;

converting a static image corresponding to each path of video stream information in the first static video frame group into a top view of the shot object according to a side view-to-top view geometric model to form a second static video frame group;

fourthly, positioning each video frame in the second static video frame group according to the positioning model, and performing panoramic rough splicing to obtain a panoramic rough splicing image;

fifthly, determining the splicing seam position with an overlapped area, the splicing seam position without a seam or a hole or a cracked area and the splicing seam position without a hole or a cracked area in the panoramic rough splicing map according to the positioning model;

step six, for the splicing seams with the overlapped areas or the splicing seams without the overlapped areas, the splicing seams are fused and spliced by utilizing a brightness and color interpolation algorithm;

seventhly, the splicing process of the splicing seams of the areas with holes or cracks is as follows:

determining the bordering relations between the sub-images corresponding to each video frame in the second static video frame group and between the sub-images and the loopholes or cracks according to the positioning model, and determining loophole or crack sub-images according to the bordering relations;

extracting line features of sub-images adjacent to the leak or crack sub-images;

matching the extracted line features to obtain line feature pairs;

utilizing the characteristics of adjacent area lines to extrapolate boundary points to compensate cracks or loopholes;

performing fusion splicing on the splicing seams of the panoramic rough splicing image after the loophole or crack compensation by using a brightness and color interpolation algorithm to obtain panoramic video frames after panoramic fine splicing;

and step eight, processing the first static video frame groups at different moments according to the steps three to seven to obtain panoramic video frames at different moments, and synthesizing the panoramic video frames according to a time sequence to obtain a real-time panoramic video stream.

2. The method according to claim 1, wherein step two is performed after receiving the synchronized video segmentation command, and the first still video frame is stored in chronological order after the step two is completed.

3. The method of claim 2, wherein the side view to top view geometric model in step three is:

wherein s is a scale factor, f_x,f_yIs the focal length of the camera, c_x,c_yAs an image correction parameter, R_x,R_y,R_zIs the three column vectors of the rotation matrix, t is the translation vector, (X, Y, Z) is the coordinates of the elements of the static image side view, and (X, Y, Z) is the top view coordinates of the corresponding elements.

4. The method of claim 3, wherein the positioning model in the fourth and fifth steps is:

wherein x is₀,y₀,z₀As coordinates of the camera lens center point, x₁,y₁,z₁The coordinate of the intersection point of the shot object and the camera plane xoy is (α, gamma) the direction angle of the cone generatrix of the corresponding scene of the camera, and x₂,y₂,z₂And the coordinates of the intersection point of the latitude circle of the corresponding field of the camera and the generatrix of the cone are shown, and x, y and z are the coordinates of the intersection point of the corresponding field of the camera and the camera plane xoy.

5. The method according to claim 4, wherein the full scene rough stitching in the fourth step is specifically:

firstly, generating a blank image with the size equal to the size of a panoramic image field of a shot object;

secondly, positioning the subimage corresponding to each video frame in the second static video frame group by using the positioning model, and determining the position, size and direction of each subimage in the blank image;

and filling the sub-images into corresponding positions in the blank image one by one according to the preset label sequence of each camera in the multi-camera array and the positioning information of the sub-images shot by the cameras, so as to realize rough splicing of the panoramic image.

6. The method according to claim 5, wherein the seventh step of extracting line features of sub-images adjacent to the leak or crack sub-image is specifically:

assuming that C (x, y) is taken as a central pixel point, and L (x, y) and R (x, y) are respectively the average gray values of left and right adjacent areas of the C (x, y) along a certain direction, the average ratio estimation is shown as formula (3);

then, RoA: C (x, y) is compared with a predetermined threshold value T₀Comparing when RoA: C (x, y) is greater than threshold T₀If so, the point C is considered as a boundary point;

and extracting line feature segments in the sub-images adjacent to the leak or crack sub-images through the algorithm, and reorganizing the line feature segments into line features.

7. The method according to claim 6, wherein in the seventh step, the matching of the extracted line features specifically includes:

describing the line features by corresponding line segment functions, assuming that there are n sub-images surrounding the leak or crack, first, the slope of the line segment function of each extracted sub-image is expressed by formula (4) as follows,

wherein, m, n, l all represent the total number of the line features extracted from the corresponding sub-images;

line feature matching between sub-images is achieved using the following equation (5),

wherein,are any one element of the set I, T₁Is a matching threshold; and (5) the line feature is successfully paired.

8. The method according to claim 7, wherein in the seventh step, the step of compensating for the crack or the leak by using the adjacent area line feature extrapolation boundary point specifically comprises:

firstly, according to the corresponding first line segment function matched with the line feature pair, constructing a second line segment function which can meet all line features in the corresponding feature pair, and meanwhile, considering that the second line segment function is reasonable fitting for the line features of the leak or the crack;

then, extrapolating the position of the hole or crack by using the second line segment function, thereby determining the position of the line feature of the matched line feature pair;

and finally, fusing and splicing the spliced seam by using the color and brightness interpolation algorithm according to the color and brightness of the matched line feature pair corresponding to the sub-image adjacent to the sub-image of the leak or crack.

9. The method according to claim 8, wherein in the sixth and seventh steps, the fusion splicing by using a color and brightness interpolation algorithm specifically comprises:

assuming that m sub-images adjacent to the leak or the crack sub-image exist, the gray scale, the color and the brightness value of one point P in the crack or the leak can be obtained by calculation according to the gray scale, the color and the brightness value of the point closest to the point P in the m sub-images through the formula (6)

Wherein g (P) represents any one of the gray value, color value and brightness value of the P point, g_i(x_i,y_i) A gray value, a color value or a luminance value corresponding to g (P) representing the closest point of the ith sub-image to the point P, the function ξ (x) being a linear weighting function;

and performing the fusion operation on each pixel point in the crack or the leak one by one to obtain the complete panoramic video frame.

10. The method according to claim 9, wherein in step eight, the panoramic video stream is compressed, stored and displayed.