CN103248911B - A Virtual Viewpoint Rendering Method Based on Space-Time Combination in Multi-viewpoint Video - Google Patents

Abstract

The invention proposes a new view synthesis method based on a depth map. First, obtain the color image and depth image of the virtual viewpoint through 3D image transformation, and remove small holes and mapping error points; then fill the holes in the depth image of the virtual viewpoint, and record the pixel coordinates of the holes at the same time; after that, reverse 3D image transformation, locate the target area (the area that causes the hole) in the target frame of the reference viewpoint, and use the front and rear frames to restore the background of the target area; finally, use the sample-based image repair algorithm to repair the remaining holes. In this method, by using the image information of the front and back frames, the space-time domain is combined to fill holes, which can make the results more accurate and improve the quality of virtual viewpoint images compared with purely space-based hole filling operations; in addition, through reverse mapping , locate the target area, perform background restoration in a targeted manner, and perform large-scale background restoration to achieve more efficient hole filling when the results are similar.

Description

A Virtual Viewpoint Rendering Method Based on Space-Time Combination in Multi-viewpoint Video

technical field

The invention relates to a method for synthesizing virtual viewpoints based on depth images, and belongs to the technical field of video and multimedia signal processing.

Background technique

With good user interaction and vivid visual experience, 3D TV has become the leader of the new generation of multimedia technology. The construction of the depth information of the real scene makes the stereoscopic TV give people a feeling that the scene is stretched out of the screen and can be touched by hand. Among them, multi-viewpoint video is considered to have extremely broad application prospects. Its main realization goal is that on the playback side, for the same scene, users can choose different viewpoints to enjoy the scene from different angles according to their own needs, so as to obtain a strong sense of presence and reality. However, the limitation of transmission bandwidth and transmission rate increases the difficulty of realizing multi-view video. Therefore, virtual view synthesis technology came into being.

There are many methods for virtual view synthesis, among which, view synthesis based on depth images is a typical method. It can use the color image and depth image of one video to synthesize video information of other arbitrary viewpoints. In the process of view synthesis, the occluded background part in the original view may be exposed in the virtual view, which is the so-called hole problem. Since this part of the exposed area cannot find reference information in the original view, the hole problem has become the most challenging problem in view synthesis. The problem of how to complete view synthesis efficiently and with high quality is transformed into how to accurately fill the hole. The problem.

In the traditional method of view synthesis based on depth image, it is mainly divided into three basic steps: smoothing of reference view depth image, obtaining virtual view depth image and color image through 3D image transformation, and hole filling. There will be some small cracks and large hollow areas in the virtual viewpoint color images obtained by such methods. Especially for the latter, no matter whether it is filled by neighborhood assignment, linear interpolation, or image restoration algorithm with high complexity, it is only a process of guessing using the surrounding information of the hole in the airspace, and it is difficult to have a high In terms of credibility, the obtained virtual viewpoint images are also difficult to accurately reflect the real scene.

Contents of the invention

In order to solve the inaccurate problem of virtual viewpoint synthesis caused by image post-processing based on spatial domain in the existing viewpoint synthesis method based on depth image, the present invention proposes a new viewpoint synthesis method based on depth map. Since in the reference viewpoint, the occluded background parts (i.e., the parts that cause holes in the virtual viewpoint) will be exposed over time, this means that we can find the unfixed Accurate information on the hollow part of the . Taking advantage of this, the present invention proposes a method for combining the space domain and the time domain to perform virtual viewpoint synthesis, in which some holes can be filled more accurately with time domain information, and for other holes that cannot be filled with time domain information , still use the method of hole filling in the airspace to fill. In this way, since some of the holes can be said to be filled with real background information, the overall performance is better than speculative filling in a single image.

In the present invention, firstly, the color image and the depth image of the virtual viewpoint are obtained through 3D image conversion, and small cracks and pixels with wrong mapping are removed; Record the pixel coordinates of the hole; after that, reverse the 3D image transformation, locate the target area in the target frame of the reference viewpoint (causing the occlusion area generated by the hole), and use the front and rear frames to restore the background of the target area in the target frame , which is used to fill part of the holes in the virtual viewpoint color image; finally, the remaining holes are repaired using the sample-based image inpainting algorithm. In this method, by using the image information of the front and rear frames of the reference viewpoint, the space-time domain is combined to fill holes, which can make the result more accurate and improve the quality of the virtual viewpoint image compared with the hole filling operation based solely on the space domain. In addition, through the reverse 3D image transformation, the target area is located, and the background restoration is carried out in a targeted manner. Large-scale background restoration can achieve more efficient hole filling with similar performance.

The technical scheme adopted in the present invention is:

A view synthesis method based on a depth map, characterized in that, using a combination of space domain and time domain, using the image information of the frame before and after the target frame in the reference view, and based on the positioning results of the occlusion area that causes the hole to be generated, perform Targeted background restoration, so as to fill some holes in the virtual viewpoint image, the specific steps are as follows:

(1) 3D image transformation: According to the principle of camera projection, through 3D image transformation, the image information of the reference viewpoint is projected onto the virtual viewpoint, and the color image and depth image of the virtual viewpoint are obtained, and the small cracks and wrongly mapped pixels are corrected. point for processing;

(2) Coordinate recording and filling of the holes in the depth image of the virtual viewpoint: record the coordinates of the holes in the depth image, and then use the neighborhood assignment to fill the holes in the depth image;

(3) Reverse 3D image transformation to locate the target area: starting from the virtual viewpoint, according to the principle of camera projection, perform reverse 3D image transformation, and at the same time refer to the target in the viewpoint target frame according to the coordinate recording results in step (2) positioning of the area;

(4) Restoring the background of the target area in the time domain: using the frame information before and after the target frame in the reference viewpoint to restore the background of the target area, which is used to fill some holes in the color image of the virtual viewpoint;

(5) Restoration of remaining holes: The remaining holes in the virtual viewpoint color image are filled using sample-based image restoration technology.

The specific steps of the step (1) are:

a. According to the depth information of the reference viewpoint, realize the coordinate transformation from the reference viewpoint plane to the three-dimensional space, and then from the three-dimensional space to the virtual viewpoint plane;

b. According to the result of coordinate transformation, project the pixels in the reference viewpoint into the virtual viewpoint plane, thereby obtaining the virtual viewpoint color image and depth image;

c. Detect the wrongly mapped points in the virtual viewpoint color image and depth image, and perform neighborhood assignment. At the same time, for some small cracks, also use the neighborhood assignment method to fill.

The specific implementation steps of the step (2) are as follows:

a. Record the coordinates of the large hole pixels in the depth image of the virtual viewpoint and the color image;

b. Use background pixels adjacent to the hole area to fill holes in the virtual viewpoint depth image line by line, and obtain a complete virtual viewpoint depth image.

The specific implementation steps of the step (3) are as follows:

a. According to the virtual viewpoint depth image obtained in step (2), realize the coordinate transformation from the virtual viewpoint plane to the three-dimensional space, and then from the three-dimensional space to the reference viewpoint plane;

b. According to the coordinate transformation result and the coordinate recording result in step (2), locate the target area in the target frame of the reference viewpoint.

The specific implementation steps of the step (4) are as follows:

a. In the time domain, use the front and back frames to restore the background of the target area in the target frame;

b. According to the coordinate transformation result in step (3), fill some holes in the color image in the virtual viewpoint.

The specific implementation steps of the step (5) are as follows:

a. Detect the edge of the hole area, calculate the priority of each pixel on the edge, and determine the order of repair;

b. Take the edge pixel as the center, obtain a sample block of a specific size, and use the sample block as a unit, and search for the best matching block from the source image according to the color information in it;

c. After finding the best matching block, copy the pixel information in the best matching block to the hole in the sample block to realize filling.

Description of drawings

Figure 1: Flowchart of the present invention.

Figure 2: Color and depth images of the reference viewpoint.

Figure 3: Color and depth images of the virtual viewpoint.

Figure 4: Color image of the virtual viewpoint after removal of small holes and mismapped points.

Figure 5: Virtual viewpoint depth image after hole filling.

Figure 6: Object regions in reference viewpoint object frames.

Figure 7: Foreground and background classification result map of the next frame image.

Figure 8: Color image of the target frame after background restoration.

Figure 9: Virtual viewpoint color image after partial hole repair.

Figure 10: Virtual viewpoint color image after remaining hole inpainting.

detailed description

The invention was experimented with a "Mobile" video sequence. The video sequence collects videos from 9 viewpoints of a scene, and provides corresponding depth information as well as internal and external parameters of each camera. In the experiment, we selected the No. 4 viewpoint as the reference viewpoint, and the No. 5 viewpoint as the virtual viewpoint.

Fig. 1 shows the flowchart of the present invention, according to the flowchart, we introduce its specific implementation.

(1) 3D image transformation. The so-called 3D image transformation is to project the pixels in the reference viewpoint into the virtual viewpoint plane according to the camera projection principle. This process is mainly divided into two parts. First, the pixels in the reference viewpoint are projected into the 3D space, and then projected from the 3D space into the virtual viewpoint plane. Figure 2 shows the color image and depth image of the reference viewpoint. Assuming that the internal reference matrix of camera i is A _i , and the translation matrix and rotation matrix in the external reference matrix are R _i and t _i respectively, the mapping equation can be expressed as:

sm＝A _i [R _i |t _i ]M (1)

Where s is a scalar, m=[u,v,1] ^T is the homogeneous coordinates of image pixels in the image coordinate system, M=[X,Y,Z,1] ^T is the corresponding space point in the world coordinate system The homogeneous coordinates below.

Using equation (1), the mapping from the reference viewpoint to the 3D space and the mapping from the 3D space to the virtual viewpoint can be realized. Fig. 3 is a color image and a depth image of a virtual viewpoint.

It can be seen from Figure 3 that there are some wrongly mapped pixels on the virtual viewpoint color image and depth image. This type of pixel is mainly divided into two types: the foreground pixel mapped to the background part and the foreground pixel mapped to the foreground part. background pixels. Since this type of wrong pixels usually has only one or two widths, we use their depth relationship with surrounding pixels to detect them, and then correct them by neighborhood assignment. In addition, it can also be seen from Figure 3 that there are some small cracks on the virtual viewpoint color image and depth image. For these cracks, we also use simple neighborhood assignment to eliminate them. Figure 4 is the processed virtual viewpoint color image.

(2) Hole filling and coordinate recording of virtual viewpoint depth images. To localize object regions in the reference viewpoint object frame, we record the coordinates of hollow regions in the virtual viewpoint depth image. Then, using the smooth gradient of the depth map, the background pixels adjacent to the hole area are used to fill the holes in the virtual viewpoint depth image line by line to obtain a complete virtual viewpoint depth image, as shown in Figure 5.

(3) Reverse 3D image transformation to locate the target area. The so-called reverse 3D image transformation is the inverse process of the traditional 3D image transformation. First project the pixels in the virtual viewpoint to the three-dimensional space, and then map the three-dimensional space to the plane of the reference viewpoint, and establish the corresponding relationship between the virtual viewpoint and the reference viewpoint. Then combine the coordinate recording results in step (2) to locate the target area. Figure 6 shows the target area for final positioning.

(4) The background restoration of the target area is carried out in the time domain. Since the exposed part of the virtual viewpoint image should be background information, what we are really interested in is the background part occluded by the foreground in the target area. Considering that the movement of the foreground object will expose the background part to varying degrees over time, the information of the frames before and after the target frame is particularly important in the background restoration process of the target area. In order to accurately and efficiently use the information of the front and back frames for background restoration, we use the K-means clustering algorithm to divide the depth images of the front and back frames into foreground and background. Only the image information divided into the background part can be used as candidate information for the target. The background of the frame is restored. The so-called clustering algorithm is the process of classifying and organizing data members that are similar in some respects in a data set. The K-means clustering algorithm is to manually or automatically determine K cluster centers according to a certain principle before dividing, and then assign the data in the data set to K cluster centers one by one according to a defined distance function. Whenever a data is classified into a class, the cluster center is updated by calculating the mean of the data in the class. Then continue clustering according to the updated cluster centers until all the data are classified. In this invention, we first perform histogram analysis on the depth image to be classified, determine the gray distribution of the foreground and background in the entire image, and use this as a basis to determine two (K=2) cluster centers, and then use the gray difference As a distance function, the division of foreground and background is completed according to the above steps. FIG. 7 is a diagram showing the division results of the foreground and the background of the next frame image of the target frame, wherein the black part represents the background, and the white part represents the foreground.

After determining the candidate information, considering that the influence of various environmental factors such as light intensity increases with the increase of the frame spacing, we start from the nearest adjacent frame and gradually restore the background of the target frame. Fig. 8 is the target frame after restoration. The background information in the target area can be used to fill some holes in the virtual viewpoint color image. FIG. 9 is a color image of a virtual viewpoint after partial hole filling is completed.

(5) Repair of remaining voids. The holes generated in the virtual viewpoint can be divided into two types according to their positions: the holes at the junction of the foreground and the background and the holes at the boundary of the scene. The former is background exposure due to occlusion, while the latter is the exposure of new scenes due to scene transitions. For the former, we can use the restored background information to fill in through the above steps. But for the latter, reference information cannot be found in all frames of the reference viewpoint. To address this part of the hole, we use a sample-based image inpainting algorithm for inpainting. Specific steps are as follows:

First, we detect the edge of the hole in the image to be repaired. After detecting all the pixels on the edge, we calculate the priority to determine the order of repair. The calculation formula of the priority is shown in (2).

P(p)=C(p)D(   (2)

Among them, C(p) is the confidence item, D(p) is the data item, defined as follows:

$\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; \&cap;</span>\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; ,</span>$ $\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; \&dtri;</span>\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&perp;</span>}<span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

The numerator in C(p) represents the number of non-hole pixels in the sample block, and the denominator | _Ψp | represents the total number of pixels in the sample block. In D(p), n _p represent the direction and normal vector of the iso-illuminance line at point p on the edge of the cavity, respectively. α is a normalization factor with a value of 255. Through C(p) and D(p), the sample-based image inpainting technique can fill in the texture while maintaining the linear structure of the image.

After the inpainting order is determined, we take the block with the highest priority as the sample block and search for the best matching block from the source image. The matching principles for searching for matching blocks are shown in equations (4), (5).

$\mathrm{<span\; class="notranslate">\; SSD</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; p</span>}}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; \&cap;</span>\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; p</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; p</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; G</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; p</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; the\; best</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; arg</span>}\underset{{\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; \&Phi;</span>}}{\mathrm{<span\; class="notranslate">\; min</span>}}\mathrm{<span\; class="notranslate">\; SSD</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; p</span>}}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

Equation (5) defines the calculation method of the error sum of squares (SSD). For each pair of pixel points in two pixel blocks, first calculate the sum of the squares of the R, G, and B component differences between them; then The calculation results of all pixel point pairs are added together, which is the error sum of squares (SSD) of the two pixel blocks.

In the matching process, the difference in color information between the non-hole pixels in the sample block and the pixels at the corresponding position in the matching block is calculated by using the error sum of squares (SSD). The smaller the difference, the higher the matching degree. The best matching block Ψ _{q_best} is finally determined by finding the minimum value among all SSDs.

After the best matching block is found, the pixels in the sample block are filled with the pixels in the best matching block. Then update the confidence, the formula is as follows:

$\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; C</span>}\stackrel{<span\; class="notranslate">\; ^</span>}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&ForAll;</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; \&Psi;</span>}}_{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; p</span>}}}<span\; class="notranslate">\; \&cap;</span>\mathrm{<span\; class="notranslate">\; \&Omega;</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

Repeat the above process until the repair is complete. The repaired virtual viewpoint color image is shown in Figure 10.

Claims (5)

1. the visual point synthesizing method based on depth map, it is characterized in that, adopt the mode that spatial domain and time domain combine, utilize the image information of the front and back frame of target frame in reference view, based on the positioning result to the occlusion area causing cavity to produce, carry out background targetedly to restore, thus fill the part cavity in virtual visual point image, concrete steps are as follows:

(1) 3D rendering conversion: according to video camera projection theory, converted by 3D rendering, with reference to the visual information projection of viewpoint on virtual view, obtain virtual view coloured image and depth image, and the pixel of little crackle wherein and mapping error is processed;

(2) empty in virtual viewpoint depth image coordinate record and filling thereof: carry out coordinate record to the hollow sectors in depth image, then utilize neighborhood assignment, carry out cavity to depth image and fill;

(3) reverse 3D rendering conversion, localizing objects region: from virtual view, according to video camera projection theory, carry out reverse 3D rendering conversion, carry out the location of target area in reference view target frame simultaneously according to the coordinate record result in step (2);

(4) background of carrying out target area in time domain is restored: the front and back frame information utilizing target frame in reference view, and the background of carrying out target area is restored, for the filling in part cavity in virtual view coloured image;

(5) reparation in residue cavity: utilize the image repair technology based on sample to fill the residue cavity in virtual view coloured image, specific implementation step is as follows:

A. detect the edge of hole region, the priority of each pixel in edge calculation, determine reparation order;

B., centered by edge pixel, obtain the sample block of specific dimensions, in units of sample block, according to colouring information wherein, from source images, search for best matching blocks;

C. after finding best matching blocks, the Pixel Information in best matching blocks is copied to the cavity place in sample block, realize filling.

2. the visual point synthesizing method based on depth map according to claim 1, is characterized in that, the concrete steps of described step (1) are:

A. according to the depth information of reference view, realize from reference view plane to three dimensions, then from three dimensions to the coordinate transform of virtual view plane;

B. according to the result of coordinate transform, project in virtual view plane with reference to the pixel in viewpoint, thus obtain virtual view coloured image and depth image;

C. the point of mapping error in virtual view coloured image and depth image is detected, and carry out neighborhood assignment, meanwhile, for the crackle that some of them are little, also use the mode of neighborhood assignment to fill.

3. the visual point synthesizing method based on depth map according to claim 1, is characterized in that, the specific implementation step of described step (2) is as follows:

A. coordinate record is carried out to the large empty pixel in virtual viewpoint depth image and coloured image;

B. the cavity using the contiguous background pixel of hole region to carry out virtual viewpoint depth image is line by line filled, and obtains complete virtual viewpoint depth image.

4. the visual point synthesizing method based on depth map according to claim 1, is characterized in that, the specific implementation step of described step (3) is as follows:

A. according to the virtual viewpoint depth image obtained in step (2), realize from virtual view plane to three dimensions, then from three dimensions to the coordinate transform of reference view plane;

B. according to the coordinate record result in coordinate transform result and step (2), localizing objects region in reference view target frame.

5. the visual point synthesizing method based on depth map according to claim 1, is characterized in that, the specific implementation step of described step (4) is as follows:

A., in time domain, before and after utilizing, frame carries out the background recovery of target area in target frame;

B. according to the coordinate transform result in step (3), the filling in part cavity in coloured image in virtual view is carried out.