CN101931819B - Temporal error concealment method - Google Patents

Abstract

The invention provides a temporal error concealment method, which comprises the following steps: detecting a damaged macroblock, the damaged macroblock consisting of 4 8 × 8 damaged blocks; obtaining motion vectors of a plurality of 4 x 4 neighboring blocks surrounding the damaged macroblock; and for each 8 × 8 damaged block, determining a predicted motion vector of the 8 × 8 damaged block from motion vectors of 6 4 × 4 neighboring blocks of the plurality of 4 × 4 neighboring blocks that are closest to the 8 × 8 damaged block.

Description

Temporal Error Concealment Method

technical field

The present invention relates to a temporal error concealment method, in particular to a method for predicting motion vectors from neighboring blocks, determining block partitioning modes based on the predicted motion vectors, and using partial distortion comparison A Temporal Error Concealment Method for Motion Vector Refinement Procedures. the

Background technique

As the application of multimedia technology becomes more and more popular, in order to effectively compress video files, many video coding techniques have been developed. The purpose of compression is to remove redundant data in image data, so as to reduce image storage space or image transmission volume. In the H.264 compression standard, two predictive coding techniques, intra prediction and inter prediction, are used. Intra-frame prediction utilizes the spatial correlation of adjacent blocks in the same frame for prediction, while inter-frame prediction utilizes the temporal correlation between adjacent frames for prediction. the

For inter-picture prediction, the H.264 compression standard defines 7 partition modes of different block sizes for each 16×16 macroblock: 16×16 (T1 mode), 16×8 (T2 mode), 8×16 (T3 mode), 8×8 (T4 mode), 8×4 (T5 mode), 4×8 (T6 mode), 4×4 (T7 mode), as shown in Figure 1. After encoding, each block can be represented by a residual value (residual) and a motion vector (motion vector). The smaller the size of the block coding is selected, the better the picture quality can be obtained, but the required calculation amount and time are also larger. Therefore, in order to balance the picture quality and coding efficiency, blocks of different sizes are generally used according to the picture complexity to obtain better compression performance. the

Video data is compressed into bitstreams that are easy to transmit and store. However, during the transmission of these highly compressed video bitstreams (especially in the wireless channel environment), it is easy to occur such as packet erasure, etc. question. In order to avoid affecting the video picture quality due to video packet loss, commonly used protection mechanisms include automatic repeat request (ARQ), forward error correction (FEC) and error concealment (Error Concealment). Compared with ARQ and FEC, error concealment technology does not require additional bandwidth, and has better performance in the case of broadcast (Broadcast) and multicast (Multicast). The error concealment techniques implemented at the decoder are mainly divided into two types: spatial error concealment and temporal error concealment. Spatial error concealment mainly utilizes spatial redundant information in the same picture to reply Damaged video sequences, while temporal error concealment uses the high correlation between successive pictures in the coded sequence to reconstruct the damaged sequence. Since there is generally a high correlation between adjacent pictures, compared with spatial error concealment, except in some special cases (such as scene transitions, sudden appearance or disappearance of objects, etc.), the temporal error concealment method Usually provides better picture quality. the

When the temporal error concealment method is used to reconstruct the damaged block, the motion vector of the damaged block needs to be obtained first because the information of other reference frames is used. At present, some simple methods have been developed to predict the motion vector of the damaged block, such as predicting the motion vector to be 0, using the average value of the motion vector of adjacent blocks in space, or using the same location block in the reference picture methods for moving vectors, etc. The above method can refer to "Error control and concealment for videocommunication: a review" published by Y.Wang et al. in Proc.IEEE, vol.86, no.5, pp.974-997, May 1998, by D. "Enhanced error concealment with mode selection" published by Agrafiotis et al in IEEE Trans. Circuits Syst. Video Technology, vol.16, no.8, pp.960-973, Aug.2006, published by S. Trans.Consumer Electronics, vol.47, no.3, pp.568-578, "Anefficient error concealment implementation for MPEG-4 video streams" published in Aug.2001, published by B.Yan et al. in IEEE Trans.Consumer "A novel selective motion vector matching algorithm for error concealment in MPEG-4 video transmission over error-prone channels" published in Electronics, vol.49, no.4, pp.1416-1423, Nov.2003, by J. Zhang "A cell-loss concealment technique for MPEG-2coded video" published in IEEE Trans.Circuits Syst.Video Technol., vol.10, no.4, pp.659-665, Jun.2000, by J.Y. "Robust error concealment for visual communications in burst-packet-loss networks" published by Pyun et al. in IEEE Trans.Consum.Electron., vol.49, no.4, pp.1013-1019, Nov.2003, and by S.C.Huang et al. inProc.Int.Conf.MultiMedia Modeling (MMM), Jan.2008 , LNCS 4903, "Temporal Error Concealment for H.264 Using Optimum Regression Plane" published in pp.402-412, the content of which will be incorporated into this article as a reference. the

In addition, for various other improved methods proposed for the temporal error concealment technique, reference may also be made to "Temporal error concealment using motion field interpolation", "Vector rational interpolation schemes for erroneous motion field" published by S.Tsekeridou et al. in IEEE Trans.Multimedia, vol.6, no.6, pp.876-885, Dec.2004 estimation applied to MPEG-2 error concealment", "A motion vector recovery algorithm for digital" published by J. Zheng et al. in IEEE Trans.Broadcast., vol.49, no.4, pp.383-389, Dec.2003 video using Lagrange interpolation", "Error-concealment algorithm for H.26L using first" published by J.Zheng et al. in IEEE Trans.Multimedia, vol.6, no.6, pp.801-805, Dec.2004 -order plane estimation", "Efficient motion vector recovery algorithm for H.264 based" published by J. Zheng et al. in IEEE Trans.Multimedia, vol.7, no.3, pp.507-513, Jun.2005 on apolynomial model", "A concealment method for video communications in an error-prone environment" published by S.Shirani et al. in IEEE Journal on Selected Areas inCommunication, Vol.18, pp.1122-1128, June 2000, by Y.C.Lee et al. in IEEE Trans.Image Process., vol.11, n o.11, pp.1314-1331, "Multiframe error concealment for MPEG-coded video delivery overerror-prone networks" published in Nov 2002, by G.S.Yu et al. in IEEE Trans.Circuits Syst.VideoTechnol., vol.8 , pp.422-434, "POCS-based error concealment for packet video using multiframe overlap information" published in Aug.1998, published by S.Belfiore et al. in IEEE Trans.Multimedia, vol.7, no.2, pp. 316-329, "Concealment of whole-frame losses for wireless low bit-rate videobased on multiframe optical flow estimation" published in Apr.2005, published by P.Baccichet et al. in IEEE Trans.Consumer Electronics, vol.51, no .1, pp.227-233, "Frameconcealment for H.264/AVC decoders" published in Feb.2005, the content of which will be incorporated into this article as a reference. the

Although many studies have been conducted to improve temporal error concealment methods, there is still room for improvement in the prediction accuracy of motion vectors and the effect and efficiency of compensation. the

Contents of the invention

In view of the problems in the prior art, the present invention provides a high-efficiency temporal error concealment method suitable for H.264, which can effectively improve the prediction accuracy and error concealment performance of motion vectors. the

According to an aspect of the present invention, a method for temporal error concealment applied in video decoding is provided, the method comprising the following steps: a. detecting a damaged macroblock, which is composed of four 8×8 damaged blocks; b. Obtain the motion vectors of multiple 4×4 adjacent blocks around the damaged macro block; c. For each of the 8×8 damaged blocks, the The motion vectors of the six 4*4 adjacent blocks closest to the 8*8 damaged block among the plurality of 4*4 adjacent blocks determine a predicted motion vector of the 8*8 damaged block; d. compare The predicted motion vectors of the four 8×8 damaged blocks are used to determine not to merge the four 8×8 damaged blocks, and to merge the four 8×8 damaged blocks into two 16× 8 Merge damaged blocks, merge the four 8×8 damaged blocks into two 8×16 damaged blocks, or merge all the four 8×8 damaged blocks into one 16×16 merged damaged blocks; e. For the merged 16×8 merged damaged blocks, the 8×16 merged damaged blocks, or the 16×16 merged damaged blocks, determine each of the A predicted motion vector of the merged damaged block is an average value of predicted motion vectors of the 8×8 damaged blocks included in the merged damaged block; and f. For each of the 8×8 damaged blocks, The 16×8 merged damaged block, the 8×16 merged damaged block, or the 16×16 merged damaged block is moved according to the prediction corresponding to each damaged block or merged damaged block The vector is a starting point for pixel comparison with multiple reference blocks in a search window in a reference frame, so as to find a matching damaged block or the merged damaged block in the search window corresponding to the reference block. the

Other aspects of the present invention will partly be stated in the subsequent description, and partly can be easily understood from the description, or can be known from the embodiments of the present invention. Aspects of the present invention can be understood and achieved by utilizing the elements and combinations particularly pointed out in the above claims. It should be understood that the foregoing summary of the invention and the following detailed description are for illustrative purposes only, and are not intended to limit the present invention. the

Description of drawings

The drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principle of the invention. The embodiments described herein are preferred embodiments of the present invention, however, it must be understood that the present invention is not limited to the configuration and elements shown, wherein:

Figure 1 shows the partitioning modes of different block sizes defined by the H.264 compression standard for inter-picture prediction;

Fig. 2 is the flowchart of temporal error concealment method provided by the present invention;

FIG. 3 shows a macroblock lost or damaged during transmission, and a plurality of adjacent 4×4 blocks;

Fig. 4 is the upper left 8 × 8 blocks of the macro block in Fig. 3;

FIG. 5 shows an active regression plane calculated according to six adjacent 4×4 blocks in one embodiment;

FIG. 6 shows a flowchart of a method for determining a variable block size mode provided by an embodiment of the present invention;

Fig. 7 shows the alignment sequence adopted by an embodiment of the present invention; and

FIG. 8 and FIG. 9 are schematic diagrams and flow charts illustrating the refinement procedure of an embodiment of the present invention by taking the T4 (8×8) mode as an example. the

[Description of main component labels]

300 damaged macroblocks

300A, 300B, 300C, 300D 8×8 blocks

310, 312, 314, 316 4×4 blocks

320, 322, 324, 326 4×4 blocks

330, 332, 334, 336 4×4 blocks

340, 342, 344, 346 4×4 blocks

500 active return to plane

8008×8 damaged blocks

810, 8208×8 adjacent blocks

812, 814, 816, 8184×4 blocks

822, 824, 826, 8284×4 blocks

Detailed ways

In order to make the description of the present invention more detailed and complete, reference may be made to the following description together with the diagrams of FIGS. 2 to 9 . However, the devices, components and method steps described in the following embodiments are only used to illustrate the present invention, and are not intended to limit the scope of the present invention. the

FIG. 2 is a flow chart of the temporal error concealment method provided by the present invention. First, in step S200, a video signal including a plurality of frames is received and decoded, at least one frame has missing or damaged macroblocks. In step S210, for a lost or damaged macroblock, the motion vectors of its spatially neighboring blocks are determined. Next, in step S220, the motion vector of the damaged block is predicted by using an active regression plane constructed from the motion vectors of the adjacent blocks. In step S230, the division mode of the damaged macroblock is determined according to the predicted motion vector of the damaged block. Finally, in step S240, a motion compensation method with a variable block size is used for motion refinement to search for a better compensation block in the reference frame. In addition, the present invention proposes several methods for terminating the step S240 in advance, so as to reduce the time required for the refinement procedure and improve the compensation efficiency. Steps S220 to S240 will be described in more detail below. the

Referring to FIG. 3, a macroblock (MB) 300 is a 16×16 macroblock that is lost or damaged during transmission, and blocks 310, 312, 314, 316, 320, 322, 324, 326, 330, 332, 334, 336, 340, 342, 344, and 346 are adjacent 4x4 blocks of MB 300. The present invention utilizes the high correlation between motion vectors of spatially adjacent blocks to predict the motion vector of the damaged MB 300. First, as shown in FIG. 3 , the damaged MB 300 is divided into four 8×8 blocks (upper left block 300A, lower left block 300B, lower right block 300C, and upper right block 300D). According to the correlation between each damaged 8×8 block and its adjacent six 4×4 blocks, the present invention proposes an active regression plane to predict the motion vector of each damaged 8×8 block. This plane is the second-order plane calculated according to the positions of adjacent blocks and their corresponding motion vectors. the

With reference to Fig. 4, the method for predicting the motion vector with the active regression plane of the present invention is described by taking the upper left 8×8 block 300A in Fig. 3 as an example. Since blocks that are spatially closer to each other generally have higher correlation, the present invention selects the six blocks 312, 314, 316, 320, 322, and 324 closest to block 300A to calculate the motion vector of block 300A . Set the center point coordinates of block 300A as (0, 0), then the center point coordinates of six adjacent 4×4 blocks 312, 314, 316, 320, 322, 324 are (6, 6) respectively , (2, 6), (-2, 6), (-6, 2), (-6, -2), and (-6, -6), and their corresponding motion vectors are denoted as V1, V2 respectively , V3, V4, V5, and V6. The active regression plane proposed by the present invention is:

Z _i (x, y)＝α ₁ x _i ² +α ₂ x _i y _i +α ₃ y _i ² +α ₄ x _i +α ₅ y _i +α ₆

Wherein, x and y represent the center point coordinates of six adjacent 4×4 blocks, and z is the corresponding moving vector. Put the center coordinates of six adjacent 4×4 blocks and the corresponding motion vectors V1, V2, V3, V4, V5, and V6 into the above formula, and the coefficients α ₁ , α ₂ , α ₃ , α can be calculated ₄ , α ₅ , and α ₆ . Referring to FIG. 5 , it shows an active regression plane 500 calculated from six adjacent 4×4 blocks in one embodiment, on which the motion vector of the missing 8×8 block 300A will be located. Expressed mathematically, the motion vector of the missing 8×8 block can be expressed as:

${\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="V"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">V</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}\frac{{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="2"\; label="V"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">V</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\frac{{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="3"\; label="V"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">V</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}\frac{{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}_{\mathrm{<span\; class="notranslate">\; 3</span>}}}{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="4"\; label="V"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">V</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}\frac{{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}_{\mathrm{<span\; class="notranslate">\; 4</span>}}}{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="5"\; label="V"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">V</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}\frac{{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}_{\mathrm{<span\; class="notranslate">\; 5</span>}}}{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="6"\; label="V"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">V</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}\frac{{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}_{\mathrm{<span\; class="notranslate">\; 6</span>}}}{\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}$

in,

Δ=det(M), $m=\left[\begin{array}{cccccc}{x_1}^2& x_1{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& {{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^2& x_1& {\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& 1\\ {x_2}^2& x_2{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& {{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2}^2& x_2& {\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& 1\\ {x_3}^2& x_3{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& {{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3}^2& x_3& {\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& 1\\ {x_4}^2& x_4{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& {{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4}^2& x_4& {\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& 1\\ {x_5}^2& x_5{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& {{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5}^2& x_5& {\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& 1\\ {x_6}^2& x_6{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& {{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6}^2& x_6& {\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& 1\end{array}\right]$

Δ ₁ =det(M ₁ ), ${\mathrm{<figure-callout\; id="1"\; label="m"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">m</figure-callout>}}_1=\left[\begin{array}{cccccc}{x}^2& \mathrm{xy}& {\mathrm{the\; y}}^2& x& \mathrm{the\; y}& 1\\ {x_2}^2& x_2{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& {{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2}^2& x_2& {\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& 1\\ {x_3}^2& x_3{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& {{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3}^2& x_3& {\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& 1\\ {x_4}^2& x_4{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& {{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4}^2& x_4& {\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& 1\\ {x_5}^2& x_5{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& {{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5}^2& x_5& {\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& 1\\ {x_6}^2& x_6{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& {{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6}^2& x_6& {\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& 1\end{array}\right]$

Δ ₂ =det(M ₂ ), ${\mathrm{<figure-callout\; id="2"\; label="m"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">m</figure-callout>}}_2=\left[\begin{array}{cccccc}{x_1}^2& x_1{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& {{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^2& x_1& {\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& 1\\ x^2& \mathrm{xy}& {\mathrm{the\; y}}^2& x& \mathrm{the\; y}& 1\\ {x_3}^2& x_3{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& {{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3}^2& x_3& {\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& 1\\ {x_4}^2& x_4{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& {{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4}^2& x_4& {\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& 1\\ {x_5}^2& x_5{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& {{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5}^2& x_5& {\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& 1\\ {x_6}^2& x_6{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& {{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6}^2& x_6& {\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& 1\end{array}\right]$

Δ ₃ =det(M ₃ ), ${\mathrm{<figure-callout\; id="3"\; label="m"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">m</figure-callout>}}_3=\left[\begin{array}{cccccc}{x_1}^2& x_1{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& {{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^2& x_1& {\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& 1\\ {x_2}^2& x_2{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& {{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2}^2& x_2& {\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& 1\\ x^2& \mathrm{xy}& {\mathrm{the\; y}}^2& x& \mathrm{the\; y}& 1\\ {x_4}^2& x_4{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& {{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4}^2& x_4& {\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& 1\\ {x_5}^2& x_5{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& {{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5}^2& x_5& {\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& 1\\ {x_6}^2& x_6{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& {{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6}^2& x_6& {\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& 1\end{array}\right]$

Δ ₄ =det(M ₄ ), ${\mathrm{<figure-callout\; id="4"\; label="m"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">m</figure-callout>}}_4=\left[\begin{array}{cccccc}{x_1}^2& x_1{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& {{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^2& x_1& {\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& 1\\ {x_2}^2& x_2{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& {{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2}^2& x_2& {\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& 1\\ {x_3}^2& x_3{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& {{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3}^2& x_3& {\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& 1\\ x^2& \mathrm{xy}& {\mathrm{the\; y}}^2& x& \mathrm{the\; y}& 1\\ {x_5}^2& x_5{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& {{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5}^2& x_5& {\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& 1\\ {x_6}^2& x_6{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& {{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6}^2& x_6& {\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& 1\end{array}\right]$

Δ ₅ =det(M ₅ ), ${\mathrm{<figure-callout\; id="5"\; label="m"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">m</figure-callout>}}_5=\left[\begin{array}{cccccc}{x_1}^2& x_1{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& {{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^2& x_1& {\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& 1\\ {x_2}^2& x_2{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& {{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2}^2& x_2& {\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& 1\\ {x_3}^2& x_3{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& {{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3}^2& x_3& {\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& 1\\ {x_4}^2& x_4{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& {{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4}^2& x_4& {\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& 1\\ x^2& \mathrm{xy}& {\mathrm{the\; y}}^2& x& \mathrm{the\; y}& 1\\ {x_6}^2& x_6{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& {{\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6}^2& x_6& {\mathrm{the\; <figure-callout\; id="6"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_6& 1\end{array}\right]$

Δ ₆ =det(M ₆ ), ${\mathrm{<figure-callout\; id="6"\; label="m"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">m</figure-callout>}}_6=\left[\begin{array}{cccccc}{x_1}^2& x_1{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& {{\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^2& x_1& {\mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}}_1& 1\\ {x_2}^2& x_2{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& {{\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2}^2& x_2& {\mathrm{the\; <figure-callout\; id="2"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_2& 1\\ {x_3}^2& x_3{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& {{\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3}^2& x_3& {\mathrm{the\; <figure-callout\; id="3"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& 1\\ {x_4}^2& x_4{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& {{\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4}^2& x_4& {\mathrm{the\; <figure-callout\; id="4"\; label="y"\; filenames="H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_4& 1\\ {x_5}^2& x_5{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& {{\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5}^2& x_5& {\mathrm{the\; <figure-callout\; id="5"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00071.png"\; state="\{\{state\}\}">y</figure-callout>}}_5& 1\\ x^2& \mathrm{xy}& {\mathrm{the\; y}}^2& x& \mathrm{the\; <figure-callout\; id="1"\; label="y"\; filenames="H2009101498028E00071.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout>}& 1\end{array}\right]$

After calculating the motion vectors of the four 8×8 blocks 300A, 300B, 300C, and 300D in FIG. vector, and it is decided to use T1 mode (16×16), T2 mode (16×8), T3 mode (8×16), or T4 (8×8) mode for subsequent motion vector refinement procedure. FIG. 6 shows a flowchart of a method for determining a variable block size mode provided by an embodiment of the present invention. First, in step S600 , taking the damaged macroblock 300 in FIG. 3 as an example, the predicted motion vectors of the four 8×8 blocks 300A, 300B, 300C, and 300D in the damaged macroblock 300 are obtained. Next, in step S610, compare the motion vectors of two adjacent 8×8 blocks in the horizontal direction, and judge whether the following conditions are met:

|MV1-MV2|≤TH _H1 and |MV3-MV4|≤TH _H2 ;

Wherein MV1, MV2, MV3 and MV4 are respectively the upper left 8×8 block, the upper right 8×8 block, the lower left 8×8 block, and the lower right 8×8 block (respectively block 300A, block 300A, 300D, 300B and 300C), TH _H1 and TH _H2 are the threshold difference of horizontal motion vectors, which can be adjusted according to different practical applications. In one embodiment, both TH _H1 and TH _H2 can be set to 1. If the above conditions are satisfied, the horizontal merging can be performed and the process proceeds to step S620; otherwise, if the above conditions are not met, the horizontal merging is not performed and the process proceeds to step S630. In steps S620 and S630, compare the motion vectors of two vertically adjacent 8×8 blocks, and determine whether the following conditions are met:

|MV1-MV3|≤TH _V1 and |MV2-MV4|≤TH _V2 ;

Wherein TH _V1 and TH _V2 are the difference thresholds of the motion vectors in the vertical direction, and both may be set to 1 in an embodiment. If it is judged yes in step S620, then the program proceeds to step S622, adopts T1 mode, and combines four 8×8 blocks to form a 16×16 macro block, otherwise, then proceeds to step S624, adopts T2 mode , only merge in the horizontal direction to form two 16×8 sub-macroblocks. In addition, in step S630, if the judgment is yes, then the program proceeds to step S632, adopts the T3 mode, and performs vertical merging to form two 8×16 sub-macroblocks; otherwise, the program proceeds to step S634, does not For any merging, take the T4 mode with four 8x8 sub-macroblocks.

After determining the division mode (T1, T2, T3, or T4 mode), perform a motion vector refinement process for each sub-macroblock after division to search for a better compensation block in the reference picture . The motion vector refinement procedure is similar to the motion estimation (motion estimation) procedure at the encoding end, with the motion vector predicted by each sub-macroblock as the starting point (if the sub-macroblock contains more than two 8×8 blocks, then its The corresponding motion vector is the average value of the predicted motion vectors of each 8×8 block), compared with the error value of its peripheral boundary pixels (such as the sum of absolute errors (SAD)), in order to find a better compensation block. Fig. 7 shows the alignment sequence adopted by an embodiment of the present invention, which starts from the predicted motion vector (i.e., point 0 in Fig. 7), and adopts an outward spiral search sequence in a search window of the reference frame Sequentially compare with each candidate block in the search window. the

In the comparison process, the present invention proposes a partial distortion comparison method and two early termination methods to save the time required for motion vector refinement. The first early termination method is to set a critical value according to the original residual value (residual) of the corresponding boundary block during encoding: DT _a =μ×N+γ, where N represents the total The number of pixels, μ represents the average residual value of each corresponding pixel, and γ represents a constant coefficient. The second method is to set another critical value according to the comparison results of other compensated damaged blocks in the current frame: DT _b =EBME(u,v)×EBME _β /EBME _α ×λ+ε, where (u, v) and EBME(u, v) represent the motion vector corresponding to the sub-macroblock to be compared at present and the peripheral border comparison error value obtained from the first comparison, EBME _α and EBME _β respectively Represents the outer border comparison error value and the smallest outer border comparison error value obtained from the first comparison of other damaged blocks that have been compared before in the current picture, and λ is a constant proportional coefficient (for example, it can be set to 0.6 ), ε is a constant coefficient (for example, it can be set to 0). In the process of comparing the boundary blocks, when the comparison error value is less than or equal to the critical value DT _a or DT _b , it means that a matching block has been found, and the search for the damaged sub-macroblock can be stopped , so as to reduce the computation load of peripheral boundary comparison. In addition, the partial distortion comparison method proposed by the present invention uses 16 pixels (4×4) as a unit, and performs comparison by accumulating one pixel at a time for each unit, so as to reduce the number of motion vector refinement procedures. The number of pixels to compare for error.

FIG. 8 and FIG. 9 are schematic diagrams and flow charts illustrating the refinement procedure of an embodiment of the present invention by taking the T4 (8×8) mode as an example. The block 800 is the upper left 8×8 block of a damaged macroblock, and the pixel error values of the upper and left peripheral neighboring blocks 810 and 820 will be compared in the refinement process. In general, the SAD value of block 800 can be expressed as:

$\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 8</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 8</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; Top</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="0"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="0"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; +</span>$

$\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 8</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 8</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; Left</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="0"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="0"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>$

Among them, A _Top and A _Left represent the pixels in the current picture, R(x, y) represents the pixel corresponding to the reference picture, (x ₀ , y ₀ ) represents the coordinates of the upper left pixel of the corresponding block, (u, v) represents the motion vector corresponding to block 800 .

Referring to FIG. 8, the partial distortion comparison method of the present invention takes a 4×4 block as a unit, and divides the peripheral adjacent blocks 810 and 820 into four 4×4 blocks 812, 814, 816, 818, 822, and 824, respectively. , 826, 828, each 4×4 block contains 16 pixels. The present invention divides the total distortion value D of blocks 810 and 820 into 16 partial distortions (d _p , p=1-16), wherein each partial distortion (d _p ) includes 8 pixels, and these 8 pixels are respectively located at Same location in 4x4 blocks 812 , 814 , 816 , 818 , 822 , 824 , 826 , 828 . For example, the partial distortion d ₁ includes 8 pixels indicated by oblique lines in FIG. 8 . In one embodiment, the order of d ₁ to d ₁₆ in each 4×4 block is shown in FIG. 8 , and each partial distortion d _p can be expressed as follows:

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\begin{array}{c}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; Top</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="0"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; -</span>\\ \mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="0"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>\end{array}<span\; class="notranslate">\; +</span>$

$\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\begin{array}{c}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; Left</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="0"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; -</span>\\ \mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="0"\; label="y"\; filenames="H2009101498028E00051.png,H2009101498028E00091.png"\; state="\{\{state\}\}">y</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>\end{array}$

The (s _p , t _p ) value corresponding to each p value is listed in the following table:

p (s _p , t _p ) 1 (0,0) 2 (2, 2) 3 (2,0) 4 (0, 2) 5 (1,1) 6 (3,3) 7 (3,1) 8 (1, 3)

[0063] 9 (1,0) 10 (3, 2) 11 (0,1) 12 (2, 3) 13 (3,0) 14 (1, 2) 15 (2,1) 16 (0, 3)

In addition, the p-th accumulated distortion value D _p accumulated to the p-th partial distortion is defined as:

${\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}$

The following will describe the motion vector refinement process for a damaged block in an embodiment of the present invention according to the above definition. Referring to FIG. 9, in step S900, it is judged whether the first peripheral boundary comparison is performed for this damaged block, and if so, proceed to step S910, and calculate all partial distortions d ₁ -d ₁₆ and the total distortion value D ₁₆ =d ₁ +d ₂ +...+d ₁₆ . Next, in steps S912 and S914, it is judged whether D ₁₆ is less than or equal to DT _a and DT _b (that is, whether the condition for early termination is satisfied), and if so, proceed to step S950 to end the search procedure for the damaged block. If both steps S912 and S914 are judged to be negative, the procedure proceeds to step S916, and the total distortion value D ₁₆ of the initial comparison is set as D _min as a criterion for comparing the next candidate block. Next, in steps S920 to S925, compare the outer boundary of the next candidate block, starting from p=1, calculate d _p (8 points) and D _p respectively, if D _p >p×D _mim /16 , the comparison of the candidate block can be ended, and proceed to step S940. If D ₁ <D _mim /16, then repeat the above steps to calculate d ₂ and D ₂ , and compare D2 with 2×D _mim /16. The sequence of comparison is from p=1 to p=16, and once D _p is greater than p×D _mim /16 during the comparison process, the comparison of the candidate block is terminated. When comparing to p=16, if D ₁₆ is less than D _mim , then proceed to steps S930 and S932, respectively judge whether D ₁₆ is less than or equal to DT _a and DT _b (that is, whether the condition for early termination is satisfied), and if so, proceed to Step S950, ending the search procedure for the damaged block. If both steps S930 and S932 are judged to be negative, then the program proceeds to step S934, resetting the total distortion value _D16 obtained by this comparison as a new _Dmin value, as a judgment for comparing the next candidate block benchmark. Next, in step S940, it is judged whether the comparison of the last candidate block in the search window has been completed, if so, return to step S920 and continue the peripheral boundary comparison of the next candidate block, otherwise proceed to step S950, and end this process Refined comparison program for damaged blocks.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the scope of the claims of the present invention; all other equivalent changes or modifications that do not deviate from the spirit disclosed in the present invention should be included in the above-mentioned within the scope of the claims. the

Claims (8)

1. A method for temporal error concealment in video decoding, the method comprising:

a. detecting a damaged macroblock, the damaged macroblock consisting of 4 8 × 8 damaged blocks;

b. obtaining motion vectors of a plurality of 4 x 4 neighboring blocks surrounding the damaged macroblock;

c. for each 8 × 8 damaged block, determining a predicted motion vector of the 8 × 8 damaged block from motion vectors of 64 × 4 neighboring blocks of the 4 × 4 neighboring blocks that are closest to the 8 × 8 damaged block;

d. comparing the predicted motion vectors of the 4 8 × 8 damaged blocks to determine whether to merge the 4 8 × 8 damaged blocks, merge the 4 8 × 8 damaged blocks two by two into 2 16 × 8 merged damaged blocks, merge the 4 8 × 8 damaged blocks two by two into 28 × 16 merged damaged blocks, or merge all the 4 8 × 8 damaged blocks into 1 16 × 16 merged damaged block;

e. determining a predicted motion vector of each of the merged damaged blocks as an average of the predicted motion vectors of the 8 × 8 damaged blocks included in the merged damaged block for the merged 16 × 8 merged damaged block, the 8 × 16 merged damaged block, or the 16 × 16 merged damaged block; and

f. for each of the 8 × 8 damaged block, the 16 × 8 merged damaged block, the 8 × 16 merged damaged block, or the 16 × 16 merged damaged block, the predicted motion vector corresponding to each damaged block or merged damaged block is used as a starting point to perform pixel matching with a plurality of reference blocks in a search window of a reference picture, so as to find a corresponding reference block matching with the damaged block or the merged damaged block in the search window.

2. The method according to claim 1, wherein step c further comprises:

providing an active regression plane as: z_i(x，y)＝α₁x_i ²+α₂x_iy_i+α₃y_i ²+α₄x_i+α₅y_i+α₆；

Respectively substituting the coordinates of the center points of the 64 × 4 adjacent blocks and the corresponding motion vectors into x of the active regression plane_i、y_iAnd Z_iTo find the coefficient alpha₁、α₂、α₃、α₄、α₅And alpha₆(ii) a And

determining that the predicted motion vector is located on the active regression plane.

3. The method of claim 1, wherein the pixel matching is to calculate an error value between a peripheral boundary pixel of each of the damaged blocks or merged damaged blocks and a peripheral boundary pixel of the reference block in the search window.

4. The method of claim 3, wherein the error value is an absolute error sum.

5. The method according to claim 3, wherein step f is performed by performing pixel matching in a spiral path.

6. The method of claim 5, wherein during the pixel matching process, the pixel matching is stopped when an error value of a specific reference block is smaller than a threshold value, and the specific reference block is used as the corresponding reference block, wherein the threshold value is DT_aμ × N + γ, where N represents the total number of pixels in the peripheral boundary pixel, μ represents the average residual value of the peripheral boundary pixel, and γ represents a constant coefficient.

7. The method of claim 5, wherein during the pixel matching process, the pixel matching is stopped when an error value of a specific reference block is smaller than a threshold value, and the specific reference block is used as the corresponding reference block, wherein the threshold value is DT_b＝EBME(u，v)×EBME_β/EBME_αX lambda + epsilon, where (u, v) and EBME (u, v) represent the motion vector corresponding to the damaged block or the merged damaged block to be compared currently and the error value obtained by the first comparison, respectively, EBME_αAnd EBME_βRespectively representing the error value and the minimum error value obtained by the first comparison of other damaged blocks which are compared, wherein lambda is a constant proportionality coefficient, and epsilon is a constant coefficient.

8. The method of claim 1, wherein the pixel matching for each 8 x 8 damaged block or each merged damaged block further comprises:

calculating an error value D between the peripheral boundary pixel of a first reference block corresponding to the starting point and the peripheral boundary pixel of the damaged block or the merged damaged block_mim(ii) a And

comparing the pixels of the second reference block according to a spiral path, dividing the peripheral boundary pixels of the damaged block or the merged damaged block into 16 equal parts, defining the error value of the 1 st equal part of the 16 equal parts to the error value of the p th equal part as a p equal part accumulated error value, wherein p is an integer from 1 to 16, calculating the p equal part accumulated error value in sequence and judging whether the p equal part accumulated error value is greater than p multiplied by D_mimAnd/16, if yes, finishing the pixel comparison of the second reference block.

Images