CN104243991B - A kind of side information generation method and device - Google Patents

Abstract

The invention belongs to the technical field of image processing, and discloses a method and device for generating side information, so as to improve the accuracy of generated side information. A method for generating side information, comprising: acquiring key frames adjacent to each other in time domain in a video image output by an encoding end, wherein the key frames are acquired based on human eye features in the image, and in the two adjacent key frames There is an interpolation frame in the middle of the key frame; the forward motion vectors of two adjacent key frames in the key frame are obtained; for any interpolation frame, according to the information of the interpolation frame and the forward motion of the relevant key frame of the interpolation frame Vector motion compensation is performed to generate side information.

Description

A method and device for generating side information

technical field

The invention relates to image processing technology, in particular to a method and device for generating side information.

Background technique

The distributed video coding framework is a new asymmetric video compression framework. The system independently encodes each video frame at the encoding end, that is, intra-frame coding, and jointly decodes each video frame at the decoding end, that is, inter-frame decoding. .

In a distributed video coding system, information obtained through lossy source coding (Wyner-Ziv coding) is called main information, and side information (Side information) is an estimate of the main information generated at the decoding end. The side information helps the decoder in the decoding process of the main information, and assists the decoder in decoding.

The most effective way for the system to obtain excellent rate-distortion performance and compression efficiency is to make the generated side information as close as possible to the real value of the current frame to be decoded. Therefore, the generation of side information is a research focus of distributed video coding. However, in the process of implementing the present invention, the inventors found that the side information obtained by the general side information generation method will cause deviations in the motion vectors of many blocks in the obtained side information frame. Therefore, the accuracy of side information is not high.

Contents of the invention

The technical problem to be solved by the present invention is to provide a method and device for generating side information to improve the accuracy of generated side information.

A method for generating side information, comprising:

Acquiring each key frame adjacent in the time domain in the video image output by the encoding end, wherein the key frame is obtained based on human eye characteristics in the image, and there is an interpolation frame between the two adjacent key frames;

Obtain the forward motion vectors of two adjacent key frames in the key frame;

For any interpolation frame, perform motion compensation according to the information of the interpolation frame and the forward motion vector of the related key frame of the interpolation frame to generate side information.

Wherein, said acquiring the forward motion vectors of two adjacent key frames in said key frame includes:

Respectively divide two adjacent key frames into blocks of predetermined size;

Calculating the sum of absolute differences between corresponding blocks in the two adjacent key frames;

If the absolute difference sum between the corresponding blocks is less than the first preset value, use the pixel block in the forward key frame in the two adjacent key frames as a candidate pixel block to replace the two corresponding key frames the corresponding pixel block of the interpolated frame in the middle of the frame;

If the absolute difference sum between the corresponding blocks is greater than a second preset value, perform motion compensation on the corresponding blocks;

If the sum of absolute differences between corresponding blocks is greater than the first preset value and smaller than a second preset value, the corresponding block is kept.

Wherein, performing motion compensation according to the information of the interpolation frame and the forward motion vector of the relevant key frame of the interpolation frame, generating side information includes:

Decomposing the motion trajectory of the object in the horizontal and vertical directions, using the forward motion vector between the first two key frames of the interpolation frame and the forward motion vector between the last two key frames of the interpolation frame, using the second The function simulates the trajectory of the object;

According to the simulation result, calculate the distance of the matching block of the interpolation frame relative to the front key frame and the back key frame of the interpolation frame, and adjust the sum of the front key frame and the back key frame according to the distance to interpolate The weight of the frame generates side information.

Wherein, after the key frames adjacent to each other in the time domain in the video image output by the encoding end are acquired, the method further includes:

Low-pass filtering is performed on each of the key frames.

Wherein, the method also includes:

Image quality evaluation is performed on the generated side information.

A device for generating side information, comprising:

A frame acquisition unit, configured to acquire key frames adjacent in time domain in the video image output by the encoding end, wherein the key frames are acquired based on human eye characteristics in the image, between the two adjacent key frames has an interpolated frame;

a vector acquisition unit, configured to acquire the forward motion vectors of two adjacent key frames in the key frame;

The generation unit is configured to, for any interpolation frame, perform motion compensation according to the information of the interpolation frame and the forward motion vector of the relevant key frame of the interpolation frame, and generate side information.

Wherein, the vector acquisition unit includes:

A block division module is used to divide adjacent two key frames into blocks of predetermined size respectively;

A calculation module, configured to calculate the sum of absolute differences between corresponding blocks in the two adjacent key frames;

A processing module, if the sum of the absolute differences between the corresponding blocks is less than a first preset value, use the pixel block in the forward key frame in the two adjacent key frames as a candidate pixel block to replace the The corresponding pixel block of the interpolation frame between the two key frames; if the absolute difference sum between the corresponding blocks is greater than the second preset value, motion compensation is performed on the corresponding block; if the absolute difference sum between the corresponding blocks is greater than The first preset value is smaller than the second preset value, and the corresponding block is kept.

Wherein, the generating unit includes:

The simulation module is used to decompose the motion trajectory of the object in the horizontal and vertical directions, using the forward motion vector between the first two key frames of the interpolation frame and the forward motion between the last two key frames of the interpolation frame Vector, use the quadratic function to simulate the trajectory of the object;

A generating module, configured to calculate the distance of the matching block of the interpolation frame relative to the front key frame and the back key frame of the interpolation frame according to the simulation result, and adjust the front key frame and the back key frame according to the distance Frames are summed to interpolate frame weights to generate side information.

Wherein, the frame obtaining unit is further configured to perform low-pass filtering on the video image to obtain the key frames.

Wherein, the device also includes:

The quality evaluation unit performs image quality evaluation on the generated side information.

The beneficial technical achievement that the present invention obtains is:

The present invention first obtains the key frames adjacent in the time domain in the video image output by the encoding end, wherein the key frames are obtained based on the characteristics of human eyes in the image, and there is an interpolation frame between the two adjacent key frames , and then obtain the forward motion vectors of two adjacent key frames in the key frame, and for any interpolation frame, perform motion compensation according to the information of the interpolation frame and the forward motion vectors of the relevant key frames of the interpolation frame , and then generate side information. Since the present invention adopts key frames acquired based on human eye features, performs motion compensation according to the information of the interpolation frame and the forward motion vector of the relevant key frame of the interpolation frame, and then generates side information, therefore, using the present invention The accuracy of side information obtained by the method and device is relatively high.

Description of drawings

Fig. 1 is the flowchart of side information generation method of the present invention;

Fig. 2 is motion vector selection figure in this method;

Fig. 3 is a schematic diagram of spatial smoothing filtering in the method;

Fig. 4 is a schematic diagram of overlapping block motion compensation in the present method;

Fig. 5 is a schematic diagram comparing the algorithm of the present invention and the traditional algorithm in bidirectional motion estimation;

Fig. 6 is a structural similarity method (SSIM) method block diagram in this method;

Fig. 7 is a schematic diagram of a side information generating device of the present invention;

Fig. 8 is a structural diagram of a side information generating device of the present invention.

Detailed ways

In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will describe in detail with reference to the drawings and specific embodiments.

In distributed video coding, a video sequence X consists of odd frames X _2k±1 (k is an integer) and even frames X _2k (k is an integer). In the present invention, it is assumed that odd-numbered frames are key frames, and even-numbered frames are WZ frames (interpolation frames). It can also be assumed that even frames are key frames, and odd frames are WZ frames. It is only necessary to ensure that there are key frames before and after the WZ frame. Then the side information is mainly generated by performing motion compensation interpolation on adjacent key frames X _2k±1 in the temporal domain.

As shown in Figure 1, the side information generating method of the present invention includes:

Step 11. Obtain key frames adjacent in time domain in the video image output by the encoding end, wherein the key frames are acquired based on human eye characteristics in the image, and there is an interpolation frame between the two adjacent key frames .

In this step, after obtaining key frames adjacent in time domain in the video image output by the encoding end, low-pass filtering may also be performed on the key frames.

Firstly, low-pass filtering is performed on the input key frame in combination with the visual characteristic HVS (Human Visual System, Human Visual System) of the human eye. Human vision has unique brightness characteristics, spatial frequency characteristics and time frequency characteristics. According to the spatial frequency characteristics of vision, components with higher frequencies in the image represent smaller details of the image, and the lower the detection ability of the human eye. The spatial frequency components of ~6CPD are most sensitive). Among them, low-pass filtering can smooth the image, remove image noise, and enhance the reliability of the motion vector, so that the obtained motion vector is closer to the real vector field. The basic implementation process is as follows: First, the original image and the target image spectrum after two-dimensional discrete Fourier transform (2-DDFT) are divided into subbands according to HVS, and two-dimensional Fourier inverse transform is performed on the obtained low-frequency subband spectrum (2-DIDFT), to build an image of the low frequency subband.

Step 12. Obtain forward motion vectors of two adjacent key frames in the key frame.

In this step, forward motion estimation is performed on the image of the low-frequency sub-band established in step 11. In the embodiment of the present invention, a block matching algorithm is mainly used to perform motion estimation between preceding and subsequent key frames.

As shown in Figure 2, the search window is set to be a rectangle of 8×8, the search range is a rectangle of 12×12, and the step size is 1. The search window is the basic unit for performing motion estimation, and the search range refers to the size of the area searched in the reference frame X2k-1 (k is an integer), within which the search and X2k+1 (k is an integer) frame The closest block to the current block, the step size is the distance between the two matching blocks before and after the search in the key frame. Among them, the step size can reduce the computational complexity of the framework and provide a rough estimate of the true vector field.

There are many criteria for the block matching method, and the present invention uses SAD (Sum of absolute differences, sum of absolute differences). The basic principle is: for an m×n block, use the following formula (1) as the motion vector distortion function

In the formula, X _2k-1 (i, j) is the pixel value of the macroblock in the previous reference frame, X _2k+1 (i+dx, j+dy) is the pixel value of the corresponding macroblock in the next reference frame, (dx, dy) is the motion vector between two macroblocks. The block corresponding to the minimum value of SAD is taken as the best matching block. The obtained motion vectors are used as candidate motion vectors for each non-overlapping block in the WZ frame. Wherein, each candidate motion vector intersects with the WZ frame at a point, and the motion vector whose intersection point is closest to the block center of the desired motion vector is selected as the motion vector of the best matching block.

In the present invention, an optimization is made to the block matching algorithm. Firstly, two adjacent key frames before and after are divided into small blocks of predetermined size, such as 8×8, and the summed absolute error value (SAD) of the corresponding block is calculated. At the same time, the first preset value T1 and the second preset value T2 are set according to actual experience. The SAD value of each group was analyzed and judged. If the SAD is less than the threshold T1, the degree of motion change is small and has little impact on motion interpolation estimation. In order to simplify the decoding process and ignore its change, the pixels in the forward key frame in the two adjacent key frames are As a candidate pixel block, the pixel block in the WZ frame is directly replaced by the corresponding pixel block in the forward key frame. If SAD>T2, it means that the motion changes between the pixel blocks are severe, and the size of each pixel block is divided into 4×4 sub-blocks, and then its motion compensation is performed. If the SAD value is between T1 and T2, the size of the pixel block remains unchanged during motion compensation. In this step, the size of the search block is adjusted according to the size of the SAD value of the corresponding block during the block matching process, which can significantly reduce the ghosting phenomenon.

Through the above method, the forward motion vectors of two adjacent key frames are obtained.

Step 13. For any interpolation frame, perform motion compensation according to the information of the interpolation frame and the forward motion vector of the relevant key frame of the interpolation frame, and generate side information.

In this step, two-way motion estimation is mainly performed, including: decomposing the motion trajectory of the object in the horizontal and vertical directions, using the forward motion vector between the first two key frames of the interpolation frame and the For the forward motion vector between the last two key frames, a quadratic function is used to simulate the motion trajectory of the object. According to the simulation result, calculate the distance of the matching block of the interpolation frame relative to the front key frame and the back key frame of the interpolation frame, and adjust the sum of the front key frame and the back key frame according to the distance to interpolate The weight of the frame generates side information.

The above process is described in detail below.

After obtaining the final motion vector field, the interpolated frame can be obtained by bi-directional motion compensation in the standard video coding framework. The present invention decomposes the motion trajectory of the object in the horizontal and vertical directions, and uses the quadratic function as the correction criterion of the block motion according to the two motion vectors between the previous two frames and the next key frame of the interpolation frame, so as to match the difference of the object. Regular movement.

Suppose the forward motion vector of the two frames before the interpolation frame is (x ₁ , y ₁ ), the forward motion vector between the two key frames before and after the interpolation frame is (x ₂ , y2), (x is the motion in the horizontal direction, y is the vertical motion movement in the vertical direction), and the time between key frames is t, then after calculation, the accelerations in the horizontal and vertical directions of the object during this period are a _x , a _y respectively:

The horizontal and vertical displacements of the interpolated frame relative to the previous keyframe are (s _x , s _y ):

The invention utilizes the information of three frames before and after the interpolation frame, and adopts a quadratic function to simulate the motion track of the object, which can effectively match the acceleration and curve motion of the object. In addition, the weighted compensation interpolation technology is adopted, and the weight of the front and rear key frames is calculated by using the quadratic function motion trajectory, which saves the correlation calculation at the encoding end, which is simple and effective.

In addition, in this step, after obtaining the bidirectional motion vector of the frame to be decoded, if the direction of the motion vector of a certain block is inconsistent with the motion direction of the surrounding blocks, the frame after motion compensation will have a strong block effect . At this time, the motion vector can be adjusted by spatial smoothing technology. The neighborhood averaging method is one of the spatial domain smoothing techniques. In this method, a neighborhood is taken for all pixels in the estimated video frame one by one, and then the values of all pixels in the neighborhood are averaged, and the obtained value is used to replace original pixel value.

Among them, S represents the size of the neighborhood; (m, n) is the pixel points in the neighborhood; M is the number of all pixels in the neighborhood. Here, different weights can also be set for different pixels, as shown in Figure 3, the weight of the pixel B5 to be adjusted can be set to 70%, and the weights of other pixel values in the neighborhood can be summed and averaged as 030%, which can make the pixel value after the mean more accurate and in line with the real situation.

An overlapping block motion compensation technique is used to mitigate the effects of block artifacts. As shown in Figure 4, assuming that the 8×8 macroblock to be interpolated is B, before performing overlapping block motion compensation, first expand it from 8×8 to 12×12, and the adjacent macroblocks will have overlapping areas, Let its motion vector be MV ₄ =(i ₄ ,j ₄ ), and the motion vectors of the other three 8×8 blocks are MV ₁ =(i ₁ ,j ₁ ), MV ₂ =(i ₂ ,j ₂ ), MV ₃ =(i ₃ , j ₃ ). For the different areas in the macroblock to be interpolated, the parameters of reconstruction should be determined according to the number of overlapping blocks.

As shown in Figure 5, for the pixel values in area 1 that overlaps with all 4 blocks, the calculation formula is:

For pixel values in area 2 that overlaps with 2 blocks, the calculation formula is:

For the pixel values in area 3 that do not overlap with other blocks, the calculation formula is:

Since the present invention adopts key frames acquired based on human eye features, performs motion compensation according to the information of the interpolation frame and the forward motion vector of the relevant key frame of the interpolation frame, and then generates side information, therefore, using the present invention The accuracy of the side information obtained by the method is high.

In addition, on the basis of the method shown in FIG. 1 , the method may further include: performing image quality evaluation on the generated side information. Among them, the traditional method is to use PSNR (Peak Signal to Noise Ratio, Peak Signal to Noise Ratio) for image quality evaluation. Although this method has the advantages of simplicity and easy implementation, it does not consider human visual and psychological feelings. Disjointed subjective evaluation. The present invention uses the improved structure similarity algorithm to evaluate the quality of the generated WZ frame, and the process is shown in FIG. 6 .

Finally, these three parts combine to form the overall structural similarity:

The closer the value of SSIM is to 1, the higher the quality of side information generation.

As shown in Figure 7, the side information generating device of the present invention includes:

The frame acquisition unit 51 is configured to acquire each key frame adjacent in the time domain in the video image output by the encoding end, wherein the key frame is acquired based on human eye characteristics in the image, and the two adjacent key frames There is an interpolation frame in the middle; a vector acquisition unit 52 is used to obtain the forward motion vectors of two adjacent key frames in the key frame; a generation unit 53 is used for any interpolation frame, according to the information of the interpolation frame and the Motion compensation is performed on the forward motion vectors of the relevant key frames of the interpolated frame to generate side information.

Wherein, the frame acquisition unit 51 is specifically configured to perform low-pass filtering on the video image to acquire the key frames.

Wherein, the vector acquisition unit 52 includes:

The block division module is used to divide the adjacent two key frames into blocks of predetermined size; the calculation module is used to calculate the absolute difference between the corresponding blocks in the two adjacent key frames; the processing module, if the corresponding The absolute difference sum between the blocks is less than the first preset value, and the pixel block in the forward key frame in the two adjacent key frames is used as a candidate pixel block to replace the interpolation between the two adjacent key frames Corresponding pixel blocks of the frame; if the sum of absolute differences between the corresponding blocks is greater than a second preset value, perform motion compensation on the corresponding blocks; if the sum of absolute differences between corresponding blocks is greater than the first preset value and less than a second preset value, keep the corresponding block.

Wherein, the generating unit 53 includes:

The simulation module is used to decompose the motion trajectory of the object in the horizontal and vertical directions, using the forward motion vector between the first two key frames of the interpolation frame and the forward motion between the last two key frames of the interpolation frame Vector, using a quadratic function to simulate the motion track of the object; a generation module, used to calculate the distance of the matching block of the interpolation frame relative to the front key frame and the back key frame of the interpolation frame according to the simulation result, and according to The distance adjusts the weights of the front key frame and the back key frame in the sum interpolation frame to generate side information.

As shown in Figure 8, the device may also include:

The quality evaluation unit 54 evaluates the image quality of the generated side information.

For the working principle of the device of the present invention, reference may be made to the description of the foregoing method embodiments.

Since the present invention adopts key frames acquired based on human eye features, performs motion compensation according to the information of the interpolation frame and the forward motion vector of the relevant key frame of the interpolation frame, and then generates side information, therefore, using the present invention The accuracy of the side information obtained by the device is relatively high.

The above description is a preferred embodiment of the present invention, it should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications can also be made. It should be regarded as the protection scope of the present invention.

Claims (4)

1. A method for generating side information, characterized in that, comprising: Obtain each key frame adjacent to the time domain in the video image output by the encoding end, wherein the key frame is obtained based on the characteristics of the human eye in the image, and low-pass the input key frame in combination with the visual characteristic HVS of the human eye Filtering processing, the low-pass filtering processing implementation process includes: for the original image and the target image spectrum through the two-dimensional discrete Fourier transform (2-DDFT), sub-band segmentation is respectively performed according to HVS, and the obtained low-frequency sub-band spectrum Carry out two-dimensional Fourier inverse transform (2-DIDFT), set up the image of low-frequency subband; There is an interpolation frame in the middle of described two adjacent key frames; Obtain the forward motion vectors of two adjacent key frames in the key frame; For any interpolation frame, perform motion compensation according to the information of the interpolation frame and the forward motion vector of the relevant key frame of the interpolation frame, and generate side information; The obtaining the forward motion vectors of two adjacent key frames in the key frame includes: Respectively divide two adjacent key frames into blocks of predetermined size; Calculating the sum of absolute differences between corresponding blocks in the two adjacent key frames; If the absolute difference sum between the corresponding blocks is less than the first preset value, use the pixel block in the forward key frame in the two adjacent key frames as a candidate pixel block to replace the two corresponding key frames the corresponding pixel block of the interpolated frame in the middle of the frame; If the absolute difference sum between the corresponding blocks is greater than a second preset value, divide each pixel block into sub-blocks, and then perform motion compensation on the corresponding blocks; if the sum of absolute differences between corresponding blocks is greater than the first preset value and smaller than a second preset value, maintaining the corresponding block; In this process, the size of the search block is adjusted according to the size of the SAD value of the corresponding block during the block matching process; For any interpolation frame, performing motion compensation according to the information of the interpolation frame and the forward motion vector of the relevant key frame of the interpolation frame, generating side information includes: The motion trajectory of the object is decomposed in the horizontal and vertical directions, using the forward motion vector between the first two key frames of the interpolation frame and the forward motion vector between the two key frames before and after the interpolation frame, using two The secondary function simulates the trajectory of the object; According to the simulation result, calculate the distance of the matching block of the interpolation frame relative to the front key frame and the back key frame of the interpolation frame, and adjust the sum of the front key frame and the back key frame according to the distance to interpolate The weight of the frame generates side information; Image quality evaluation is performed on the generated side information, and the pixel deviation between the decoded image and the original image is calculated by the following formula: in, In the formula, fk(x, y) and fk-1(x, y) are the gray values of the pixels in the original frame and the decoded frame respectively, and M×N is the size of the image. 2. The method according to claim 1, characterized in that, after obtaining each key frame in the video image output by the encoding end in the temporal domain, the method further comprises: Low-pass filtering is performed on each key frame video image. 3. A device for generating side information, comprising: The frame acquisition unit is used to acquire each key frame adjacent in the time domain in the video image output by the encoding end, wherein the key frame is acquired based on the characteristics of the human eye in the image, combined with the visual characteristic HVS of the human eye, for Input key frame and carry out low-pass filter processing, described low-pass filter processing implementation process comprises: to the original image and target image frequency spectrum through two-dimensional discrete Fourier transform (2-DDFT), according to HVS respectively give sub-band segmentation, and to The obtained low-frequency sub-band spectrum is carried out two-dimensional inverse Fourier transform (2-DIDFT), and an image of the low-frequency sub-band is established; there is an interpolation frame between the two adjacent key frames; a vector acquisition unit, configured to acquire the forward motion vectors of two adjacent key frames in the key frame; A generating unit, configured to, for any interpolation frame, perform motion compensation according to the information of the interpolation frame and the forward motion vector of the relevant key frame of the interpolation frame, and generate side information; The vector acquisition unit includes: A block division module is used to divide adjacent two key frames into blocks of predetermined size respectively; A calculation module, configured to calculate the sum of absolute differences between corresponding blocks in the two adjacent key frames; A processing module, if the sum of the absolute differences between the corresponding blocks is less than a first preset value, use the pixel block in the forward key frame in the two adjacent key frames as a candidate pixel block to replace the The corresponding pixel block of the interpolation frame between the two key frames; if the absolute difference sum between the corresponding blocks is greater than a second preset value, each pixel block is divided into sub-blocks, and then motion compensation is performed on the corresponding block; If the sum of absolute differences between the corresponding blocks is greater than the first preset value and less than the second preset value, keep the corresponding block; in this process, adjust according to the size of the SAD value of the corresponding block in the block matching process the size of the search block; The generating unit includes: The simulation module is used to decompose the motion trajectory of the object in the horizontal and vertical directions, using the forward motion vector between the first two key frames of the interpolation frame and the forward motion vector between the two key frames before and after the interpolation frame. Motion vector, using a quadratic function to simulate the trajectory of the object; A generating module, configured to calculate the distance of the matching block of the interpolation frame relative to the front key frame and the back key frame of the interpolation frame according to the simulation result, and adjust the front key frame and the back key frame according to the distance The weights of the frame and interpolation frame are generated to generate side information; The quality evaluation unit performs image quality evaluation on the generated side information, and the quality evaluation unit calculates the pixel deviation between the decoded image and the original image by the following formula: in, In the formula, fk(x, y) and fk-1(x, y) are the gray values of the pixels in the original frame and the decoded frame respectively, and M×N is the size of the image. 4. The device according to claim 3, wherein the frame acquisition unit is further configured to perform low-pass filtering on the video image to acquire the key frames.