CN114125448B - Video coding method, decoding method and related devices - Google Patents

Abstract

An embodiment of the present application provides a video encoding method, a decoding method, and a related device, the method including: determining the first image in the target format according to the image in the YUV4:4:4 format; the fourth part in the image in the YUV4:4:4 format The pixel value, the fifth part of the pixel value, and the sixth part of the pixel value are used to convert to the Y component, U component, and V component in the first image respectively; the three parts of the pixel value respectively occupy the pixels of the image in the YUV4:4:4 format 2/3, 1/6 and 1/6 of the total number of values; encode the first image to obtain the first code stream; correspondingly, the video decoding device parses the first code stream including the target format data to obtain the first image; An image in YUV4:4:4 format is determined according to the first image. This method is not only compatible with the codec/decoder of the target format, but also retains the color display advantages of the YUV4:4:4 format.

Description

Video encoding method, decoding method and related device

technical field

The present application relates to the technical field of video coding, and in particular, to a video coding method, a decoding method and related devices.

Background technique

In recent years, with the rapid development of smart phones, smart surveillance, Internet video and other industries, the demand for network bandwidth and storage capacity is increasing. Video codec technology can compress the original video data, significantly reducing the consumption of storage and transmission bandwidth, and then perform video decoding at the receiving end, and finally obtain an experience close to the quality of the original video data.

Since the human eye's perception of chroma components is weaker than that of luminance components, in most scenarios, the display effect of images in YUV4:2:0 format is similar to that of images in YUV4:4:4 format. Therefore, those skilled in the art can usually perform 1/2 downsampling on the row and column of the color (chroma) component of the image in YUV4:4:4 format, respectively, to obtain an image in YUV4:2:0 format, and then perform encoding and transmission .

However, for specific display elements, such as computer desktop text, icons, animations, etc., which have significant color boundaries, or video content containing a large amount of color information, the human eye can clearly perceive specific display elements in YUV4:4:4 format than YUV4:2 Certain elements in the :0 format have better color rendering. Therefore, how to be compatible with the YUV4:2:0 encoder/NV12 encoder while retaining the color display advantages of the YUV4:4:4 format during the encoding process is a technical issue being studied by those skilled in the art.

Contents of the invention

The embodiment of the present application discloses a video coding method, a decoding method and related devices, which are compatible with the coder/decoder of the target format (such as YUV4:2:0), and can also retain the color display advantages of the YUV4:4:4 format .

In the first aspect, the embodiment of the present application provides a video encoding method, the method including: determining the first image in the target format according to the image in the YUV4:4:4 format; wherein, the fourth image in the image in the YUV4:4:4 format Part of the pixel value is used to convert to the Y component in the first image, and the fifth part of the pixel value in the image in the YUV4:4:4 format is used to convert to the U component in the first image. The YUV4:4:4 The sixth part of the pixel value in the format image is used to convert to the V component in the first image; the fourth part of the pixel value, the fifth part of the pixel value and the sixth part of the pixel value respectively occupy the YUV4:4:4 2/3, 1/6, and 1/6 of the total number of pixel values of the image in the format; the target format includes one of YUV4:2:0, NV12, and NV21; encode the first image to obtain the first code flow.

Using the above method, the Y component, U component, and V component in the image of the YUV4:4:4 format are converted at the encoding end to obtain an image of the target format, and then the encoder of the target format (such as YUV4:2:0) pairs The image in the target format is encoded; after the image in the target format is decoded at the decoding end, the Y component, U component, and V component in the image in the target format are converted to obtain an image in the YUV4:4:4 format. This method is not only compatible with the codec/decoder of the target format, but also retains the color display advantages of the YUV4:4:4 format, and also reduces the color difference and the complexity of reference frame management.

In conjunction with the first aspect, in an optional solution, determining the first image in the target format according to the image in the YUV4:4:4 format includes: storing the image in the YUV4:4:4 format as the first image in the target format An image, wherein the fourth part of the pixel values in the YUV4:4:4 format image is used to store as the Y component in the first image, and the fifth part of the pixel values in the YUV4:4:4 format image is used for Stored as the U component in the first image, the sixth part of pixel values in the image in the YUV4:4:4 format is used to be stored as the V component in the first image.

In combination with the first aspect, or any of the above optional solutions of the first aspect, in yet another optional solution, before determining the first image in the target format according to the image in the YUV4:4:4 format, further includes :

Convert the image in RGB format to the image in the YUV4:4:4 format; if the image in the RGB format is applied to the first scene, determine the first image in the target format according to the image in the YUV4:4:4 format; if the RGB format The image of the image is applied to the second scene, the image in the YUV4:4:4 format is down-sampled, and the image after the down-sampling is encoded according to the encoder in the YUV4:2:0 format to obtain the second code stream; to the video The decoder sends the second code stream; wherein, the image resolution requirement of the second scene is lower than the image resolution requirement of the first scene.

In this method, for scenes with lower image resolution requirements, the traditional downsampling method is used for preprocessing and encoding, and for scenes with higher image resolution requirements, no loss of pixel values (that is, no loss of Y component, U component, and V component), which can not only meet the special requirements for chroma in scenes with high image resolution requirements to the greatest extent, but also minimize the The amount of encoded data in the scene.

In combination with the first aspect, or any of the above optional solutions of the first aspect, in another optional solution, the first scene includes at least one of the office document scene, the desktop icon scene, the text scene, and the LOGO scene One; the second scene includes at least one of a video scene, a game scene, and a natural scenery scene.

In combination with the first aspect, or any of the above optional solutions of the first aspect, in another optional solution, it also includes: if the difference between the pixel value of the RGB image and the pixel value of the previous frame RGB image exceeds If the preset threshold value is used, the image in RGB format is converted to the image in YUV4:4:4 format; if the difference between the pixels of the RGB image and the pixels of the previous RGB image does not exceed the preset threshold value, it is sent to the decoder Indication information, where the indication information is used to indicate that the pixels of the RGB image are the same as the pixels of the previous frame of RGB image.

In this way, when the pixel values of a frame of RGB image (may also be a part of RGB image) and the previous frame of RGB image (maybe a part of RGB image) have a small difference, there is no need to A frame of RGB image is re-encoded, and the previous frame of RGB image can be directly multiplexed, which can reduce the calculation pressure and improve the overall encoding efficiency.

In combination with the first aspect, or any of the above optional solutions of the first aspect, in another optional solution, determining the first image in the target format according to the image in the YUV4:4:4 format includes: The image in the YUV4:4:4 format is converted into a transitional image in the NV12 format, wherein the fourth part of the pixel values in the image in the YUV4:4:4 format is used to convert to the Y component in the transitional image, the YUV4:4 The fifth part of the pixel value in the :4 format image is used to convert to the U component in the transition image, and the sixth part of the pixel value in the YUV4:4:4 format image is used to convert to the V component in the transition image. The conversion method mentioned here may be to redefine the pixel values in the image in the YUV4:4:4 format, for example, redefining (or marking) the U component as the Y component. Then store the transition image in NV12 format as the first image in YUV4:2:0 format, and store the transition image in NV12 format as the first image in YUV4:2:0 format, which is a relatively mature technology, and there are many ways to implement it. They are not listed here. In combination with the first aspect, or any of the above optional solutions of the first aspect, in another optional solution: the fourth part of pixel values includes all Y components in the image in the YUV4:4:4 format and all U components; the fifth part of the pixel value includes half of all the V components in the image of the YUV4:4:4 format; the sixth part of the pixel value includes another part of all the V components in the image of the YUV4:4:4 format half.

Wherein, half of the all V components are V components of odd columns among all the V components, and the other half of all V components are V components of even columns among all V components.

Alternatively, half of the total V components are V components of odd lines in the total V components, and the other half of the total V components are V components of even lines in the total V components.

In a second aspect, an embodiment of the present application provides a video decoding method, the method including:

Obtaining a first code stream including target format data, wherein the target format includes one or more combinations of YUV4:2:0, NV12 or NV21; parsing the first code stream to obtain a first image; according to The first image determines an image in the YUV4:4:4 format; wherein, the first part of the pixel values in the first image is used to convert to the Y component of the image in the YUV4:4:4 format, and the second part in the first image The pixel value is used to convert to the U component of the image in the YUV4:4:4 format, and the third part of the pixel value in the first image is used to convert to the V component of the image in the YUV4:4:4 format; the first part of the pixels value, the second part of pixel values and the third part of pixel values each account for 1/3 of the total number of pixel values of the first image.

Using the above method, the Y component, U component, and V component in the image of the YUV4:4:4 format are converted at the encoding end to obtain an image of the target format, and then the encoder of the target format (such as YUV4:2:0) pairs The image in the target format is encoded; after the image in the target format is decoded at the decoding end, the Y component, U component, and V component in the image in the target format are converted to obtain an image in the YUV4:4:4 format. This method is not only compatible with the codec/decoder of the target format, but also retains the color display advantages of the YUV4:4:4 format, and also reduces the color difference and the complexity of reference frame management.

In conjunction with the second aspect, in an optional solution of the second aspect, determining the image in YUV4:4:4 format according to the first image includes: storing the first image as an image in YUV4:4:4 format Image, the first part of the pixel values in the first image is used to store the Y component of the image in the YUV4:4:4 format, and the second part of the pixel values in the first image is used to store the YUV4:4:4 format The U component of the image, the third part of the pixel values in the first image is used to store as the V component of the image in the YUV4:4:4 format.

In combination with the second aspect, or any of the above optional solutions of the second aspect, in yet another optional solution of the second aspect, the first image is an image in YUV4:2:0 format, according to the The first image determines the image in YUV4:4:4 format, including: storing the image in YUV4:2:0 format as a transitional image in NV12 format, and the first part of the pixel values in the image in YUV4:2:0 format are used to store as The Y component of the transition image, the second part of the pixel values in the YUV4:2:0 format image is used to store the U component of the transition image, and the third part of the pixel values in the YUV4:2:0 format image is used for It is relatively mature technology to store the first image in YUV4:2:0 format as the transition image in NV12 format as the V component of the transition image, and there are many implementation methods, which are not listed here. Then, convert the transitional image into an image in YUV4:4:4 format. The conversion method mentioned here can be to redefine the pixel values in the transitional image in NV12 format, for example, redefining part of the Y component (or marking ) is the U component.

In combination with the second aspect, or any of the above optional solutions of the second aspect, in yet another optional solution: the first part of pixel values includes half of all Y components in the first image; the second part The pixel value includes the other half of all Y components in the first image; the third part of pixel values includes all U components and all V components in the first image;

Wherein, all the U components in the first image are used as the odd row V components of the image in the YUV4:4:4 format, and all the V components in the first image are used as the image in the YUV4:4:4 format The even-numbered row V component of ;

Or, all the U components in the first image are used as the odd column V components of the image in the YUV4:4:4 format, and all the V components in the first image are used as the image in the YUV4:4:4 format The even-numbered columns of V components.

In a third aspect, an embodiment of the present application provides a video encoding device, which includes:

A determining unit, configured to determine the first image of the target format according to the image of the YUV4:4:4 format;

Wherein, the fourth part of pixel values in the YUV4:4:4 format image is used to convert to the Y component in the first image, and the fifth part of the pixel values in the YUV4:4:4 format image is used to convert to the The U component in the first image, the sixth part of the pixel value in the YUV4:4:4 format image is used to convert to the V component in the first image; the fourth part of the pixel value, the fifth part of the pixel value and The pixel values of the sixth part respectively account for 2/3, 1/6 and 1/6 of the total number of pixel values of the image in the YUV4:4:4 format; the target format includes one of YUV4:2:0, NV12 and NV21 kind;

The first encoding unit is configured to encode the first image to obtain a first code stream.

Using the above method, the Y component, U component, and V component in the image of the YUV4:4:4 format are converted at the encoding end to obtain an image of the target format, and then the encoder of the target format (such as YUV4:2:0) pairs The image in the target format is encoded; after the image in the target format is decoded at the decoding end, the Y component, U component, and V component in the image in the target format are converted to obtain an image in the YUV4:4:4 format. This method is not only compatible with the codec/decoder of the target format, but also retains the color display advantages of the YUV4:4:4 format, and also reduces the color difference and the complexity of reference frame management.

With reference to the third aspect, in an optional solution, the determining unit is specifically configured to:

The image in the YUV4:4:4 format is stored as the first image in the target format, wherein the fourth part of the pixel values in the image in the YUV4:4:4 format is used to store as the Y component in the first image, the The fifth part of the pixel values in the YUV4:4:4 format image is used to store as the U component in the first image, and the sixth part of the pixel values in the YUV4:4:4 format image is used to store as the first image The V component in .

In combination with the third aspect, or any of the above optional solutions of the third aspect, in another optional solution, the video encoding device further includes:

A conversion unit for converting an image in RGB format into an image in this YUV4:4:4 format;

If the image in RGB format is applied to the first scene, the determining unit is used to determine the first image in the target format according to the image in YUV4:4:4 format;

If the image in RGB format is applied to the second scene, the second encoding unit is used to downsample the image in YUV4:4:4 format, and encode the downsampled image according to the encoder in YUV4:2:0 format , get the second code stream;

The sending unit is further configured to send the second code stream to the video decoder;

Wherein, the image resolution requirement of the second scene is lower than the image resolution requirement of the first scene.

In this method, for scenes with lower image resolution requirements, the traditional downsampling method is used for preprocessing and encoding, and for scenes with higher image resolution requirements, no loss of pixel values (that is, no loss of Y component, U component, and V component), which can not only meet the special requirements for chroma in scenes with high image resolution requirements to the greatest extent, but also minimize the The amount of encoded data in the scene.

In combination with the third aspect, or any of the above optional solutions of the third aspect, in yet another optional solution, the first scene includes at least one of the office document scene, the desktop icon scene, the text scene, and the LOGO scene One; the second scene includes at least one of a video scene, a game scene, and a natural scenery scene.

In combination with the third aspect, or any of the above optional solutions of the third aspect, in another optional solution:

If the difference between the pixel of the RGB image and the pixel value of the previous frame of RGB image exceeds a preset threshold, the conversion unit is used to convert the image in RGB format into the image in YUV4:4:4 format;

If the difference between the pixels of the RGB image and the pixels of the previous RGB image does not exceed a preset threshold, the sending unit is used to send indication information to the video decoder, and the indication information is used to indicate that the pixels of the RGB image are different from the pixels of the RGB image. The pixels of the RGB image from the previous frame are the same.

In this way, when the pixel values of a frame of RGB image (may also be a part of RGB image) and the previous frame of RGB image (maybe a part of RGB image) have a small difference, there is no need to A frame of RGB image is re-encoded, and the previous frame of RGB image can be directly multiplexed, which can reduce the calculation pressure and improve the overall encoding efficiency.

In combination with the third aspect, or any of the above optional solutions of the third aspect, in another optional solution, the first image aspect of the target format is determined according to the image in the YUV4:4:4 format, the determination The unit is specifically used to: convert the image in YUV4:4:4 format into a transitional image in NV12 format, wherein the fourth part of the pixel values in the image in YUV4:4:4 format is used to convert to Y in the transitional image component, the fifth part of the pixel value in the YUV4:4:4 format image is used to convert to the U component in the transition image, and the sixth part of the pixel value in the YUV4:4:4 format image is used to convert to the transition For the V component in the image, the conversion method mentioned here may be to redefine the pixel value in the image in the YUV4:4:4 format, for example, redefining (or marking) the U component as the Y component. Then store the transition image in NV12 format as the first image in YUV4:2:0 format, and store the transition image in NV12 format as the first image in YUV4:2:0 format, which is a relatively mature technology, and there are many ways to implement it. They are not listed here. In combination with the first aspect, or any of the above optional solutions of the first aspect, in another optional solution: the fourth part of pixel values includes all Y components in the image in the YUV4:4:4 format and all U components; the fifth part of the pixel value includes half of all the V components in the image of the YUV4:4:4 format; the sixth part of the pixel value includes another part of all the V components in the image of the YUV4:4:4 format half.

Wherein, half of the all V components are V components of odd columns among all the V components, and the other half of all V components are V components of even columns among all V components.

Alternatively, half of the total V components are V components of odd lines in the total V components, and the other half of the total V components are V components of even lines in the total V components.

In a fourth aspect, the embodiment of the present application provides a video decoding device, which includes:

An acquisition unit, configured to acquire a first code stream including target format data, wherein the target format includes one or more combinations of YUV4:2:0, NV12 or NV21;

A parsing unit, configured to parse the first code stream to obtain a first image;

A determining unit, configured to determine an image in YUV4:4:4 format according to the first image; wherein, the first part of pixel values in the first image is used to convert to the Y component of the image in YUV4:4:4 format, and the first part The second part of pixel values in an image is used to convert to the U component of the image in the YUV4:4:4 format, and the third part of the pixel values in the first image is used to convert to the V of the image in the YUV4:4:4 format Components; the first part of pixel values, the second part of pixel values and the third part of pixel values each account for 1/3 of the total number of pixel values of the first image.

Using the above method, the Y component, U component, and V component in the image of the YUV4:4:4 format are converted at the encoding end to obtain an image of the target format, and then the encoder of the target format (such as YUV4:2:0) pairs The image in the target format is encoded; after the image in the target format is decoded at the decoding end, the Y component, U component, and V component in the image in the target format are converted to obtain an image in the YUV4:4:4 format. This method is not only compatible with the codec/decoder of the target format, but also retains the color display advantages of the YUV4:4:4 format, and also reduces the color difference and the complexity of reference frame management.

With reference to the fourth aspect, in an optional solution, the determining unit is specifically configured to:

The image in the YUV4:4:4 format is stored as a first image in the YUV4:2:0 format, and the first part of the pixel values in the first image is used to store the Y component of the image in the YUV4:4:4 format, the The second part of the pixel values in the first image is used to store as the U component of the image in the YUV4:4:4 format, and the third part of the pixel values in the second image is used to store as the U component of the image in the YUV4:4:4 format V component.

In combination with the fourth aspect, or any of the above optional solutions of the fourth aspect, in yet another optional solution of the fourth aspect, the first image is an image in YUV4:2:0 format, according to the The first image determines the image in YUV4:4:4 format, including: storing the image in YUV4:2:0 format as a transitional image in NV12 format, and the first part of the pixel values in the image in YUV4:2:0 format are used to store as The Y component of the transition image, the second part of the pixel values in the YUV4:2:0 format image is used to store the U component of the transition image, and the third part of the pixel values in the YUV4:2:0 format image is used for It is relatively mature technology to store the first image in YUV4:2:0 format as the transition image in NV12 format as the V component of the transition image, and there are many implementation methods, which are not listed here. Then, convert the transitional image into an image in YUV4:4:4 format. The conversion method mentioned here can be to redefine the pixel values in the transitional image in NV12 format, for example, redefining part of the Y component (or marking ) is the U component.

In combination with the fourth aspect, or any of the above optional solutions of the fourth aspect, in another optional solution,

The first partial pixel value includes half of all Y components in the first image;

The second portion of pixel values includes the other half of all Y components in the first image;

The third part of pixel values includes all U components and all V components in the first image;

Wherein, all the U components in the first image are used as the odd row V components of the image in the YUV4:4:4 format, and all the V components in the first image are used as the image in the YUV4:4:4 format The even-numbered row V component of ;

Or, all the U components in the first image are used as the odd column V components of the image in the YUV4:4:4 format, and all the V components in the first image are used as the image in the YUV4:4:4 format The even-numbered columns of V components.

In the fifth aspect, the embodiment of the present application provides an electronic device, the electronic device includes a processor and a memory, the processor is coupled to the memory, and the memory is used to store computer program code, the computer program code includes computer software instructions, when When the computer software instructions are executed by the electronic device, the electronic device executes the video coding method described in the first aspect or any optional solution of the first aspect, or executes the second aspect or any optional solution of the second aspect. An alternative scheme is described for the video decoding method.

In a sixth aspect, the embodiment of the present application provides a computer-readable storage medium, including: computer software instructions;

When the computer software instructions run in the electronic device, the electronic device executes the video encoding method described in the first aspect or any optional solution of the first aspect, or executes the second aspect or the second aspect The video decoding method described in any optional solution.

In the seventh aspect, the embodiment of the present application provides a computer program product. When the computer program product is run on a computer, the electronic device executes the video described in the first aspect or any optional solution of the first aspect. An encoding method, or execute the video decoding method described in the second aspect or any optional solution of the second aspect.

By implementing the embodiment of the present application, the Y component, U component, and V component in the image of the YUV4:4:4 format are converted at the encoding end to obtain an image in the target format, and then the target format (such as YUV4:2:0) The encoder encodes the image in the target format; after the image in the target format is decoded at the decoder, the Y component, U component, and V component in the image in the target format are converted to obtain an image in the YUV4:4:4 format. This method is not only compatible with the codec/decoder of the target format, but also retains the color display advantages of the YUV4:4:4 format, and also reduces the color difference and the complexity of reference frame management.

Description of drawings

The accompanying drawings used in the embodiments of the present application are introduced below.

FIG. 1A is a schematic diagram of the pixel value ratio relationship of an image in a YUV4:4:4 format provided by an embodiment of the present application;

FIG. 1B is a schematic diagram of the pixel value ratio relationship of an image in a YUV4:2:0 format provided by an embodiment of the present application;

Fig. 1C is a schematic diagram of the pixel value ratio relationship of an image in NV12 format provided by the embodiment of the present application;

FIG. 2A is a schematic diagram of a scene for encoding and decoding an image in YUV4:4:4 format provided by an embodiment of the present application;

FIG. 2B is a schematic diagram of a scene encoding an image in YUV4:4:4 format provided by an embodiment of the present application;

Fig. 3 is a schematic diagram of another scene encoding an image in YUV4:4:4 format provided by the embodiment of the present application;

Fig. 4 is a block diagram of a YUV4:4:4-based video codec system provided by an embodiment of the present application;

FIG. 5 is a schematic flowchart of a video encoding and decoding method provided by an embodiment of the present application;

FIG. 6A is a schematic flowchart of a video encoding and decoding method provided by an embodiment of the present application;

FIG. 6B is a schematic diagram of a scene comparing image differences provided by an embodiment of the present application;

FIG. 6C is a schematic diagram of a scene where a YUV4:4:4 format is converted into a YUV4:2:0 format according to an embodiment of the present application;

FIG. 6D is a schematic diagram of another scene where the YUV4:4:4 format is converted into the YUV4:2:0 format provided by the embodiment of the present application;

FIG. 6E is a schematic diagram of another scene where the YUV4:4:4 format is converted into the YUV4:2:0 format provided by the embodiment of the present application;

FIG. 6F is a schematic diagram of a scene where the YUV4:4:4 format is converted to the NV12 format and then converted to the YUV4:2:0 format provided by the embodiment of the present application;

FIG. 6G is a schematic diagram of another scene where the YUV4:4:4 format is converted to the NV12 format and then converted to the YUV4:2:0 format provided by the embodiment of the present application;

FIG. 6H is a schematic diagram of a scene where a YUV4:2:0 format is converted to a YUV4:4:4 format according to an embodiment of the present application;

FIG. 61 is a schematic diagram of another scene where the YUV4:2:0 format is converted into the YUV4:4:4 format provided by the embodiment of the present application;

FIG. 6J is a schematic diagram of another scene where the YUV4:2:0 format is converted into the YUV4:4:4 format provided by the embodiment of the present application;

FIG. 6K is a schematic diagram of a scene where the YUV4:2:0 format is converted to the NV12 format and then converted to the YUV4:4:4 format provided by the embodiment of the present application;

FIG. 6L is a schematic diagram of another scene where the YUV4:2:0 format is converted to the NV12 format and then converted to the YUV4:4:4 format provided by the embodiment of the present application;

FIG. 7 is a schematic structural diagram of a video encoding device provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a video decoding device provided by an embodiment of the present application;

FIG. 9 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

Embodiments of the present application are described below with reference to the drawings in the embodiments of the present application.

First, the relevant technologies and technical terms involved in this application are introduced below.

Video encoding (video encoding): The process of compressing an image sequence into a video stream.

Video decoding (video decoding): the process of restoring the video code stream into a reconstructed image according to specific grammatical rules and processing methods.

Peak Signal to Noise Ratio (PSNR): A data processing method that calculates the mean square error between the original image and the processed image relative to a fixed value, the unit is db, and is often used to evaluate image quality objective criteria.

Bit rate: the average number of bits encoded per second when an image sequence is encoded.

BD-rate: Used to evaluate the rate-distortion (RD) performance of different video encoders. For example, take the 4 code streams of the encoder and the corresponding PSNR points, perform curve fitting, calculate the R-D curve integral, and compare the difference between the R-D curve integrals of the two encoding algorithms to obtain the rate-distortion.

Coding complexity: determined by the texture complexity and motion complexity of the video. Generally, the more uniform the video image and the smoother the motion, the lower the encoding complexity; otherwise, the higher the encoding complexity.

Bilinear interpolation method: use the four neighboring pixel values of the pixel to be obtained to make a linear difference in the horizontal and vertical directions.

The nearest neighbor method: By judging the distance from the position to be interpolated to the known pixel, the nearest pixel value is taken as the pixel value of the position to be interpolated.

YUV: is a color coding method, where Y is brightness information, Luminance, U and V are chrominance information Chrominance, which is used to describe the color and saturation of the image, and is used to specify the color of the pixel.

YUV4:4:4: It is a storage format of YUV. A Y component corresponds to a U component and a V component, where the Y component, U component, and V component occupy the same number of bytes, and the chroma value (U, V) Without sampling, the storage order in the memory is to store the Y component first, then the U component, and finally the V component. As shown in Figure 1A.

YUV4:2:0: is a storage format of YUV. Every 4 Y components share a set of UV components. Among them, U and V occupy 1/4 of the bytes of Y, and the chroma value (U, V ) is the 1/2 sampling of the row and column of the chroma component of YUV4:4:4. The storage order in the memory is to store the Y component first, then the U component, and finally the V component, as shown in Figure 1B.

NV12: It is a storage format of YUV, in which every 4 Y components share a set of UV components, among which, the U component and V component occupy 1/4 of the bytes of the Y component, and the storage order in the memory is to store Y first component, U component, and V component are stored alternately, as shown in Figure 1C.

NV21: It is a storage format of YUV, in which every 4 Y components share a set of UV components, among which, the U component and V component occupy 1/4 of the bytes of the Y component, and the storage order in memory is to store Y first component, U component, and V component are stored alternately. The difference from NV12 is that in NV12, the U component is in the front and the V component is in the back. In NV21, the V component is in the front and the U component is in the back.

The applicant of this application cites three technical solutions that are compatible with the YUV4:2:0 encoder and retain the color display advantages of the YUV4:4:4 format during the encoding process, but these three technical solutions are all There are different degrees of defects, which are introduced separately below.

Option 1, as shown in Figure 2A, the row and column of the chrominance component of the image in the YUV4:4:4 format are subjected to 1/2 downsampling at the encoding end to obtain an image in the YUV4:2:0 format, and then the YUV format The image is encoded, and the encoded code stream is transmitted to the decoding end. After the decoding end decodes the image in the YUV4:2:0 format, the chroma component is up-sampled and recovered to obtain the image in the YUV4:4:4 format, and then the restored The obtained image in YUV4:4:4 format is sent to display, and there are many algorithms for downsampling and upsampling the chroma, such as bilinear interpolation, nearest neighbor method, etc.

However, the display effect of the method shown in Fig. 2A depends on downsampling from an image in YUV4:4:4 format to an image in YUV4:2:0 format, and restoring from an image in YUV4:2:0 format to YUV4:4: 4-format chroma sampling algorithm for images. Among them, from the YUV4:2:0 format image to the YUV4:4:4 format image, it is generally used to estimate the vacant chroma component by referring to the distribution or proportion of the adjacent chroma or luminance components in the local range. Due to the need to refer to adjacent pixel values for calculation, the complementary color is significantly different from the real color in the boundary area with obvious color difference or the area with discrete color points.

Solution 2, convert the image in YUV4:4:4 format into two images in YUV4:2:0 format, then encode the two images in YUV4:2:0 format, and transmit the coded stream to the decoding end After that, the decoding end decodes the two images in YUV4:2:0 format, and then combines the two images in YUV4:2:0 format into one image in YUV4:4:4 format, and then the one YUV4 :Send images in 4:4 format for display. As shown in Figure 2B, the total number of pixels of the image in the YUV4:4:4 format is twice the total number of pixels in the image in the YUV4:2:0 format, so it can be divided into two images in the YUV4:2:0 format, The first image in YUV4:2:0 format is called the main image, which is a standard YUV4:2:0 format image, its Y component is the Y component of the YUV4:4:4 format image, and its U and V components They are 1/4 samples of the U and V components in the image in the YUV4:4:4 format, respectively. The second image in YUV4:2:0 format is called the secondary image, and its Y component is 1/2 sampling of the U and V components in the image in the YUV4:4:4 format. The U and V components are placed up and down, and the U of the secondary image The component and the V component are the remaining 1/4 samples of the U and V components in the image in the YUV4:4:4 format. The image in YUV4:2:0 format obtained according to this conversion method can fully guarantee the geometric consistency of the Y, U, and V components, which is beneficial for the YUV4:2:0 encoder to encode in combination with the geometric consistency, and obtain lower code rate. After decoding by the decoder, the two images in YUV4:2:0 format are then inversely transformed, and the two images in YUV4:2:0 format can be combined into one image in YUV4:4:4 format, which prevents the Chroma loss.

However, the image in YUV4:4:4 format is divided into two sub-images in YUV4:2:0 format, where the Y, U, and V components of the sub-image are all sampled for the U and V components of YUV4:4:4, The number of sampling points is large, and the amount of calculation is large. Secondly, the contents of the Y, U, and V components of the picture are all arranged in a top-down structure, and the arrangement structure is complex, which will increase the code rate after encoding, and will output two images in the YUV4:2:0 format and send them to the encoder Finally, it leads to the complexity of encoder reference frame management.

Solution 3, convert the image in YUV4:4:4 format into two images in YUV4:2:0 format, then encode the two images in YUV4:2:0 format, and transmit the coded stream to the decoding end After that, the decoding end decodes the two images in YUV4:2:0 format, and then combines the two images in YUV4:2:0 format into one image in YUV4:4:4 format, and then the one YUV4 :Send images in 4:4 format for display. As shown in Figure 3, label 103 is the image of YUV4:4:4 format, label 104 has shown the format conversion process, label 105 has shown the image of output YUV4:2:0 format, and wherein, label 202 and 203 are two respectively A subimage in YUV4:2:0 format.

In the image of the YUV4:2:0 format of the label 202, the Y component is the Y component in the image of the YUV4:4:4 format of the label 103, and in the image of the YUV4:2:0 format of the label 203, the Y component is the label 103 The U component of the image in the YUV4:4:4 format in the middle, and then the V component in the image of the YUV4:4:4 format of the label 103 is sampled to obtain four subgraphs in the label 201, which are respectively represented as V11, V12, V21, V22, then use V11 and V12 as the U and V components in the YUV4:2:0 format image of the label 202 respectively, and use V21 and V22 as the U and V components in the YUV4:2:0 format image of the label 203 respectively . In this scheme, when converting an image in YUV4:4:4 format to two images in YUV4:2:0 format, the Y components of the two images in YUV4:2:0 format are images in YUV4:4:4 format The Y and U components of , without sampling, require less calculation.

However, dividing an image in YUV4:4:4 format into two subimages in YUV4:2:0 format requires sampling the V component of an image in YUV4:4:4 format, and taking the sampling results as two YUV4:2 The U and V components of the :0 format image have a large amount of calculation during the sampling process, and will output 2 frames of YUV4:2:0, which will complicate the encoder's reference frame management after being sent to the encoder.

The applicant of this application further provides a video coding method, a decoding method and related devices, which can not only be compatible with the YUV4:2:0 encoder, but also retain the color display advantages of the YUV4:4:4 format, while avoiding the above three solutions Defects.

Please refer to FIG. 4, which is a block diagram of a YUV4:4:4-based video encoding and decoding system described in the embodiment of the present application. As shown in FIG. 4 , the video coding system includes source device 10 and destination device 20 . Source device 10 generates encoded video data. Source device 10 may thus be referred to as a video encoding device. Destination device 20 may decode the encoded video data generated by source device 10 . Accordingly, destination device 20 may be referred to as a video decoding device. Various implementations of source device 10, destination device 20, or both may include one or more processors and memory coupled to the one or more processors. The memory may include, but is not limited to, RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures accessible by the computer, as described herein.

Source device 10 and destination device 20 may include a variety of devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, etc. computer, television, camera, display device, digital media player, video game console, vehicle computer, or the like.

Destination device 20 may receive encoded video data from source device 10 via link 30 . Link 30 may include one or more media or devices capable of moving encoded video data from source device 10 to destination device 20 . In one example, link 30 may include one or more communication media that enable source device 10 to transmit encoded video data directly to destination device 20 in real time. In this example, source device 10 may modulate encoded video data according to a communication standard, such as a wireless communication protocol, and may transmit the modulated video data to destination device 20, which may be referred to as code stream, or code stream. The one or more communication media may include wireless and/or wired communication media, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (eg, the Internet). The one or more communication media may include routers, switches, base stations, or other equipment that facilitates communication from source device 10 to destination device 20 .

In another example, encoded data may be output from output interface 140 to storage device 40 . Similarly, encoded data can be accessed from the storage device 40 through the input interface 240 . Storage device 40 may comprise any of a variety of distributed or locally accessed data storage media, such as hard drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

In another example, storage device 40 may correspond to a file server or another intermediate storage device that may hold encoded video generated by source device 10 . Destination device 20 may access the stored video data from storage device 40 via streaming or download. The file server may be any type of server capable of storing and transmitting encoded video data to destination device 20 . File servers include web servers (eg, for websites), FTP servers, network-attached storage (NAS) devices, or local disk drives. Destination device 20 may access the encoded video data through any standard data connection, such as an Internet connection. This may include wireless channels (eg, Wi-Fi connections), wired connections (eg, DSL, cable modem, etc.), or a combination of both suitable for accessing encoded video data stored on a file server. The transmission of encoded video data from storage device 40 may be a streaming transmission, a download transmission, or a combination of both.

The motion vector prediction technology of the present application can be applied to video codec to support various multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, streaming video transmission (for example, via the Internet), for storage in data storage Encoding of video data on a medium, decoding of video data stored on a data storage medium, or other applications. In some examples, a video coding system may be used to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

The video coding system illustrated in FIG. 4 is merely an example, and the techniques of this disclosure may be applicable to video coding settings (eg, video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other instances, data is retrieved from local storage, streamed over a network, and so on. A video encoding device may encode and store data to memory, and/or a video decoding device may retrieve and decode data from memory. In many instances, encoding and decoding are performed by devices that do not communicate with each other but merely encode and/or retrieve data from memory and decode data to memory.

In the example of FIG. 4 , source device 10 includes video source 120 , video encoder 100 and output interface 140 . In some examples, output interface 140 may include a conditioner/demodulator (modem) and/or a transmitter. Video source 120 may include a video capture device (e.g., a video camera), a video archive containing previously captured video data, a video feed interface for receiving video data from a video content provider, and/or a computer for generating video data A graphics system, or a combination of such sources of video data.

Video encoder 100 may encode video data from video source 120 . In some examples, source device 10 transmits the encoded video data directly to destination device 20 via output interface 140 . In other examples, the encoded video data may also be stored on storage device 40 for later access by destination device 20 for decoding and/or playback.

In the example of FIG. 4 , destination device 20 includes input interface 240 , video decoder 200 and display device 220 . In some examples, input interface 240 includes a receiver and/or a modem. Input interface 240 may receive encoded video data via link 30 and/or from storage device 40 . The display device 220 may be integrated with the destination device 20 or may be external to the destination device 20 . In general, display device 220 displays decoded video data. The display device 220 may include various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

Although not shown in FIG. 4 , in some aspects video encoder 100 and video decoder 200 may each be integrated with an audio encoder and decoder, and may include appropriate multiplexer-demultiplexer units. or other hardware and software to handle the encoding of both audio and video in a common data stream or in separate data streams. In some examples, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP), if applicable.

Video encoder 100 and video decoder 200 may each be implemented as any of a variety of circuits such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), Field Programmable Gate Array (FPGA), discrete logic, hardware, or any combination thereof. If the application is implemented partially in software, a device may store instructions for the software in a suitable non-transitory computer-readable storage medium and may execute the instructions in hardware using one or more processors to thereby Implement the technology of this application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors. Each of video encoder 100 and video decoder 200 may be included in one or more encoders or decoders, either of which may be integrated as a combined encoder/decoder in a corresponding device. Part of a decoder (codec).

This application may generally refer to video encoder 100 as “signaling” or “transmitting” certain information to another device, such as video decoder 200 . The term "signaling" or "transmitting" may generally refer to the transmission of syntax elements and/or other data used to decode compressed video data. This transfer can occur in real time or near real time. Alternatively, this communication may take place after a period of time, such as when encoding stores a syntax element in an encoded bitstream to a computer-readable storage medium, and the decoding device may then do so after the syntax element is stored on this medium. This syntax element is retrieved at any time.

The video encoder 100 and the video decoder 200 may operate according to a video compression standard such as High Efficiency Video Coding (HEVC) or an extension thereof, and may comply with the HEVC test model (HM). Alternatively, video encoder 100 and video decoder 200 may also operate according to other industry standards, such as ITU-T H.264, H.265 standards, or extensions of such standards. However, the techniques of this application are not limited to any particular codec standard.

The video encoder 100 in this embodiment of the present application may be a YUV4:2:0 encoder, or an NV12 encoder, or an NV21 encoder. The video decoder 100 in the embodiment of the present application may be a YUV4:2:0 decoder, or an NV12 decoder, or an NV21 decoder. Usually, what format the video encoder uses to encode the image, the video decoder needs to decode the image in the same format. For example, if a video decoder encodes an image through a YUV4:2:0 encoder, the video decoder needs to decode the code stream from the video encoder through a YUV4:2:0 decoder.

Video encoding and decoding are aimed at videos. Since videos are composed of image sequences, in the following descriptions of video encoding and decoding, the objects of encoding and decoding are explained using images in videos as examples.

Please refer to Figure 5. Figure 5 is a schematic diagram of a coding and decoding scenario provided by the embodiment of the present application. This scenario includes the following modules:

RGB forward conversion unit 501, configured to convert the source image in RGB format into an image in YUV4:4:4 format.

The YUV forward conversion unit 502 is used to convert the Y, U, and V components in the image of the YUV4:4:4 format according to the target format to obtain an image of the target format. The conversion process in this application does not discard the original Y, U , the V component, for example, convert (or redefine) the U component in the image in the YUV4:4:4 format to the Y component.

The purpose of the conversion is to keep the YUV4:4 in the code stream obtained after the target encoder 503 encodes the image in the YUV4:4:4 format when the image in the YUV4:4:4 format is encoded by the target encoder 503 : All Y, U, V components in the 4-format image.

In the embodiment of this application, the source image in RGB format is converted into an image in YUV4:4:4 format, and then the image in YUV4:4:4 format is converted into an image in target format, which can be done through a function (or algorithm flow) To achieve, can also be divided into two functions to achieve. For example, when it is implemented by one function, the processing process has one input and one output, the input is an image in RGB format, and the output is an image in the target format; for another example, when it is implemented by two functions, the processing process has two inputs and two times Output, the first input is an image in RGB format, the first output is an image in YUV4:4:4 format, the second input is an image in YUV4:4:4 format, and the second output is an image in the target format.

The target encoder 503 is configured to encode the image in the target format converted by the YUV forward conversion unit 502, where the image in the target format may be referred to as a first image for the convenience of subsequent description.

The target decoder 504 is configured to decode the code stream containing the first image, so as to decode the first image. The target decoder 504 adopts the technology coupled with the target encoder 503 , that is, the technical principle of decoding by the target decoder 504 is the reverse operation of the technical principle of encoding by the target encoder 503 .

Optionally, the target encoder 503 and the target decoder 504 may be respectively deployed on different devices. As shown in FIG. 4, the target encoder 503 may be deployed in the source device 10, and the target decoder may be deployed in the destination In the device 20, therefore, the first image can output a code stream through the output interface 140 in the source device 10, and the code stream includes the first image; correspondingly, the destination device 20 obtains the code stream through the input interface 240, and then The first image is decoded from the code stream by the target decoder 504 .

The YUV inverse conversion unit 505 is used to redefine the Y, U, and V components in the first image. The redefinition process in this application does not discard the original Y, U, and V components, but only converts the original Y, U, and V components. The types of U and V components have been completely or partially redefined. For example, part of the Y component in the first image is redefined as a U component.

The purpose of the redefinition is to make the distribution of the existing Y, U, and V components in the first image of the target format comply with the YUV4:4:4 format.

The RGB inverse conversion unit 506 is configured to convert the image conforming to the YUV4:4:4 format converted by the YUV inverse conversion unit 505 into an image in RGB format, that is, an image to be displayed.

Please refer to FIG. 6A. FIG. 6A is a schematic flowchart of an encoding and decoding method provided by an embodiment of the present application. This method can be implemented based on the system architecture shown in FIG. 4 and the scenario shown in FIG. 5. The method includes but Not limited to the following steps:

Step S600: The video encoding device compares the pixels of the RGB image with the pixels of the previous frame of the RGB image.

In the embodiment of the present application, the RGB image can be a complete image, or it can be considered as a region divided from a completed image, that is, a sub-image, and the image division method is not limited, for example, height, etc. can be used Divide, or width equal division, or height unequal division, or the mode of width unequal division, of course, can also be divided in other ways.

It should be noted that when the RGB image is an area divided from a complete image, the previous frame image of this complete image will also be divided in the same way, which can be called the previous frame of a complete image. The frame image is a reference image to facilitate subsequent descriptions. In the embodiment of the present application, the position of the RGB image in the complete image may be the same as or different from the position of the previous frame image of the RGB image in the reference image. , that is, the location of the RGB image and the previous frame of the RGB image may or may not be the same. For example, referring to Fig. 6B, the complete image is the nth frame image, which can be divided into image area 1, image area 2, image area 3, ..., image area m in turn according to the preset division method, so the target The image is the n-1th frame image, and the same division method can be divided into image area 1, image area 2, image area 3, ..., image area m, so the target image is the n-1th frame image. An optional scheme, if the RGB image is the image area 1 in the nth frame image, then the previous frame image of the GRB image is the image area 1 of the n-1th frame image; if the RGB image is the nth image image area 2 in the frame image, then the previous frame image of the GRB image is the image area 2 of the n-1th frame image; and so on. Another optional solution, if the RGB image is the image area 1 in the nth frame image, then the previous frame image of the GRB image is any image area of the n-1th frame image, for example, the GRB image The previous frame of image is the image area 1, image area 2, image area 3, or other image areas of the n-1th frame image.

If the difference (such as the difference) between the pixel of the RGB image and the pixel value of the previous frame RGB image exceeds the preset threshold, then perform step S601; If the pixel value difference of the image area 1 of the (n-1)th frame (ie, the previous frame image of the RGB image) exceeds a preset threshold, step S601 is executed.

If the difference (such as difference) between the pixel of the RGB image and the pixel value of the previous frame RGB image does not exceed the preset threshold, the RGB image is not encoded, but the instruction information is sent to the video decoder, and the instruction information is used for Indicates that the decoding information of the RGB image is the same as that of the previous frame of RGB image, that is, when the decoding end (such as a video decoder) needs to determine the pixel value of the RGB image, it directly multiplexes the previous frame of the RGB image pixel value. As shown in Figure 6B, if the difference in pixel values between the image area 1 of the nth frame (ie the RGB image) and the image area 1 of the n-1 frame (ie the previous frame image of the RGB image) does not exceed the preset threshold, Then do not encode, but send instruction information to the video decoder.

The preset threshold is a preset reference value for measuring pixel values, and its specific size can be configured according to actual needs. Optionally, different reference values can be set for different image application scenarios.

In an optional solution, the execution of the subsequent step S601 does not need to depend on the step S600.

Step S601: the video encoding device converts the source image in RGB format into an image in YUV4:4:4 format.

Optionally, the conversion operation may be specifically performed by an RGB forward conversion unit in the video encoding device.

In an optional solution, after the converted image in YUV4:4:4 format is obtained, step S602 is continued.

In another optional solution, after converting the image in YUV4:4:4 format, it is necessary to judge the application scenario of the RGB image, and perform different operations for different application scenarios, as follows:

If the image in RGB format is applied to the first scene, step S602 is performed. Optionally, in this case, a flag can be carried in the first code stream obtained after steps S602 and S603 are performed, so that the object that has obtained the data of the first code stream can know the application scenario through the flag, or It is said to know in what way the first code stream needs to be decoded. For example, if the flag bit is set to 11, then the object that has obtained the first code stream will know that it will be decoded in the following steps S606 and S607; if the flag bit is not 11, then steps S606 and S607 will not be passed. way to decode.

If the image in the RGB format is applied to the second scene, the image in the YUV4:4:4 format is down-sampled, and then the down-sampled image is encoded according to the encoder of the target format (such as YUV4:2:0) ( For example, the second scheme shown in Figure 2B and the third scheme shown in Figure 3 both obtain two images in the YUV4:2:0 format by downsampling, and then perform a process on the two images in the YUV4:2:0 format Encoding. Of course, there are many other implementations of first downsampling and then encoding, which are not listed here), to obtain the second code stream, and then send the second code stream to the video decoder. Optionally, in this case, a flag bit can be carried in the obtained second code stream, so that the object that obtains the second code stream can know the application scenario through the flag bit, or know what method needs to be used to The second code stream is decoded. For example, if the flag bit is set to 00, then the object that has obtained the second code stream will know that it will be decoded by the method of step S608 later, if the flag bit is not 00, it will not be decoded by the method of step S608 .

In this embodiment of the present application, the code stream includes data obtained after encoding images or videos.

Wherein, the image resolution requirement of the second scene is lower than the image resolution requirement of the first scene.

For example, the first scene includes office documents, desktop icons, text, logos (LOGO) and other scenes; the second scene includes video, games, natural scenery and other scenes. Specifically which ones should be classified as the first scenario and which ones should be classified as the second scenario can be pre-configured by the user according to requirements.

Step S602: The video encoding device determines the first image in the target format according to the image in the YUV4:4:4 format.

Among them, the fourth part of the pixel value in the YUV4:4:4 format image is used to convert to the Y component in the first image, and the fifth part of the pixel value in the YUV4:4:4 format image is used to convert to the first image The U component, the sixth part of the pixel value in the YUV4:4:4 format image is used to convert to the V component in the first image; the fourth part of the pixel value, the fifth part of the pixel value and the sixth part of the pixel value respectively account for YUV4 : 2/3, 1/6, and 1/6 of the total number of pixel values of an image in 4:4 format; the target format includes one of YUV4:2:0, NV12, and NV21.

Several optional ways of determining the first image in the target format are exemplified below.

Method 1: Specify the encoding rules of the encoder in the target format in the encoding protocol, so that when the encoder in the target format encodes an image in the YUV4:4:4 format, the fourth image in the image in the YUV4:4:4 format Part of the pixel value is regarded as the Y component, the fifth part of the pixel value in the image of the YUV4:4:4 format is regarded as the U component, and the sixth part of the pixel value in the image of the YUV4:4:4 format is regarded as the V component; During the period, the number of pixel values in the fourth part accounts for 2/3 of the total number of pixel values of the image in the YUV4:4:4 format, and the number of pixel values in the fifth part accounts for the pixel values in the image in the YUV4:4:4 format 1/6 of the total number, the number of pixel values in the sixth part accounts for 1/6 of the total number of pixel values of the image in the YUV4:4:4 format.

For example, the Y component and the U component in the image of the YUV4:4:4 format are regarded as the Y component, that is, the fourth part of the pixel value includes all the Y components and all the U components in the image of the YUV4:4:4 format; Half of the V component in the image of the YUV4:4:4 format is regarded as the U component, that is, the fifth part of the pixel value includes half of all the V components in the image of the YUV4:4:4 format; and YUV4:4: The other half of the V component in the image in the 4 format is regarded as the V component, and the sixth part of pixel values includes the other half of all the V components in the image in the YUV4:4:4 format.

Optionally, half of the all V components are V components of odd lines among the all V components; the other half of all V components are V components of even lines among all V components.

The second way is to re-store the Y, U, V components (all or part) of the image in the YUV4:4:4 format. The process of re-storage does not discard the original Y component, U component, V component, and does not change the Y Component, U component, V component content, the pixel value stored in the Y component position is equivalent to redefining the Y component, and the pixel value stored in the U component position is equivalent to redefining the U component and storing in the V component position The pixel value is equivalent to being redefined as the V component.

Specifically, the fourth part of the pixel value in the YUV4:4:4 format image is stored as the Y component, the fifth part of the pixel value in the YUV4:4:4 format image is stored as the U component, and the YUV4:4:4 format The sixth part of the pixel value in the image is stored as the V component; the number of the fourth part of the pixel value accounts for 2/3 of the total number of pixel values of the image in the YUV4:4:4 format, and the number of the fifth part of the pixel value accounts for the 1/6 of the total number of pixel values of the image in the YUV4:4:4 format, and the number of the sixth part of the pixel values accounts for 1/6 of the total number of pixel values of the image in the YUV4:4:4 format.

For example, as shown in Figure 6C, the Y component in the image in the YUV4:4:4 format is stored as the Y component, and the U component in the image in the YUV4:4:4 format is stored as the Y component, that is, the above-mentioned fourth part of pixels Values include all Y components and all U components in images in YUV4:4:4 format. Store half of the V components in the image in the YUV4:4:4 format as the U component, that is, the fifth part of pixel values includes half of all the V components in the image in the YUV4:4:4 format. The other half of the V component in the image in the YUV4:4:4 format is stored as the V component, that is, the sixth part of the pixel value includes the other half of all the V components in the image in the YUV4:4:4 format, and Figure 6C is obtained by storing The image in the target format is an image in YUV4:2:0 format as an example.

For ease of understanding, the position distribution of pixel values in the image is shown in more detail below in conjunction with Figure 6D. Optionally, as shown in Figure 6D, the Y component in the image in the YUV4:4:4 format is kept as Y Component, store the U component in the YUV4:4:4 format image as the Y component, store the odd lines in the V component in the YUV4:4:4 format image as the U component, and store the YUV4:4:4 format The even-numbered lines in the V components in the image remain as V components, that is, half of the above-mentioned all V components are the V components of the odd-numbered lines in the above-mentioned all V components; the other half of the above-mentioned all V components are the above-mentioned all V components The V components of the even rows in .

In another optional solution, it is also possible to keep the Y component in the YUV4:4:4 format image as the Y component, store the U component in the YUV4:4:4 format image as the Y component, and store the YUV4 : The odd columns in the V component in the 4:4 format image are stored as U components, and the even columns in the V component in the YUV4:4:4 format image are kept as V components, that is to say, all the above V Half of the components are V components of odd columns among all the V components; the other half of all V components are V components of even columns among all V components, see FIG. 6E for details.

In the embodiment of the present application, the module that re-stores the types of the Y, U, and V components in the image in the YUV4:4:4 format can be called the YUV forward conversion unit 502, and of course there can be other names.

After the Y, U, and V components in the YUV4:4:4 format image are re-stored, the original YUV4:4:4 format image has essentially become the first image that meets the encoding requirements of the target format encoder.

Method 3, the above target format is YUV4:2:0 format; you can first convert the image in YUV4:4:4 format to the transitional image in NV12 format, and then convert the transitional image in NV12 format to the first transitional image in YUV4:2:0 format an image.

Specifically, redefine the type of the fourth part of the pixel value in the image in the YUV4:4:4 format as the Y component, redefine the type of the fifth part of the pixel value in the image in the YUV4:4:4 format as the U component, and use YUV4 : The type of the sixth part of the pixel value in the 4:4 format image redefines the V component; the number of the fourth part of the pixel value accounts for 2/3 of the total number of pixel values of the YUV4:4:4 format image, and the fifth The number of partial pixel values accounts for 1/6 of the total number of pixel values of the image in the YUV4:4:4 format, and the number of the sixth part of pixel values accounts for 1/6 of the total number of pixel values of the image in the YUV4:4:4 format .

For example, as shown in Figure 6F, the type of the Y component in the image of the YUV4:4:4 format remains unchanged, and the type of the U component in the image of the YUV4:4:4 format is redefined as the Y component, that is, the above-mentioned fourth Partial pixel values include all Y components and all U components in an image in YUV4:4:4 format. Redefine the type of half of the V components in the image in the YUV4:4:4 format to the U component, that is, the fifth part of the pixel values includes half of all the V components in the image in the YUV4:4:4 format. The type of the other half of the V component in the image in the YUV4:4:4 format is still the V component, that is, the sixth part of the pixel value includes the other half of all the V components in the image in the YUV4:4:4 format, thus obtaining NV12 format transition image.

For ease of understanding, the distribution of pixel values in the image is shown in more detail below in conjunction with Figure 6G. Optionally, as shown in Figure 6G, the Y component in the image in the YUV4:4:4 format remains as the Y component , redefining the U component in the YUV4:4:4 format image as the Y component, redefining the odd column in the V component in the YUV4:4:4 format image as the U component, and redefining the YUV4:4:4 format Even columns in the V component of the image remain as V components. That is to say, half of the above-mentioned all V components are the V components of the odd-numbered columns in the above-mentioned all V components; the other half of the above-mentioned all V components are the V components of the even-numbered columns in the above-mentioned all V components, as shown in Figure 6G ; After processing in this way, an image in NV12 format can be obtained.

In the embodiment of the present application, the process of redefining the pixel value is the process of re-dividing or marking the pixel value, for example, marking the original V component with the marking information of U, so that the V component is redefined as a U component. For ease of description, this application refers to the image in NV12 format after redefining the image in YUV4:4:4 format as a transitional image.

In the embodiment of the present application, the module that redefines the types of Y, U, and V components in the YUV4:4:4 format image may be called the YUV component forward conversion unit 502, and of course there may be other names.

After obtaining the transitional image in NV12 format, convert the transitional image into the first image in YUV4:2:0 format, and there are many ways to convert the transitional image into the first image in YUV4:2:0 format, for example, you can convert the NV12 format All or part of the pixel values in the transition image are re-stored to obtain the first image in YUV4:2:0 format. Usually, 2/3 of the pixel values in the transition image are converted to Y in the first image component, 1/6 of the pixel values in the transition image will be converted to the U component in the first image, and 1/6 of the pixel values in the transition image will be converted to the V component in the first image; The pixel values of all come from the image in YUV4:4:4 format. Therefore, it can be considered that there is an indirect relationship as follows: in the image in YUV4:4:4 format, a fourth part of the pixel values is finally converted into the Y component (the fourth part Part of the pixel value is first converted to the fourth part of the pixel value in the transition image, and the fourth part of the pixel value in the transition image is then converted to the Y component in the first image), the image in the YUV4:4:4 format has the first Five parts of pixel values are finally converted into U components (the fifth part of pixel values are first converted into the fifth part of pixel values in the transition image, and the fifth part of pixel values in the transition image are then converted into U components in the first image ), the sixth part of the pixel value in the YUV4:4:4 format image is finally converted to the V component (the sixth part of the pixel value is first converted into the sixth part of the pixel value in the transition image, and the sixth part of the pixel value in the transition image Six parts of pixel values are then converted into V components in the first image), the fourth part of pixel values accounts for 2/3 of the total number of pixel values of the image in the YUV4:4:4 format, and the fifth part of pixel values The number accounts for 1/6 of the total number of pixel values of the image in the YUV4:4:4 format, and the number of the sixth part of pixel values accounts for 1/6 of the total number of pixel values of the image in the YUV4:4:4 format.

Optionally, in the process of re-storing the pixel values in the transition image in NV12 format to obtain the first image in YUV4:2:0 format, only some of the pixel values may be stored, for example, in the transition image in NV12 format The Y component is still used as the Y component in the first image in the YUV4:2:0 format, so there is no need to store it again; while part of the U component in the transition image in the NV12 format needs to be stored in the first image in the YUV4:2:0 format Part of the V component in the transition image in the NV12 format needs to be stored as the U component in the first image in the YUV4:2:0 format, see FIG. 6G for details.

In the embodiment of this application, according to the ratio relationship of Y component, U component and V component in the image of YUV4:4:4 format, and the Y component in the image of the target format (such as YUV4:2:0, or NV12, or NV21) It can be seen from the proportioning relationship of component, U component and V classification that, under the condition that Y component, U component and V component are not lost in the conversion process, the resolution of the first image in the converted target format is YUV4:4 before conversion: 4 format images with twice the resolution. Optionally, if compared from the perspective of length or width, if the length of the first image is equal to the image in YUV4:4:4 format before conversion, then the width of the first image is in YUV4:4:4 format before conversion twice the width of the image; if the width of the first image is equal to that of the image in YUV4:4:4 format before conversion, then the length of the first image is twice the length of the image in YUV4:4:4 format before conversion .

Step S603: The video encoding device encodes the first video to obtain a first code stream.

Optionally, since the first image obtained above is in the target format, here the first image is encoded by an encoder of the target format.

Step S604: the video encoding device sends the first code stream to the video decoding device.

Specifically, the transmission may be performed through the output interface 140 shown in FIG. 4 , and the specific principle will not be repeated here.

It should be noted that step S604 is an optional step, that is, after the video encoding device obtains the first stream through encoding, it may not send it to the decoding video decoding device, but store it in itself, or output it through other means.

Step S605: The video decoding device acquires the code stream or indication information sent by the video encoding device.

Specifically, the reception may be performed through the input interface 240 shown in FIG. 4 , and the specific principle will not be repeated here.

It should be noted that the manner of acquiring the code stream or indication information here may specifically be receiving, for example, the previous video encoding device sends the code stream or indication information, and correspondingly, the video decoding device receives the code stream or indication information.

The code stream may be the above-mentioned first code stream or the above-mentioned second code stream, and the specific code stream depends on what type of code stream is sent by the video encoding device.

If the first code stream is received (as known from the foregoing, the first code stream is a code stream including data in the target format), step S606 is executed.

If the second code stream is received, step S608 is executed.

If indication information is received, step S609 is performed.

Step S606: The video decoding device parses the first code stream to obtain the first image.

Optionally, since the above-mentioned first image is encoded by an encoder of the target format at the encoding end, the above-mentioned first image may be decoded from the first code stream by a decoder of the target format.

For example, if the above-mentioned first image is encoded by a YUV4:2:0 format encoder at the encoding end, the above-mentioned first image can be decoded from the first code stream by a YUV4:2:0 format decoder . For another example, if the above-mentioned first image is encoded by an NV12-format encoder at the encoding end, the above-mentioned first image can be decoded from the first code stream by an NV12-format decoder.

Step S607: The video decoding device determines an image in YUV4:4:4 format according to the first image.

Wherein, the first part of pixel values in the first image is used to convert to the Y component of the image in the YUV4:4:4 format, and the second part of the pixel values in the first image is used to convert to the YUV4:4:4 format The U component of the image, the third part of the pixel values in the second image is used to convert to the V component of the image in the YUV4:4:4 format; the number of the first part of pixel values, the number of the second part of pixel values and the The number of pixel values in the third part each accounts for 1/3 of the total number of pixel values in the first image.

Optionally, determining the image in the YUV4:4:4 format according to the first image may be completed by a YUV4:4:4 inverse conversion unit.

The following are examples of several optional ways of determining an image in YUV4:4:4 format according to the first image:

Method 4 is the reverse operation of the above method 1. In the decoding protocol, the decoding rules of the decoder of the target format are stipulated, so that after the image of the target format is decoded by the decoder of the target format, the subsequent processing process will use the image of the target format Treated as an image in YUV4:4:4 format. Specifically, the first part of the pixel values in the image of the target format is regarded as the Y component, the second part of the pixel values in the image of the target format is regarded as the U component, and the third part of the pixel values in the image of the target format is regarded as V component; wherein, the number of the first part of pixel values, the number of the second part of pixel values and the number of the third part of pixel values each account for 1/3 of the total number of pixel values of the image in the target format.

For example, half of all Y components in the first image in the target format are still regarded as Y components; the other half of all Y components in the first image in the target format are regarded as U components; All U components and all V components are regarded as V components. Optionally, all U components in the first image are specifically regarded as odd line V components of an image in YUV4:4:4 format, and all V components in the first image are specifically regarded as images in YUV4:4:4 format The even-numbered row V components.

Method 5 is the reverse operation of the above method 2, and re-stores the Y, U, and V components (all or part) of the first image in the target format to obtain an image in YUV4:4:4 format. The re-storage process does not discard the original Y component, U component, V component, nor change the content of the Y component, U component, and V component. The pixel value stored in the Y component position is equivalent to redefining the Y component and storing it in the The pixel value at the position of the U component is equivalent to being redefined as the U component, and the pixel value stored at the position of the V component is equivalent to being redefined as the V component.

Specifically, store the first part of pixel values in the first image in the target format as the Y component, store the second part of the pixel values in the first image in the target format as the U component, and store the third part of the pixel values in the first image in the target format The value is stored as a V component; the first part of the pixel value, the second part of the pixel value and the third part of the pixel value each account for 1/3 of the pixel value of the image in the target format, thereby obtaining an image in the YUV4:4:4 format .

For example, as shown in Figure 6H, the type of half of all Y components in the first image of the target format remains unchanged, still being the Y component; the other half of all the Y components in the first image of the target format is stored as U component; All U components in the first image of the target format are stored as V components; All V components in the first image of the target format are stored as V components, and Fig. 6H stores the image of the target format obtained as YUV4: An image in 2:0 format is used as an example for illustration.

In order to facilitate understanding, the position distribution of pixel values in the image is shown in more detail below in conjunction with FIG. 6I. Optionally, as shown in FIG. 6I, the type of half of all Y components in the first image in the target format Keep unchanged, it is still Y component; store the other half of all Y components in the first image of the target format as U components; store all U components in the first image of the target format as odd-numbered row V components; store the target All the V components in the first image of the format are stored as the V components of the even lines, that is, all the U components in the above-mentioned first image are used to store the V components of the odd lines of the image in the YUV4:4:4 format, All the V components in the first image are used to store the V components of the even lines of the image in the YUV4:4:4 format.

In another optional solution, half of the types of all Y components in the first image of the target format can also be kept unchanged, still being Y components; the types of all Y components in the first image of the target format The other half is stored as the U component; store all the U components in the first image in the target format as odd column V components; store all the V components in the first image in the target format as V components in even columns, that is to say , all the U components in the above first image are used to store the odd column V components of the image in the YUV4:4:4 format, and all the V components in the first image are used to store the image in the YUV4:4:4 format Even column V components, as shown in Figure 6J.

In the embodiment of the present application, the module for re-storing the types of the Y, U, and V components in the image in the target format may be called the YUV inverse conversion unit 505, and of course there may be other names.

After the Y, U, and V components in the image in the target format are re-stored, the original first image in the target format has essentially become an image in the YUV4:4:4 format.

Method 6 is the reverse operation of the above method 3. The above target format is YUV4:2:0 format; you can first convert the first image in the target format to a transitional image in NV12 format, and then convert the transitional image to YUV4:4:4 format image.

Convert the first image in YUV4:2:0 format to a transitional image in NV12 format. There are many ways to convert the first image in YUV4:2:0 format to a transitional image in NV12 format. For example, you can convert the first image in the target format to All or part of the pixel values in an image are re-stored to obtain a transitional image in NV12 format. Usually, 2/3 of the pixel values in the first image are converted to the Y component in the transitional image. The first image 1/6 of the pixel values in the first image will be converted to U components in the transition image, and 1/6 of the pixel values in the first image will be converted to V components in the transition image.

Optionally, during the process of re-storing the pixel values in the first image in the YUV4:2:0 format to obtain the transition image in the NV12 format, only some of the pixel values can be stored, for example, in the YUV4:2:0 format The Y component in the first image is still used as the Y component in the transition image in NV12 format, so it does not need to be stored again; while part of the U component in the first image in YUV4:2:0 format needs to be stored as a transition image in NV12 format Part of the V component in the first image in the YUV4:2:0 format needs to be stored as the U component in the transition image in the NV12 format, as shown in FIG. 6K .

After obtaining the transition image in NV12 format, redefine the type of the first part of the pixel value in the transition image as the Y component, redefine the type of the second part of the pixel value in the transition image as the U component, and redefine the type of the third part of the pixel value in the transition image The type redefines the V component; the number of the first part of pixel values, the number of the second part of pixel values and the number of the third part of pixel values each account for 1/3 of the total number of pixel values of the transition image.

For example, as shown in Figure 6L, half of all Y components in the transition image remain unchanged, and the type of the other half of all Y components in this transition image redefines the U component, that is, the above-mentioned first part of pixel values includes Half of the Y component, the second part of pixel values includes the other half of the Y component in the transition image. Redefine the U component in the transition image as the V component of the odd column of the image in the YUV4:4:4 format, and redefine the V component in the transition image as the V component of the even column of the image in the YUV4:4:4 format, that is, the above-mentioned The three-part pixel value includes all U components and all V components in the transition image.

In the embodiment of the present application, the process of redefining the pixel value is the process of re-dividing or marking the pixel value, for example, marking the original V component with the marking information of U, so that the V component is redefined as a U component.

In the embodiment of the present application, the module that redefines the types of Y, U, and V components in the transition image may be called the YUV inverse conversion unit 505 , and of course there may be other names.

Since the pixel values in the image in the YUV4:4:4 format all come from the first image in the target format, it can be considered that there is an indirect relationship as follows: the first part of the pixel values in the first image in the target format are finally converted to Y component (the first part of the pixel value is first converted to the first part of the pixel value in the transition image, and the first part of the pixel value in the transition image is then converted to the Y component in the image in the YUV4:4:4 format), in the first image There is a second part of the pixel value that is finally converted to the U component (the second part of the pixel value is first converted to the second part of the pixel value in the transition image, and the second part of the pixel value in the transition image is then converted to YUV4:4:4 The U component in the image of format), the third part of the pixel value in the first image is finally converted to the V component (the third part of the pixel value is first converted into the third part of the pixel value in the transition image, and the transition image The third part of the pixel value is converted to the V component in the image of the YUV4:4:4 format), the number of the first part of the pixel value, the number of the second part of the pixel value and the number of the third part of the pixel value account for 1/3 of the total number of pixel values for this first image.

In the embodiment of this application, according to the ratio relationship of Y component, U component and V component in the image of YUV4:4:4 format, and the Y component in the image of the target format (such as YUV4:2:0, or NV12, or NV21) The proportioning relationship of component, U component and V classification shows that, in the case of not losing Y component, U component and V component in the conversion process, the resolution of the converted image in YUV4:4:4 format is the target format before conversion Half the resolution of the first image. Optionally, in terms of length or width, if the length of the image in YUV4:4:4 format is equal to the length of the first image before conversion, then the width of the image in YUV4:4:4 format is the first image before conversion Half the width of the image; if the image in YUV4:4:4 format is equal to the width of the first image before conversion, then the length of the image in YUV4:4:4 format is half the length of the first image before conversion.

Afterwards, the video decoding device (for example, through the above-mentioned RGB inverse conversion unit) converts the image in YUV4:4:4 format to obtain an image in RGB format, which is equivalent to restoring the image in RGB format at the encoding end at the video decoding end.

In Fig. 6E, Fig. 6G, Fig. 6J, Fig. 6L, different U components are indicated by different additional labels, for example, U11 and U13 are two different U components; different V components are indicated by different additional labels, for example , V12 and V14 are two different V components. In the embodiment of this application, in the NV12 format image and the YUV4:2:0 format image, the U with the same label is the same U component, for example, the U11 in the NV12 format image and the YUV4:2:0 format U11 in the image is the same U component. The V with the same label is the same V component, and the V12 in the NV12 format image and the V12 in the YUV4:2:0 format image are the same V component. In addition, U and V with the same label indicate that U is converted from V, or V is converted from U. For example, in FIG. 6E, V11 is used to convert to U11.

Step S608: the video decoding device parses the second code stream to obtain an image in YUV4:2:0 format.

Afterwards, the video decoding device converts the image in YUV4:2:0 format to obtain an image in RGB format, which is equivalent to restoring the image in RGB format at the encoding end at the video decoding end.

Step S609: The video decoding device parses the indication information.

According to the indication information, the video decoding device can know that the pixel values of the RGB image are the same as those of the previous frame of the RGB image, that is, the display effect of the GRB is the same as that of the previous frame of image. Therefore, when determining the pixel value of the RGB image, the video decoding device can directly multiplex the pixel value of the previous frame of the RGB image without recalculating the pixel value of the RGB image.

In the method shown in Figure 6A, the Y component in the image of YUV4:4:4 format is converted at the coding end, U component, and V component is converted to obtain the image of the target format, and then the target format (such as YUV4:2:0 ) encoder encodes the image in the target format; after the image in the target format is decoded at the decoding end, the Y component, U component, and V component in the image in the target format are converted to obtain the YUV4:4:4 format image. This method is not only compatible with the codec/decoder of the target format, but also retains the color display advantages of the YUV4:4:4 format, and also reduces the color difference and the complexity of reference frame management.

The method of the embodiment of the present application has been described in detail above, and the device of the embodiment of the present application is provided below.

Please refer to FIG. 7 . FIG. 7 is a schematic structural diagram of a video encoding device 70 provided in an embodiment of the present application. The video encoding device may be the above-mentioned source device 10 , or a device in the source device 10 , for example, a video encoder. The video coding apparatus 70 may include a determining unit 701 and a first coding unit 702, and the detailed description of each unit is as follows.

A determining unit 701, configured to determine a first image in a target format according to an image in a YUV4:4:4 format;

Wherein, the fourth part of pixel values in the YUV4:4:4 format image is used to convert to the Y component in the first image, and the fifth part of the pixel values in the YUV4:4:4 format image is used to convert to the The U component in the first image, the sixth part of the pixel value in the YUV4:4:4 format image is used to convert to the V component in the first image; the fourth part of the pixel value, the fifth part of the pixel value and The pixel values of the sixth part respectively account for 2/3, 1/6 and 1/6 of the total number of pixel values of the image in the YUV4:4:4 format; the target format includes one of YUV4:2:0, NV12 and NV21 kind;

The first encoding unit 702 is configured to encode the first image to obtain a first code stream.

Using the above method, the Y component, U component, and V component in the image of the YUV4:4:4 format are converted at the encoding end to obtain an image of the target format, and then the encoder of the target format (such as YUV4:2:0) pairs The image in the target format is encoded; after the image in the target format is decoded at the decoding end, the Y component, U component, and V component in the image in the target format are converted to obtain an image in the YUV4:4:4 format. This method is not only compatible with the codec/decoder of the target format, but also retains the color display advantages of the YUV4:4:4 format, and also reduces the color difference and the complexity of reference frame management.

In an optional solution, the determining unit 701 is specifically configured to:

The image in the YUV4:4:4 format is stored as the first image in the target format, wherein the fourth part of the pixel values in the image in the YUV4:4:4 format is used to store as the Y component in the first image, the The fifth part of the pixel values in the YUV4:4:4 format image is used to store as the U component in the first image, and the sixth part of the pixel values in the YUV4:4:4 format image is used to store as the first image The V component in .

In another optional solution, the video encoding device also includes:

A conversion unit for converting an image in RGB format into an image in this YUV4:4:4 format;

If the image in RGB format is applied to the first scene, the determining unit is used to determine the first image in the target format according to the image in YUV4:4:4 format;

If the image in RGB format is applied to the second scene, the second encoding unit is used to downsample the image in YUV4:4:4 format, and encode the downsampled image according to the encoder in YUV4:2:0 format , get the second code stream;

The sending unit is further configured to send the second code stream to the video decoder;

Wherein, the image resolution requirement of the second scene is lower than the image resolution requirement of the first scene.

In this method, for scenes with lower image resolution requirements, the traditional downsampling method is used for preprocessing and encoding, and for scenes with higher image resolution requirements, no loss of pixel values (that is, no loss of Y component, U component, and V component), which can not only meet the special requirements for chroma in scenes with high image resolution requirements to the greatest extent, but also minimize the The amount of encoded data in the scene.

In yet another optional solution, the first scene includes at least one of an office document scene, a desktop icon scene, a text scene, and a LOGO scene; the second scene includes a video scene, a game scene, and a scene of natural scenery. at least one.

In yet another alternative:

If the difference between the pixel of the RGB image and the pixel value of the previous frame of RGB image exceeds a preset threshold, the conversion unit is used to convert the image in RGB format into the image in YUV4:4:4 format;

If the difference between the pixels of the RGB image and the pixels of the previous RGB image does not exceed a preset threshold, the sending unit is used to send indication information to the video decoder, and the indication information is used to indicate that the pixels of the RGB image are different from the pixels of the RGB image. The pixels of the RGB image from the previous frame are the same.

In this way, when the pixel values of a frame of RGB image (may also be a part of RGB image) and the previous frame of RGB image (maybe a part of RGB image) have a small difference, there is no need to A frame of RGB image is re-encoded, and the previous frame of RGB image can be directly multiplexed, which can reduce the calculation pressure and improve the overall encoding efficiency.

In yet another optional solution, the first image aspect of the target format is determined according to the image in the YUV4:4:4 format, and the determining unit is specifically configured to: convert the image in the YUV4:4:4 format into an NV12 format transition image, wherein the fourth part of the pixel values in the YUV4:4:4 format image is used to convert to the Y component in the transition image, and the fifth part of the pixel values in the YUV4:4:4 format image is used for Converted to the U component in the transition image, the sixth part of the pixel value in the YUV4:4:4 format image is used to convert to the V component in the transition image, the conversion method mentioned here can be YUV4:4: 4 format image, for example, redefining (or marking) the U component as the Y component. Then store the transition image in NV12 format as the first image in YUV4:2:0 format, and store the transition image in NV12 format as the first image in YUV4:2:0 format, which is a relatively mature technology, and there are many ways to implement it. They are not listed here. In combination with the first aspect, or any of the above optional solutions of the first aspect, in another optional solution: the fourth part of pixel values includes all Y components in the image in the YUV4:4:4 format and all U components; the fifth part of the pixel value includes half of all the V components in the image of the YUV4:4:4 format; the sixth part of the pixel value includes another part of all the V components in the image of the YUV4:4:4 format half.

Wherein, half of the all V components are V components of odd columns among all the V components, and the other half of all V components are V components of even columns among all V components.

Alternatively, half of the total V components are V components of odd lines in the total V components, and the other half of the total V components are V components of even lines in the total V components.

It should be noted that the implementation and beneficial effects of each unit may also refer to the corresponding description of the method embodiment shown in FIG. 6A .

Please refer to FIG. 8. FIG. 8 is a schematic structural diagram of a video encoding device 80 provided by an embodiment of the present application. The video encoding device 80 may be the above-mentioned destination device 20, or a device in the destination device 20, for example, a video decoder . The video coding apparatus may include an acquiring unit 801, an analyzing unit 802, and a determining unit 803, and the detailed description of each unit is as follows.

An acquisition unit 801, configured to acquire a first code stream including data in a target format, where the target format includes a combination of one or more of YUV4:2:0, NV12 or NV21;

An parsing unit 802, configured to parse the first code stream to obtain a first image;

A determining unit 803, configured to determine an image in YUV4:4:4 format according to the first image; wherein, the first part of pixel values in the first image is used to convert to the Y component of the image in YUV4:4:4 format, the The second part of the pixel values in the first image is used to convert to the U component of the image in the YUV4:4:4 format, and the third part of the pixel values in the first image is used to convert to the U component of the image in the YUV4:4:4 format V component; the first part of pixel values, the second part of pixel values and the third part of pixel values each account for 1/3 of the total number of pixel values of the first image.

Using the above method, the Y component, U component, and V component in the image of the YUV4:4:4 format are converted at the encoding end to obtain an image of the target format, and then the encoder of the target format (such as YUV4:2:0) pairs The image in the target format is encoded; after the image in the target format is decoded at the decoding end, the Y component, U component, and V component in the image in the target format are converted to obtain an image in the YUV4:4:4 format. This method is not only compatible with the codec/decoder of the target format, but also retains the color display advantages of the YUV4:4:4 format, and also reduces the color difference and the complexity of reference frame management.

In an optional solution, the determining unit 803 is specifically configured to:

The image in the YUV4:4:4 format is stored as a first image in the YUV4:2:0 format, and the first part of the pixel values in the first image is used to store the Y component of the image in the YUV4:4:4 format, the The second part of the pixel values in the first image is used to store as the U component of the image in the YUV4:4:4 format, and the third part of the pixel values in the second image is used to store as the U component of the image in the YUV4:4:4 format V component.

In another optional solution, the first image is an image in YUV4:2:0 format, and determining the image in YUV4:4:4 format according to the first image includes: converting the image in YUV4:2:0 format The image is stored as a transitional image in NV12 format, the first part of the pixel value in the YUV4:2:0 format image is used to store the Y component of the transitional image, and the second part of the pixel value in the YUV4:2:0 format image is used Stored as the U component of the transition image, the third part of the pixel values in the YUV4:2:0 format image is used to store as the V component of the transition image, and the first image in the YUV4:2:0 format is stored as NV12 The transition image format is a relatively mature technology, and there are many implementation methods, which are not listed here. Then, convert the transitional image into an image in YUV4:4:4 format. The conversion method mentioned here can be to redefine the pixel values in the transitional image in NV12 format, for example, redefining part of the Y component (or marking ) is the U component.

In yet another alternative,

The first partial pixel value includes half of all Y components in the first image;

The second portion of pixel values includes the other half of all Y components in the first image;

The third part of pixel values includes all U components and all V components in the first image;

Wherein, all the U components in the first image are used as the odd row V components of the image in the YUV4:4:4 format, and all the V components in the first image are used as the image in the YUV4:4:4 format The even-numbered row V component of ;

Or, all the U components in the first image are used as the odd column V components of the image in the YUV4:4:4 format, and all the V components in the first image are used as the image in the YUV4:4:4 format The even-numbered columns of V components.

It should be noted that the implementation and beneficial effects of each unit may also refer to the corresponding description of the method embodiment shown in FIG. 6A .

FIG. 9 is a schematic block diagram of an implementation manner of an electronic device 900 according to an embodiment of the present application. The electronic device may be an encoding device or a decoding device. Wherein, the electronic device may include a processor 910 , a memory 930 and a bus system 950 . Wherein, the processor and the memory are connected through a bus system, the memory is used to store computer program codes, and the processor is used to execute the computer program codes stored in the memory. For example, the memory of the encoding device stores computer program codes, and the processor of the encoding device invokes the computer program codes in the memory to implement the steps implemented by the video encoding device in the method embodiment shown in FIG. 6A. For another example, the memory of the decoding device stores computer program codes, and the processor of the decoding device invokes the computer program codes in the memory to implement the steps implemented by the video decoding device in the method embodiment shown in FIG. 6A .

In the embodiment of the present application, the processor 910 may be a central processing unit (Central Processing Unit, referred to as "CPU"), and the processor 910 may also be other general processors, digital signal processors (DSP), application specific integrated circuits ( ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The memory 930 may include a read only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may also be used as memory 930 . Memory 930 may include code and data 931 accessed by processor 910 using bus 950 . The memory 930 may further include an operating system 933 and an application program 935, and the application program 935 includes allowing the processor 910 to implement the steps implemented by the video decoding device in the method embodiment shown in Figure 6A, or to implement the method embodiment shown in Figure 6A The steps implemented by the video encoding device. For example, the application programs 935 may include applications 1 to N, which further include applications for performing the method shown in FIG. 6A.

In addition to the data bus, the bus system 950 may also include a power bus, a control bus, and a status signal bus. However, for clarity of illustration, the various buses are labeled as bus system 950 in the figure.

Optionally, the electronic device 900 may also include one or more output devices, such as a display 970 . In one example, the display 970 may be a touch-sensitive display that incorporates a display with a tactile unit operable to sense touch input. The display 970 may be connected to the processor 910 via the bus 950 .

Optionally, the above-mentioned encoding device may be the source device 10 shown in FIG. 4 , and the above-mentioned decoding device may be the destination device 20 shown in FIG. 4 .

The embodiment of the present application also provides a chip system, the chip system includes at least one processor, a memory and an interface circuit, the memory, the interface circuit and the at least one processor are interconnected by a line, and the at least one memory A computer program is stored in the computer program; when the computer program is executed by the processor, the video encoding device realizes the corresponding operations mentioned in the method flow shown in FIG. 6A , or the video decoding device realizes the method shown in FIG. 6A The corresponding action mentioned in the process.

The embodiment of the present application also provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is run on a processor, the video encoding device implements the steps in the method flow shown in FIG. 6A . The corresponding operations mentioned, or make the video decoding device implement the corresponding operations mentioned in the method flow shown in FIG. 6A .

The embodiment of the present application also provides a computer program product. When the computer program product is run on a processor, the video encoding device is enabled to implement the corresponding operations mentioned in the method flow shown in FIG. 6A , or the video decoding device is enabled to The corresponding operations mentioned in the method flow shown in FIG. 6A are implemented.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments are realized. The processes can be completed by computer programs or hardware related to the computer programs. The computer programs can be stored in computer-readable storage media. The computer programs During execution, it may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: ROM or random access memory RAM, magnetic disk or optical disk, and other various media that can store computer program codes.

Claims (16)

1. A video decoding method, comprising:

acquiring a first code stream comprising target format data, wherein the target format comprises a combination of one or more of YUV4: 2;

analyzing the first code stream to obtain a first image;

determining an image in YUV4:4 format according to the first image; wherein a first portion of pixel values in the first image is used for conversion to the Y component of the image in YUV 4; the first part of pixel values, the second part of pixel values and the third part of pixel values respectively account for 1/3 of the total number of the pixel values of the first image;

the first portion of pixel values comprises half of all Y components in the first image;

the second portion of pixel values comprises the other half of all Y components in the first image;

the third portion of pixel values includes all U components and all V components in the first image;

wherein all U components in the first image are used as odd-line V components of the image in YUV 4;

alternatively, all U components in the first image are used as odd column V components of the image of the YUV 4.

2. The method of claim 1, wherein determining an image in YUV4:4 format from the first image comprises:

storing a first image as an image in YUV4:4 format, a first portion of pixel values in the first image for storing as a Y component of the image in YUV 4.

3. A video encoding method, comprising:

determining a first image in a target format according to the image in YUV 4;

wherein a fourth portion of pixel values in the image in YUV4:4 format is used for conversion to a Y component in the first image, a fifth portion of pixel values in the image in YUV 4; the fourth part of pixel values, the fifth part of pixel values and the sixth part of pixel values respectively account for 2/3, 1/6 and 1/6 of the total number of pixel values of the image in YUV 4; the target format comprises one of YUV4:2, NV12, and NV 21;

coding the first image to obtain a first code stream;

the fourth portion of pixel values includes all Y components and all U components in the image in YUV 4;

the fifth part of pixel values comprises half of all V components in the image in YUV 4;

the sixth portion of pixel values comprises the other half of all V components in the image in YUV4:4 format;

wherein half of the total V components are V components of odd columns in the total V components, and the other half of the total V components are V components of even columns in the total V components;

or, half of all the V components are V components of odd lines in all the V components, and the other half of all the V components are V components of even lines in all the V components.

4. The method of claim 3, wherein determining the first image in the target format from the images in YUV4:

storing the image in the YUV 4.

5. The method according to claim 3 or 4, wherein before determining the first image in the target format from the images in YUV4:

converting the image in RGB format into the image in YUV 4;

if the image in the RGB format is applied to a first scene, determining a first image in a target format according to the image in the YUV 4;

if the image in the RGB format is applied to a second scene, downsampling the image in the YUV4:4 format, and coding the downsampled image according to an encoder in the YUV4: 2;

sending the second code stream to a video decoder;

wherein the requirement of the image resolution of the second scene is lower than the requirement of the image resolution of the first scene.

6. The method of claim 5, wherein the first scene comprises at least one of an office document scene, a desktop icon scene, a text scene, a LOGO scene; the second scene comprises at least one of a video scene, a game scene and a nature scene.

7. The method of claim 5, further comprising:

if the difference between the pixel value of the RGB format image and the pixel value of the previous frame RGB format image exceeds a preset threshold value, converting the RGB format image into the YUV4:4 format image;

if the difference between the pixel of the image in the RGB format and the pixel of the image in the previous frame of RGB format does not exceed a preset threshold value, sending indication information to the decoder, wherein the indication information is used for indicating that the pixel of the image in the RGB format is the same as the pixel of the image in the previous frame of RGB format.

8. A video decoding apparatus, comprising:

the code stream processing device comprises an obtaining unit, a decoding unit and a decoding unit, wherein the obtaining unit is used for obtaining a first code stream comprising target format data, and the target format comprises one or more combinations of YUV4: 2;

the analysis unit is used for analyzing the first code stream to obtain a first image;

a determining unit, configured to determine an image in YUV4:4 format according to the first image; wherein a first portion of pixel values in the first image is used for conversion to the Y component of the image in YUV 4; the first part of pixel values, the second part of pixel values and the third part of pixel values respectively account for 1/3 of the total number of the pixel values of the first image;

the first portion of pixel values comprises half of all Y components in the first image;

the second portion of pixel values comprises the other half of all Y components in the first image;

the third portion of pixel values comprises all U components and all V components in the first image;

wherein all U components in the first image are used as odd-line V components of the YUV 4;

alternatively, all U components in the first image are used for odd column V components as an image of the YUV 4.

9. The apparatus according to claim 8, wherein the determining unit is specifically configured to:

storing a first image as an image in YUV 4.

10. A video encoding apparatus, comprising:

a determining unit, configured to determine a first image in a target format according to an image in YUV 4;

wherein a fourth portion of pixel values in the image in YUV4:4 format is used for conversion to a Y component in the first image, a fifth portion of pixel values in the image in YUV 4; the fourth part pixel value, the fifth part pixel value and the sixth part pixel value respectively account for 2/3, 1/6 and 1/6 of the total number of pixel values of the image in YUV4:4 format; the target format comprises one of YUV4:2, NV12, and NV 21;

the first coding unit is used for coding the first image to obtain a first code stream;

the fourth portion of pixel values includes all Y components and all U components in the image in YUV 4;

the fifth part of pixel values comprises half of all V components in the image in YUV 4;

the sixth portion of pixel values comprises the other half of all V components in the image in YUV 4;

wherein half of the total V components are V components of odd columns in the total V components, and the other half of the total V components are V components of even columns in the total V components;

or, half of all the V components are V components of odd-numbered lines in all the V components, and the other half of all the V components are V components of even-numbered lines in all the V components.

11. The apparatus according to claim 10, wherein the determining unit is specifically configured to:

storing the image in YUV4:4 format as a first image in a target format, wherein a fourth portion of pixel values in the image in YUV 4.

12. The apparatus according to claim 10 or 11, wherein the video encoding apparatus further comprises:

a conversion unit configured to convert an image in an RGB format into an image in the YUV 4;

if the image in the RGB format is applied to a first scene, the determining unit is used for determining a first image in a target format according to the image in the YUV 4;

if the image in the RGB format is applied to a second scene, the second coding unit is used for down-sampling the image in the YUV4:4 format, and coding the down-sampled image according to an encoder in the YUV4: 2;

a sending unit, configured to send the second code stream to a video decoder;

wherein the requirement of the image resolution of the second scene is lower than the requirement of the image resolution of the first scene.

13. The apparatus of claim 12, wherein the first scene comprises at least one of an office document scene, a desktop icon scene, a text scene, and a LOGO scene; the second scene comprises at least one of a video scene, a game scene and a nature scene.

14. The apparatus of claim 12, wherein:

if the difference between the pixel value of the image in the RGB format and the pixel value of the image in the previous frame of RGB format exceeds a preset threshold value, the conversion unit is used for converting the image in the RGB format into the image in the YUV 4;

if the difference between the pixel of the image in the RGB format and the pixel of the image in the previous frame of RGB format does not exceed the preset threshold, the sending unit is configured to send indication information to the video decoder, where the indication information is used to indicate that the pixel of the image in the RGB format is the same as the pixel of the image in the previous frame of RGB format.

15. An electronic device, characterized in that the electronic device comprises a processor and a memory, the processor and the memory being coupled, the memory being configured to store computer program code comprising computer software instructions that, when executed by the electronic device, cause the electronic device to perform the video decoding method of any of claims 1 to 2 or the video encoding method of any of claims 3 to 7.

16. A computer-readable storage medium, comprising: computer software instructions;

the computer software instructions, when run in an electronic device, cause the electronic device to perform the video decoding method of any one of claims 1 to 2, or the video encoding method of any one of claims 3 to 7.

Images