CN120731599A - Method and apparatus for intra-frame template matching prediction - Google Patents

Abstract

Methods and apparatuses for video decoding and encoding, and non-transitory computer-readable storage media thereof, are provided. In one video decoding method, a decoder may obtain multiple reference blocks for a current block based on template matching. Furthermore, the decoder may obtain a fused reference block based on the multiple reference blocks. Furthermore, the decoder may obtain a final prediction block for the current block based on the fused reference block and a linear filter.

Description

Method and apparatus for intra-frame template matching prediction

Cross Reference to Related Applications

The present application was filed and claims priority from U.S. provisional application serial No. 63/447,040 entitled "method and apparatus for intra-template matching prediction," filed on month 20 of 2023, the entire contents of which are incorporated herein by reference.

Technical Field

The present disclosure relates to video coding and compression, and in particular, but not exclusively, to methods and apparatus for improving coding efficiency of intra template matching prediction modes.

Background

Various electronic devices (e.g., digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc.) support digital video. The electronic device sends and receives or otherwise communicates digital video data over a communication network and/or stores the digital video data on a storage device. Because of the limited bandwidth capacity of the communication network and the limited memory resources of the storage device, video data may be compressed using video codec according to one or more video codec standards before it is transmitted or stored. For example, video coding standards include general video coding (VVC), joint exploration test model (JEM), high efficiency video coding (HEVC/h.265), advanced video coding (AVC/h.264), moving Picture Experts Group (MPEG) coding, and so forth. Video coding typically employs prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit redundancy inherent in video data. Video codec aims at compressing video data into a form using a lower bit rate while avoiding or minimizing degradation of video quality.

Disclosure of Invention

The present disclosure provides examples of techniques related to improving the codec efficiency of an intra Template Matching Prediction (TMP) method in a video encoding or decoding process.

According to a first aspect of the present disclosure, a video decoding method is provided. In the method, a decoder may obtain a plurality of reference blocks for a current block based on template matching, obtain a fused reference block based on the plurality of reference blocks, and obtain a final prediction block of the current block based on the fused reference block and a linear filter.

According to a second aspect of the present disclosure, a video encoding method is provided. In the method, an encoder may obtain a plurality of reference blocks for a current block based on template matching, obtain a fused reference block based on the plurality of reference blocks, and obtain a final prediction block of the current block based on the fused reference block and a linear filter

According to a third aspect of the present disclosure, a video decoding method is provided. In the method, a decoder may obtain at least one of a luma template of a luma Coded Block (CB) for template matching or a chroma template of a chroma CB, wherein the chroma templates include a first chroma template of a first chroma CB and a second chroma template of a second chroma CB. Further, the decoder may calculate a template matching cost between a current template of the current CB and at least one of the luma template, the first chroma template or the second chroma template, and obtain a final prediction block of the current CB based on the template matching cost.

According to a fourth aspect of the present disclosure, a video encoding method is provided. In the method, the encoder may obtain at least one of a luminance template of a luminance CB or a chrominance template of a chrominance CB for template matching, wherein the chrominance templates include a first chrominance template of a first chrominance CB and a second chrominance template of a second chrominance CB. Further, the encoder may calculate a template matching cost between a current template of the current CB and at least one of the luma template, the first chroma template or the second chroma template, and obtain a final prediction block of the current CB based on the template matching cost.

According to a fifth aspect of the present disclosure, a video decoding method is provided. In the method, a decoder may obtain a reference block of a current block and a reference template of the reference block based on intra template matching, wherein each of the current block and the reference block includes a luminance component and a chrominance component, and the reference template of the reference block includes a reference chrominance template corresponding to the chrominance component of the reference block and a reference luminance template corresponding to a co-located luminance block. Furthermore, the decoder may derive parameters of a cross-component prediction model based on a chroma template of a chroma component of the current block, the reference chroma template, and a reference luma template of a co-located luma block. Furthermore, the decoder may obtain a final prediction of the current chroma block by applying parameters of the cross-component prediction model to the co-located luma block.

According to a sixth aspect of the present disclosure, a video encoding method is provided. In the method, an encoder may obtain a reference block of a current block and a reference template of the reference block based on intra template matching, wherein each of the current block and the reference block includes a luminance component and a chrominance component, and the reference template of the reference block includes a reference chrominance template corresponding to the chrominance component of the reference block and a reference luminance template corresponding to a co-located luminance block. Furthermore, the encoder may derive parameters of a cross-component prediction model based on a chroma template of a chroma component of the current block, the reference chroma template, and a reference luma template of a co-located luma block. Furthermore, the encoder may obtain a final prediction of the current chroma block by applying parameters of the cross-component prediction model to the co-located luma block.

According to a seventh aspect of the present disclosure, a video decoding method is provided. In the method, a decoder may generate weighted prediction by weighting an intra prediction block and an intra Template Matching Prediction (TMP) block, obtain coefficients of a linear filter from a plurality of templates including a current template of a current block, an intra prediction template, and a reference template of a reference block, and obtain final prediction by applying the coefficients of the linear filter to the weighted prediction.

According to an eighth aspect of the present disclosure, a video encoding method is provided. In the method, an encoder may generate weighted prediction by weighting an intra prediction block and an intra TMP block, obtain coefficients of a linear filter from a plurality of templates including a current template of a current block, an intra prediction template, and a reference template of a reference block, and obtain final prediction by applying the coefficients of the linear filter to the weighted prediction.

According to a ninth aspect of the present disclosure, there is provided a video decoding apparatus. The apparatus may include one or more processors and a memory coupled to the one or more processors, the memory configured to store instructions executable by the one or more processors. Further, the one or more processors are configured, when executing the instructions, to perform the method according to the first, third, fifth or seventh aspect described above.

According to a tenth aspect of the present disclosure, there is provided a video encoding apparatus. The apparatus may include one or more processors and a memory coupled to the one or more processors, the memory configured to store instructions executable by the one or more processors. Further, the one or more processors are configured, when executing the instructions, to perform the method according to the second, fourth, sixth or eighth aspect described above.

According to an eleventh aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer-executable instructions which, when executed by one or more computer processors, cause the one or more computer processors to perform a method according to the first, third, fifth or seventh aspects described above.

According to a twelfth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer-executable instructions which, when executed by one or more computer processors, cause the one or more computer processors to perform a method according to the second, fourth, sixth or eighth aspect described above.

According to a thirteenth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing a bitstream to be decoded by a method as described above in the first, third, fifth or seventh aspect.

According to a fourteenth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing a bitstream generated by a method as described above in the second, fourth, sixth or eighth aspect.

Drawings

A more particular description of examples of the disclosure will be rendered by reference to specific examples that are illustrated in the appended drawings. Since these drawings depict only some examples, they are not therefore to be considered limiting of scope, the examples will be described and explained with additional specificity and detail through the use of the accompanying drawings.

Fig. 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks according to some embodiments of the present disclosure.

Fig. 2 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

Fig. 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

Fig. 4A-4E are block diagrams illustrating how frames are recursively partitioned into multiple video blocks of different sizes and shapes according to some embodiments of the present disclosure.

FIG. 5 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure.

Fig. 6 illustrates a schematic diagram of an intra-template matching search region used in accordance with some examples of the present disclosure.

Fig. 7 illustrates the use of intra TMP block vectors for IBC blocks according to some examples of the present disclosure.

Fig. 8 illustrates spatial locations for a linear filter model according to some examples of the present disclosure.

Fig. 9 illustrates reference regions for deriving filter coefficients for a linear filter model according to some examples of the present disclosure.

Fig. 10 illustrates a schematic diagram of the locations of left and above samples of a CU involved in cross-component linear model (CCLM) prediction, according to some examples of the present disclosure.

Fig. 11 illustrates a reference region (and its padding) for deriving CCCM filter coefficients according to some examples of the present disclosure.

Fig. 12 illustrates four Sobel (Sobel) -based gradient modes for GLM according to some examples of the present disclosure.

Fig. 13 illustrates a schematic diagram of half-pixel interpolation and template matching search according to some examples of the present disclosure.

Fig. 14 is a flowchart illustrating a video decoding method according to some examples of the present disclosure.

Fig. 15 is a flowchart illustrating a video encoding method corresponding to the video decoding method shown in fig. 14 according to some examples of the present disclosure.

Fig. 16 is a flowchart illustrating a video decoding method according to some examples of the present disclosure.

Fig. 17 is a flowchart illustrating a video encoding method corresponding to the video decoding method shown in fig. 16, according to some examples of the present disclosure.

Fig. 18 is a flowchart illustrating a video decoding method according to some examples of the present disclosure.

Fig. 19 is a flowchart illustrating a video encoding method corresponding to the video decoding method shown in fig. 18, according to some examples of the present disclosure.

Fig. 20 is a flowchart illustrating a video decoding method according to some examples of the present disclosure.

Fig. 21 is a flowchart illustrating a video encoding method corresponding to the video decoding method shown in fig. 20 according to some examples of the present disclosure.

Detailed Description

Reference will now be made in detail to the present embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth in order to provide an understanding of the subject matter presented herein. However, various alternatives may be used and the subject matter may be practiced without these specific details without departing from the scope of the claims. For example, the subject matter presented herein may be implemented on many classes of electronic devices having digital video capabilities.

The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. The singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise throughout the disclosure and the appended claims. It should also be understood that the term "and/or" as used in this disclosure refers to and includes one or any or all of the possible combinations of the various related items listed.

Reference throughout this specification to "one embodiment," "an example," "some embodiments," "some examples," or similar language means that a particular feature, structure, or characteristic described is included in at least one embodiment or example. Features, structures, elements, or characteristics described in connection with one or some embodiments may be applicable to other embodiments unless explicitly stated otherwise.

Throughout this disclosure, "first," "second," "third," etc. are used as nomenclature to refer only to relevant elements, e.g., devices, components, compositions, steps, etc., and do not imply any spatial or temporal order unless explicitly stated otherwise. For example, a "first device" and a "second device" may refer to two separately formed devices, or two portions, components, or operational states of the same device, and may be arbitrarily named.

The terms "module," "sub-module," "circuit," "sub-circuit," "unit," or "subunit" may include a memory (shared, dedicated, or group) that stores code or instructions that may be executed by one or more processors. A module may include one or more circuits with or without stored code or instructions. A module or circuit may include one or more components connected directly or indirectly. These components may or may not be physically attached to each other or adjacent to each other.

As used herein, the term "if" or "when" is understood to mean "at" or "responsive to" depending on the context. These terms, if present in the claims, may not indicate that the relevant limitations or features are conditional or optional. For example, a method may include the steps of i) performing a function or action X 'when or if condition X exists, and ii) performing a function or action Y' when or if condition Y exists. The method may have both the ability to perform a function or action X 'and the ability to perform a function or action Y'. Thus, functions X 'and Y' may be performed at different times in multiple executions of the method.

The units or modules may be implemented in pure software, in pure hardware, or in a combination of hardware and software. For example, in a software-only implementation, a unit or module may include functionally related code blocks or software components that are directly or indirectly linked together to perform a particular function.

Fig. 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. As shown in fig. 1, the system 10 includes a source device 12, the source device 12 generating and encoding video data to be later decoded by a target device 14. Source device 12 and destination device 14 may comprise any of a wide variety of electronic devices including cloud servers, server computers, desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video gaming machines, video streaming devices, and the like. In some implementations, the source device 12 and the target device 14 are equipped with wireless communication capabilities.

In some implementations, the target device 14 may receive encoded video data to be decoded via the link 16. Link 16 may comprise any type of communication medium or device capable of moving encoded video data from source device 12 to destination device 14. In one example, link 16 may include a communication medium that enables source device 12 to transmit encoded video data directly to destination device 14 in real-time. The encoded video data may be modulated according to a communication standard (e.g., a wireless communication protocol) and transmitted to the target device 14. The communication medium may include any wireless or wired communication medium such as a Radio Frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the internet. The communication medium may include routers, switches, base stations, or any other means that may be advantageous to facilitate communication from source device 12 to destination device 14.

In other embodiments, encoded video data may be sent from output interface 22 to storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28. Storage device 32 may include any of a variety of distributed or locally accessed data storage media such as a hard drive, blu-ray disc, digital Versatile Disc (DVD), compact disc read only memory (CD-ROM), flash memory, volatile or nonvolatile memory, or any other suitable digital storage media for storing encoded video data. In another example, storage device 32 may correspond to a file server or another intermediate storage device that may hold encoded video data generated by source device 12. The target device 14 may access the stored video data from the storage device 32 via streaming or download. The file server may be any type of computer capable of storing and transmitting encoded video data to the target device 14. Exemplary file servers include web servers (e.g., for web sites), file Transfer Protocol (FTP) servers, network Attached Storage (NAS) devices, or local disk drives. The target device 14 may access the encoded video data through any standard data connection suitable for accessing encoded video data stored on a file server, including a wireless channel (e.g., a wireless fidelity (Wi-Fi) connection), a wired connection (e.g., digital Subscriber Line (DSL), cable modem, etc.), or a combination of both a wireless channel and a wired connection. The transmission of encoded video data from storage device 32 may be streaming, download, or a combination of both streaming and download.

As shown in fig. 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as a video capture device (e.g., a video camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and/or a computer graphics system for generating computer graphics data as source video, or a combination of such sources. As one example, if video source 18 is a video camera of a security monitoring system, source device 12 and target device 14 may form a camera phone or video phone. However, the embodiments described in this disclosure may be generally applicable to video codecs and may be applied to wireless and/or wired applications.

Captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to the target device 14 via the output interface 22 of the source device 12. The encoded video data may also (or alternatively) be stored on the storage device 32 for later access by the target device 14 or other device for decoding and/or playback. Output interface 22 may further include a modem and/or a transmitter.

The target device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and/or modem and receives encoded video data over link 16. The encoded video data transmitted over link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included in encoded video data transmitted over a communication medium, stored on a storage medium, or stored on a file server.

In some implementations, the target device 14 may include a display device 34, and the display device 34 may be an integrated display device and an external display device configured to communicate with the target device 14. Display device 34 displays decoded video data to a user and may comprise any of a variety of display devices, such as a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or another type of display device.

Video encoder 20 and video decoder 30 may operate in accordance with a proprietary standard or industry standard (e.g., part 10 of VVC, HEVC, MPEG-4, AVC) or an extension of such standard. It should be appreciated that the present application is not limited to a particular video encoding/decoding standard and may be applicable to other video encoding/decoding standards. It is generally contemplated that video encoder 20 of source device 12 may be configured to encode video data according to any of these current or future standards. Similarly, it is also generally contemplated that the video decoder 30 of the target device 14 may be configured to decode video data according to any of these current or future standards.

Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder and/or decoder circuits, such as one or more microprocessors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), discrete logic devices, software, hardware, firmware or any combinations thereof. When implemented in part in software, the electronic device can store instructions for the software in a suitable non-transitory computer readable medium and execute the instructions in hardware using one or more processors to perform the video encoding/decoding operations disclosed in the present disclosure. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in the respective device.

In some implementations, at least a portion of the components of source device 12 (e.g., video source 18, video encoder 20, or components included in video encoder 20 described below with reference to fig. 2, and output interface 22) and/or at least a portion of the components of target device 14 (e.g., input interface 28, video decoder 30, or components included in video decoder 30 described below with reference to fig. 3, and display device 34) may operate in a cloud computing services network, such as software as a service (SaaS), platform as a service (PaaS), or infrastructure as a service (IaaS), which may provide software, platform, and/or infrastructure. In some implementations, one or more components of the source device 12 and/or the target device 14 that are not included in the cloud computing service network may be disposed in one or more client devices, and the one or more client devices may communicate with a server computer in the cloud computing service network through a wireless communication network (e.g., a cellular communication network, a short-range wireless communication network, or a Global Navigation Satellite System (GNSS) communication network) or a wired communication network (e.g., a Local Area Network (LAN) communication network or a Power Line Communication (PLC) network). In an embodiment, at least a portion of the operations described herein may be implemented as cloud-based services provided by one or more server computers implemented by at least a portion of the components of source device 12 and/or at least a portion of the components of target device 14 in a cloud computing services network, and one or more other operations described herein may be implemented by one or more client devices. In some implementations, the cloud computing service network may be a private cloud, a public cloud, or a hybrid cloud. Terms such as "cloud," "cloud computing," "cloud-based," and the like may be used interchangeably herein as appropriate without departing from the scope of the present disclosure. It should be understood that the present disclosure is not limited to implementation in the cloud computing service network described above. Rather, the disclosure may also be implemented in any other type of computing environment, whether currently known or developed in the future.

Fig. 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in this disclosure. Video encoder 20 may perform intra-prediction encoding and inter-prediction encoding on video blocks within video frames. Intra-prediction encoding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-prediction encoding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding, the term "frame" may be used as a synonym for the term "image" or "picture".

As shown in fig. 2, video encoder 20 includes a video data memory 40, a prediction processing unit 41, a Decoded Picture Buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy encoding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra prediction processing unit 46, and an intra Block Copy (BC) unit 48. In some implementations, video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63, such as a deblocking filter, may be located between adder 62 and DPB 64 to filter block boundaries to remove blocking artifacts from the reconstructed video. In addition to the deblocking filter, another loop filter, such as a Sample Adaptive Offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and/or an Adaptive Loop Filter (ALF), may be used to filter the output of adder 62. It should be noted that, for CCSAO techniques, this disclosure is not limited to the embodiments described herein, but may also be applied to the case where an offset is selected for any other one of the luma component, cb chroma component and Cr chroma component from any one of the luma component, cb chroma component and Cr chroma component to modify the any other component based on the selected offset. Further, it should also be noted that the first component referred to herein may be any one of a luminance component, a Cb chrominance component, and a Cr chrominance component, the second component may be any other one of the luminance component, the Cb chrominance component, and the Cr chrominance component, and the third component may be the remaining component of any one of the luminance component, the Cb chrominance component, and the Cr chrominance component. In some examples, the in-loop filter may be omitted and the decoded video block may be provided directly to DPB 64 by adder 62. Video encoder 20 may take the form of fixed or programmable hardware units, or may be dispersed among one or more of the fixed or programmable hardware units described.

Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data store 40 may be obtained, for example, from video source 18 shown in fig. 1. DPB 64 is a buffer that stores reference video data (e.g., reference frames or pictures) for use by video encoder 20 in encoding the video data (e.g., in intra or inter prediction encoding modes). Video data memory 40 and DPB 64 may be formed from any of a variety of memory devices. In various examples, video data memory 40 may be on-chip with other components of video encoder 20, or off-chip with respect to those components.

As shown in fig. 2, after receiving video data, a dividing unit 45 within the prediction processing unit 41 divides the video data into video blocks. This partitioning may also include partitioning the video frame into slices, tiles (tiles) (e.g., a set of video blocks), or other larger Coding Units (CUs) according to a predefined split structure (e.g., a Quadtree (QT) structure) associated with the video data. A video frame is or can be considered to be a two-dimensional array or matrix of samples having sample values. Samples in the array may also be referred to as pixels or picture elements (pels). The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and/or resolution of the video frame. For example, a video frame may be divided into a plurality of video blocks by using QT segmentation. The video block is again or can be considered to have a two-dimensional array or matrix of sample values, but with dimensions smaller than those of the video frame. The number of samples in the horizontal and vertical directions (or axes) of the video block defines the size of the video block. The video block may be further partitioned into one or more block partitions or sub-blocks (which may again form blocks) by, for example, iteratively using QT partitioning, binary Tree (BT) partitioning, or Trigeminal Tree (TT) partitioning, or any combination thereof. It should be noted that the term "block" or "video block" as used herein may be a portion of a frame or picture, especially a rectangular (square or non-square) portion. Referring to HEVC and VVC, a block or video block may be or correspond to a Coding Tree Unit (CTU), a CU, a Prediction Unit (PU), or a Transform Unit (TU) and/or may be or correspond to a respective block (e.g., a Coding Tree Block (CTB), a Coding Block (CB), a Prediction Block (PB), or a Transform Block (TB)) and/or sub-block.

The prediction processing unit 41 may select one of a plurality of possible prediction coding modes, for example, one of one or more inter prediction coding modes of a plurality of intra prediction coding modes, for the current video block based on the error result (e.g., the coding rate and the distortion level). The prediction processing unit 41 may provide the resulting intra-or inter-prediction encoded block to the adder 50 to generate a residual block and to the adder 62 to reconstruct the encoded block for subsequent use as part of a reference frame. Prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra mode indicators, partition information, and other such syntax information) to entropy encoding unit 56.

To select the appropriate intra-prediction encoding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction encoding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-prediction encoding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate encoding mode for each block of video data.

In some embodiments, motion estimation unit 42 determines the inter-prediction mode for the current video frame by generating a motion vector from a predetermined pattern within the sequence of video frames, the motion vector indicating a displacement of a video block within the current video frame relative to a predicted block within a reference video frame. The motion estimation performed by the motion estimation unit 42 is a process of generating a motion vector that estimates motion for a video block. For example, the motion vector may indicate the displacement of a video block within a current video frame or picture relative to a predicted block within a reference frame associated with the current block being encoded within the current frame. The predetermined pattern may designate video frames in the sequence as P-frames or B-frames. The intra BC unit 48 may determine the vector (e.g., block vector) for intra BC encoding in a similar manner as the motion vector used for inter prediction by the motion estimation unit 42, or may determine the block vector using the motion estimation unit 42.

In terms of pixel differences, a prediction block for a video block may be or may correspond to a block or reference block of a reference frame that is considered to closely match the video block to be encoded, and the pixel differences may be determined by Sum of Absolute Differences (SAD), sum of Square Differences (SSD), or other difference metric. In some implementations, video encoder 20 may calculate values for sub-integer pixel positions of reference frames stored in DPB 64. For example, video encoder 20 may interpolate values for one-quarter pixel positions, one-eighth pixel positions, or other fractional pixel positions of the reference frame. Accordingly, the motion estimation unit 42 can perform motion search with respect to the full pixel position and the fractional pixel position and output a motion vector having fractional pixel accuracy.

Motion estimation unit 42 calculates motion vectors for video blocks in inter-prediction encoded frames by comparing the locations of the video blocks with the locations of predicted blocks of reference frames selected from a first reference frame list (list 0) or a second reference frame list (list 1), each of which identifies one or more reference frames stored in DPB 64. The motion estimation unit 42 sends the calculated motion vector to the motion compensation unit 44 and then to the entropy encoding unit 56.

The motion compensation performed by the motion compensation unit 44 may involve acquiring or generating a prediction block based on the motion vector determined by the motion estimation unit 42. Upon receiving the motion vector for the current video block, motion compensation unit 44 may locate the prediction block to which the motion vector points in one of the reference frame lists, retrieve the prediction block from DPB 64, and forward the prediction block to adder 50. Adder 50 then forms a residual video block of pixel differences by subtracting the pixel values of the prediction block provided by motion compensation unit 44 from the pixel values of the current video block being encoded. The pixel differences forming the residual video block may include a luminance component difference or a chrominance component difference or both. Motion compensation unit 44 may also generate syntax elements associated with the video blocks of the video frames for use by video decoder 30 in decoding the video blocks of the video frames. The syntax elements may include, for example, syntax elements defining motion vectors used to identify the prediction block, any flags indicating the prediction mode, or any other syntax information described herein. It should be noted that the motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes.

In some embodiments, the intra BC unit 48 may generate vectors and obtain prediction blocks in a manner similar to that described above in connection with the motion estimation unit 42 and the motion compensation unit 44, but these prediction blocks are in the same frame as the current block being encoded, and these vectors are referred to as block vectors rather than motion vectors. In particular, the intra BC unit 48 may determine an intra prediction mode to be used to encode the current block. In some examples, intra BC unit 48 may encode the current block using various intra prediction modes, e.g., during separate encoding passes, and test their performance through rate-distortion analysis. Next, the intra BC unit 48 may select an appropriate intra prediction mode from among the various tested intra prediction modes to use and generate the intra mode indicator accordingly. For example, the intra BC unit 48 may calculate rate distortion values for various tested intra prediction modes using rate distortion analysis, and select the intra prediction mode having the best rate distortion characteristics among the tested modes to use as the appropriate intra prediction mode. Rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and an original uncoded block that is coded to produce the coded block, as well as the bit rate (i.e., number of bits) used to produce the coded block. The intra BC unit 48 may calculate ratios from the distortion and rate for the various encoded blocks to determine which intra prediction mode exhibits the best rate distortion value for the block.

In other examples, intra BC unit 48 may use motion estimation unit 42 and motion compensation unit 44, in whole or in part, to perform such functions for intra BC prediction in accordance with embodiments described herein. In either case, for intra block copying, the prediction block may be a block deemed to closely match the block to be encoded in terms of pixel differences, which may be determined by SAD, SSD, or other difference metric, and identifying the prediction block may include calculating a value for a sub-integer pixel location.

Regardless of whether the prediction block is from the same frame according to intra-prediction or from a different frame according to inter-prediction, video encoder 20 may form a residual video block by subtracting the pixel values of the prediction block from the pixel values of the current video block being encoded. The pixel differences forming the residual video block may include both a luma component difference and a chroma component difference.

As an alternative to inter prediction performed by the motion estimation unit 42 and the motion compensation unit 44 or intra block copy prediction performed by the intra BC unit 48 as described above, the intra prediction processing unit 46 may intra-predict the current video block. In particular, intra-prediction processing unit 46 may determine an intra-prediction mode for encoding the current block. To this end, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction processing unit 46 (or a mode selection unit in some examples) may select an appropriate intra-prediction mode from the tested intra-prediction modes for use. Intra-prediction processing unit 46 may provide information to entropy encoding unit 56 indicating the intra-prediction mode selected for the block. Entropy encoding unit 56 may encode information into the bitstream that indicates the selected intra-prediction mode.

After the prediction processing unit 41 determines a prediction block for the current video block via inter prediction or intra prediction, the adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform, such as a Discrete Cosine Transform (DCT) or a conceptually similar transform.

The transform processing unit 52 may send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficient to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The quantization level may be modified by adjusting the quantization parameter. In some examples, quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy encoding unit 56 may perform scanning.

After quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context Adaptive Variable Length Coding (CAVLC), context Adaptive Binary Arithmetic Coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability Interval Partition Entropy (PIPE) coding, or another entropy encoding method or technique. The encoded bitstream may then be sent to a video decoder 30 as shown in fig. 1, or archived in a storage device 32 as shown in fig. 1 for later transmission to the video decoder 30 or retrieval by the video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transforms, respectively, to reconstruct the residual video block in the pixel domain for generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values for use in motion estimation.

Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used by the intra BC unit 48, the motion estimation unit 42, and the motion compensation unit 44 as a prediction block to inter-predict another video block in a subsequent video frame.

Fig. 3 is a block diagram illustrating an exemplary video decoder 30 according to some embodiments of the present application. Video decoder 30 includes video data memory 79, entropy decoding unit 80, prediction processing unit 81, inverse quantization unit 86, inverse transform processing unit 88, adder 90, and DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra prediction unit 84, and an intra BC unit 85. Video decoder 30 may perform a decoding process that is substantially reciprocal to the encoding process described above in connection with fig. 2 with respect to video encoder 20. For example, the motion compensation unit 82 may generate prediction data based on the motion vector received from the entropy decoding unit 80, and the intra prediction unit 84 may generate prediction data based on the intra prediction mode indicator received from the entropy decoding unit 80.

In some examples, the elements of video decoder 30 may be tasked to perform embodiments of the present application. Further, in some examples, embodiments of the present disclosure may be dispersed in one or more of the units of video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in combination with other units of the video decoder 30 (e.g., the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80). In some examples, video decoder 30 may not include intra BC unit 85, and the functions of intra BC unit 85 may be performed by other components of prediction processing unit 81 (e.g., motion compensation unit 82).

Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. The video data stored in the video data memory 79 may be obtained, for example, from the storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). The video data memory 79 may include a Coded Picture Buffer (CPB) that stores coded video data from a coded video bitstream. DPB 92 of video decoder 30 stores reference video data for use by video decoder 30 (e.g., in intra-or inter-prediction encoding mode) in decoding the video data. Video data memory 79 and DPB 92 may be formed of any of a variety of memory devices, such as Dynamic Random Access Memory (DRAM), including Synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, video data memory 79 and DPB 92 are depicted in fig. 3 as two different components of video decoder 30. It will be apparent to those skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip with other components of video decoder 30, or off-chip with respect to those components.

During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks of encoded video frames and associated syntax elements. Video decoder 30 may receive syntax elements at the video frame level and/or the video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors, or intra-prediction mode indicators, as well as other syntax elements. Entropy decoding unit 80 then forwards the motion vector or intra prediction mode indicator, as well as other syntax elements, to prediction processing unit 81.

When a video frame is encoded as an intra prediction encoded (I) frame or an intra encoding prediction block used in other types of frames, the intra prediction unit 84 of the prediction processing unit 81 may generate prediction data for a video block of the current video frame based on the signaled intra prediction mode and reference data from a previously decoded block of the current frame.

When a video frame is encoded as an inter-prediction encoded (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vectors and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks may be generated from reference frames within one of the reference frame lists. Video decoder 30 may construct a list of reference frames, i.e., list 0 and list 1, using a default construction technique based on the reference frames stored in DPB 92.

In some examples, when video blocks are encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block defined by video encoder 20.

The motion compensation unit 82 and/or the intra BC unit 85 determine prediction information for the video block of the current video frame by parsing the motion vector and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine a prediction mode (e.g., intra-prediction or inter-prediction) for encoding a video block of a video frame, an inter-prediction frame type (e.g., B or P), construction information for one or more of a reference frame list for the frame, a motion vector for each inter-prediction encoded video block of the frame, an inter-prediction state for each inter-prediction encoded video block of the frame, and other information for decoding the video block in the current video frame.

Similarly, the intra BC unit 85 may use some of the received syntax elements, such as flags, to determine which video blocks of the frame are within the reconstruction region and should be stored in the DPB 92 for which the current video block was predicted using intra BC mode, the block vector for each intra BC predicted video block of the frame, the intra BC prediction status for each intra BC predicted video block of the frame, and other information for decoding the video blocks in the current video frame.

Motion compensation unit 82 may also perform interpolation using interpolation filters, such as those used by video encoder 20 during encoding of video blocks, to calculate interpolation values for sub-integer pixels of the reference block. In this case, motion compensation unit 82 may determine interpolation filters used by video encoder 20 from the received syntax elements and use these interpolation filters to generate the prediction block.

The dequantization unit 86 dequantizes quantized transform coefficients provided in the bitstream and entropy decoded by the entropy decoding unit 80 using the same quantization parameter calculated by the video encoder 20 for each video block in the video frame that is used to determine the degree of quantization. The inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to reconstruct the residual block in the pixel domain.

After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs a decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. Loop filter 91 (e.g., deblocking filter, SAO filter, CCSAO filter, and/or ALF) may be located between adder 90 and DPB 92 to further process the decoded video block. In some examples, loop filter 91 may be omitted and the decoded video block may be provided directly to DPB 92 by adder 90. The decoded video blocks in a given frame are then stored in DPB 92, and DPB 92 stores reference frames for subsequent motion compensation of the next video block. DPB 92 or a memory device separate from DPB 92 may also store decoded video for later presentation on a display device (e.g., display device 34 of fig. 1).

In a typical video codec process, a video sequence generally includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luminance samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other examples, the frame may be monochromatic, and thus include only one two-dimensional array of luminance samples.

As shown in fig. 4A, video encoder 20 (or more specifically, partitioning unit 45) generates an encoded representation of a frame by first partitioning the frame into a set of CTUs. The video frame may include an integer number of CTUs ordered consecutively from left to right and top to bottom in raster scan order. Each CTU is the largest logical coding unit and the width and height of the CTU are signaled by video encoder 20 in the sequence parameter set such that all CTUs in the video sequence have the same size of one of 128 x 128, 64 x 64, 32 x 32, and 16 x 16. It should be noted that the application is not necessarily limited to a particular size. As shown in fig. 4B, each CTU may include one CTB of a luminance sample, two corresponding coding tree blocks of a chrominance sample, and a syntax element for coding the samples of the coding tree blocks. Syntax elements describe the nature of the different types of units encoding the pixel blocks and how the video sequence may be reconstructed at video decoder 30, including inter-or intra-prediction, intra-prediction modes, motion vectors, and other parameters. In a monochrome picture or a picture having three separate color planes, a CTU may comprise a single coding tree block and syntax elements for encoding samples of the coding tree block. The coding tree block may be an nxn block of samples.

To achieve better performance, video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, trigeminal tree partitioning, quadtree partitioning, or a combination thereof, on the coded tree blocks of CTUs and divide the CTUs into smaller CUs. As depicted in fig. 4C, a 64 x 64 CTU 400 is first divided into four smaller CUs, each having a block size of 32 x 32. Among the four smaller CUs, the CUs 410 and 420 are divided into four CUs with block sizes of 16×16, respectively. Two 16×16 CUs 430 and 440 are each further divided into four CUs with block sizes of 8×8. Fig. 4D depicts a quadtree data structure showing the final result of the segmentation process of CTU 400 as depicted in fig. 4C, each leaf node of the quadtree corresponding to one CU of a respective size ranging from 32 x 32 to 8 x 8. Similar to the CTU depicted in fig. 4B, each CU may include two corresponding coding blocks of CB and chroma samples of luma samples of the same size frame, and syntax elements for coding the samples of the coding blocks. In a monochrome picture or a picture having three separate color planes, a CU may comprise a single coding block and syntax structures for encoding samples of the coding block. It should be noted that the quadtree partitions depicted in fig. 4C and 4D are for illustrative purposes only, and that one CTU may be split into multiple CUs based on quadtree partitions/trigeminal partitions/binary tree partitions to accommodate varying local characteristics. In a multi-type tree structure, one CTU is partitioned according to a quadtree structure, and each quadtree leaf CU may be further partitioned according to a binary and trigeminal tree structure. As shown in fig. 4E, there are five possible segmentation types for a coding block having a width W and a height H, namely, quaternary segmentation, horizontal binary segmentation, vertical binary segmentation, horizontal ternary segmentation, and vertical ternary segmentation.

In some implementations, video encoder 20 may further partition the coding block of the CU into one or more (mxn) PB. PB is a rectangular (square or non-square) block of samples to which the same prediction (inter or intra) is applied. The PU of a CU may include a PB of a luma sample, two corresponding PB of chroma samples, and syntax elements for predicting the PB. In a monochrome picture or a picture having three separate color planes, a PU may include a single PB and syntax structures for predicting the PB. Video encoder 20 may generate a predicted luma block, a predicted Cb block, and a predicted Cr block for luma PB, cb PB, and Cr PB of each PU of the CU.

Video encoder 20 may use intra-prediction or inter-prediction to generate a prediction block for the PU. If video encoder 20 uses intra-prediction to generate the prediction block of the PU, video encoder 20 may generate the prediction block of the PU based on decoded samples of the frame associated with the PU. If video encoder 20 uses inter prediction to generate the prediction block of the PU, video encoder 20 may generate the prediction block of the PU based on decoded samples of one or more frames other than the frame associated with the PU.

After video encoder 20 generates the predicted luma block, the predicted Cb block, and the predicted Cr block for the one or more PUs of the CU, video encoder 20 may generate a luma residual block for the CU by subtracting the predicted luma block of the CU from the original luma coded block of the CU such that each sample in the luma residual block of the CU indicates a difference between a luma sample in one of the predicted luma blocks of the CU and a corresponding sample in the original luma coded block of the CU. Similarly, video encoder 20 may generate Cb residual blocks and Cr residual blocks for the CU, respectively, such that each sample in the Cb residual block of the CU indicates a difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb encoded block of the CU, and each sample in the Cr residual block of the CU may indicate a difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr encoded block of the CU.

Further, as shown in fig. 4C, video encoder 20 may use quadtree partitioning to decompose a luma residual block, a Cb residual block, and a Cr residual block of the CU into one or more luma transform blocks, cb transform blocks, and Cr transform blocks, respectively. The transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A TU of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements for transforming the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of a luma residual block of a CU. The Cb transform block may be a sub-block of a Cb residual block of the CU. The Cr transform block may be a sub-block of a Cr residual block of the CU. In a monochrome picture or a picture having three separate color planes, a TU may comprise a single transform block and syntax structures for transforming the samples of the transform block.

Video encoder 20 may apply one or more transforms to the luma transform block of the TU to generate a luma coefficient block for the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar quantities. Video encoder 20 may apply one or more transforms to the Cb transform block of the TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to the Cr transform blocks of the TUs to generate Cr coefficient blocks for the TUs.

After generating the coefficient block (e.g., the luma coefficient block, the Cb coefficient block, or the Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to the process by which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements that indicate the quantized transform coefficients. For example, video encoder 20 may perform CABAC on syntax elements that indicate quantized transform coefficients. Finally, video encoder 20 may output a bitstream including a sequence of bits that form a representation of the encoded frames and associated data, which is stored in storage device 32 or transmitted to target device 14.

Upon receiving the bitstream generated by video encoder 20, video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct the frames of video data based at least in part on syntax elements obtained from the bitstream. The process of reconstructing video data is typically reciprocal to the encoding process performed by video encoder 20. For example, video decoder 30 may perform an inverse transform on the coefficient blocks associated with the TUs of the current CU to reconstruct residual blocks associated with the TUs of the current CU. Video decoder 30 also reconstructs the coding block of the current CU by adding samples of the prediction block for the PU of the current CU to corresponding samples of the transform block of the TU of the current CU. After reconstructing the encoded blocks for each CU of the frame, video decoder 30 may reconstruct the frame.

As described above, video coding mainly uses two modes, i.e., intra-frame prediction (or intra-frame prediction) and inter-frame prediction (or inter-frame prediction), to achieve video compression. Note that IBC may be considered as intra prediction or third mode. Between the two modes, since motion vectors are used to predict the current video block from the reference video block, inter prediction contributes more to coding efficiency than intra prediction.

But with ever-improving video data capture techniques and finer video block sizes for preserving details in video data, the amount of data required to represent the motion vector for the current frame has also increased substantially. One way to overcome this challenge benefits from the fact that not only are a set of neighboring CUs in both the spatial and temporal domains have similar video data for prediction purposes, but the motion vectors between these neighboring CUs are also similar. Thus, the motion information of spatially neighboring CUs and/or temporally co-located CUs may be used as an approximation of the motion information (e.g., motion vector) of the current CU, also referred to as the "motion vector predictor" (MVP) of the current CU, by exploring their spatial and temporal correlation.

Instead of encoding the actual motion vector of the current CU, as determined by the motion estimation unit 42, into the video bitstream as described above in connection with fig. 2, the motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to generate a Motion Vector Difference (MVD) for the current CU. By doing so, it is not necessary to encode the motion vector determined by the motion estimation unit 42 for each CU of a frame into the video bitstream, and the amount of data representing the motion information in the video bitstream can be significantly reduced.

Similar to the process of selecting a prediction block in a reference frame during inter prediction of an encoded block, both video encoder 20 and video decoder 30 need to employ a set of rules for constructing a motion vector candidate list (also referred to as a "merge list") for the current CU using those potential candidate motion vectors associated with spatially neighboring CUs and/or temporally co-located CUs of the current CU, and then select one member from the motion vector candidate list as a motion vector predictor for the current CU. By doing so, there is no need to send the motion vector candidate list itself from video encoder 20 to video decoder 30, and the index of the selected motion vector predictor within the motion vector candidate list is sufficient for video encoder 20 and video decoder 30 to use the same motion vector predictor within the motion vector candidate list to encode and decode the current CU.

In general, the basic intra prediction scheme applied in VVC remains almost the same as that of HEVC, except that several prediction tools are further extended, added and/or improved, e.g., extended intra prediction with wide-angle intra modes, multi-reference line (MRL) intra prediction, PDPC, intra sub-partition (ISP) prediction, CCLM prediction, MIP, and intra Template Matching Prediction (TMP).

In ECM, intra template matching prediction (intra TMP) is used to improve compression efficiency of intra codec. Hereinafter, an intra TMP technique and an improved method thereof are provided.

Intra-frame template matching prediction

Intra template matching prediction (Intra TMP) is a special Intra prediction mode that replicates the best prediction block of the L-shaped template matching the current template from the reconstructed portion of the current frame. For a predefined search range, the encoder searches for a template most similar to the current template in the reconstructed portion of the current frame and uses the corresponding block as a prediction block. The encoder then signals the use of this mode and performs the same prediction operation on the decoder side.

Generating a prediction signal by matching an L-shaped causal neighbor of a current block with another block in a predefined search area in fig. 6, the predefined search area being comprised of:

R1:Current CTU

R2-upper left CTU

R3:upper CTU

R4 left CTU

The Sum of Absolute Differences (SAD) is used as the cost function.

Within each region, the decoder searches for a template having the smallest SAD with respect to the current template, and uses its corresponding block as a prediction block.

The sizes (SEARCHRANGE _w, SEARCHRANGE _h) of all regions are set in proportion to the block sizes (BlkW, blkH) so that each pixel has a fixed number of SAD comparisons. Namely:

SearchRange_w=a*BlkW

SearchRange_h=a*BlkH

where 'a' is a constant that controls the gain/complexity tradeoff. In practice, 'a' is equal to 5.

For CUs with width and height dimensions less than or equal to 64, an intra template matching tool is enabled. This maximum CU size for intra template matching is configurable.

When the current CU does not use DIMD, intra templates match prediction modes at the CU level are signaled by a dedicated flag.

Block vector candidates for intra TMP derivation for IBC

In the method, a Block Vector (BV) derived from intra template matching prediction (IntraTMP) is used for Intra Block Copy (IBC). Stored IntraTMP BV of neighboring blocks and IBC BV are used as spatial BV candidates in IBC candidate list construction.

IntraTMP block vectors are stored in the IBC block vector buffer and the current IBC block may use both IBC BVs and IntraTMP BV of neighboring blocks as BV candidates for the IBC BV candidate list, as shown in fig. 7.

Intra-frame TMP using linear filter model

In JVET-AC0109, it is proposed to apply a linear filter model to the prediction of intra TMP (intra TMP-FLM). The proposed 6-tap filter consists of a 5-tap plus sign shape space component and a bias term. The input of the spatial 5 tap component of the filter is made up of the center (C) sample of the reference block, which is located at a position corresponding to the sample in the current block to be predicted, and its up/north (N), down/south (S), left/west (W) and right/east (E) neighbors, as shown in fig. 8.

The bias term B represents a scalar offset between the input and output and is set to an intermediate luminance value (512 for 10-bit content).

The output of the filter is calculated as follows:

predLumaVal=c0C+c1N+c2S+c3E+c4W+c5B

the filter coefficients ci are calculated by minimizing the MSE between the reference template and the current template, as shown in fig. 9. The area shown by the dots needs to be expanded to support the "side-sample" of the plus sign shaped spatial filter and filled in the unavailable areas.

MSE minimization is performed by computing an autocorrelation matrix of the reference template input and the current template output. LDL decomposition is performed on the autocorrelation matrix and the final filter coefficients are calculated using a back-substitution method.

The usage of the intra TMP-FLM mode is signaled by the encoded CU level flags. Specifically, the intra TMP-FLM is considered a sub-mode of the intra TMP. That is, the intra TMP-FLM flag is signaled only if the intra TMP flag is true.

Combination of intra and intra TMP

At JVET-AC conference, several methods are proposed to fuse intra mode and intra TMP mode predictions. For example, in JVET-AC0097, it is proposed to use geometric partitioning to combine intra prediction with intra TMP. In JVET-AC0201 a spatial CIIP mode is proposed, where intra prediction and intra TMP can be fused in a manner of CIIP.

CCLM prediction

In order to reduce cross-component redundancy, a CCLM prediction mode is used in VVC, wherein chroma samples of a CU are predicted based on reconstructed luma samples rec _L (i, j) of the CU using the following linear model:

pred_C(i,j)＝α·rec_L′(i,j)+β (1)

Where pred _C (i, j) represents the predicted chroma samples in the CU, rec _L' (i, j) represents the downsampled reconstructed luma samples of the CU resulting from downsampling the reconstructed luma samples rec _L (i, j), and α and β are linear model coefficients derived from up to four neighboring chroma samples and their corresponding downsampled luma samples. Assuming that the current chroma block size is w×h, W 'and H' can be obtained as follows:

-when applying LM mode, W '=w, H' =h;

-when lm_a mode is applied, W' =w+h;

when lm_l mode is applied, H' =h+w.

Wherein in LM mode, the upper and left side samples of the CU are used together to calculate the linear model coefficients, in lm_a mode, only the upper sample of the CU is used to calculate the linear model coefficients, and in lm_l mode, only the left side sample of the CU is used to calculate the linear model coefficients.

If the position of the upper sample of the chroma block is denoted as S [0, -1]. S [ W '-1, -1] and the position of the left sample of the chroma block is denoted as S [ -1,0]. S [ -1, h' -1], the positions of four neighboring chroma samples are selected as follows:

-selecting S [ W '/4, -1], S [3*W'/4, -1], S [ -1, h '/4] and S [ -1,3 x h'/4] as the locations of four adjacent chroma samples when the LM mode is applied and both the top and left samples are available;

-selecting S [ W '/8, -1], S [3*W'/8, -1], S [5*W '/8, -1] and S [7*W'/8, -1] as the locations of four adjacent chroma samples when lm_a mode is applied and the top samples are available or when only the top samples are available;

-selecting S [ -1, h '/8], S [ -1,3 x h'/8], S [ -1,5 x h '/8] and S [ -1,7 x h'/8] as the positions of four adjacent chroma samples when the lm_l mode is applied and the left sample is available or when only the left sample is available.

Four neighboring luminance samples corresponding to the selected location are obtained by a downsampling operation and the four obtained neighboring luminance samples are compared four times to find two larger values, x ⁰ _A and x ¹ _A and two smaller values, x ⁰ _B and x ¹ _B. The chroma sample values corresponding to the two larger values and the two smaller values are denoted as y ⁰ _A,y¹ _A,y⁰ _B and y ¹ _B, respectively. X _a、X_b、Y_a and Y _b were then obtained as follows:

Finally, the linear model coefficients α and β are obtained according to the following formula.

β=Y_b-α·X_b

Fig. 10 shows schematic diagrams of positions of left and upper samples of a CU related to the CCLM mode, including positions of left and upper samples of an nxn chroma block 501 in the CU, and positions of left and upper samples of a 2 nx2N luma block 503 in the CU.

The above parameter calculations are performed as part of the decoding process, so no syntax elements are used to transfer the values of α and β from video encoder 20 to video decoder 30.

Convolution cross-component intra-prediction model

In the method, a convolution cross-component model (CCCM) is applied to predict chroma samples from reconstructed luma samples, the spirit of which is similar to the current CCLM mode. As with CCLM, when chroma subsampling is used, reconstructed luma samples are downsampled to match the lower resolution chroma grid. Similar to CCLM, the top reference sample, the left reference sample, or both the top reference sample and the left reference sample are used as templates for model derivation.

Furthermore, similar to CCLM, single-or multi-model variants of CCCM may be chosen for use. The multiple model variant uses two models, one derived for samples above the average luminance reference value and the other derived for the remaining samples (following the spirit of the CCLM design). For PUs having at least 128 reference points available, a multi-model CCCM mode may be selected.

Convolution filter

The convolution 7-tap filter is composed of a 5-tap plus sign shape space component, a non-linear term, and a bias term. The spatial 5-tap component of the filter is input consisting of a center (C) luminance sample co-located with the chrominance sample to be predicted and its up/north (N), down/south (S), left/west (W) and right/east (E) neighboring samples, as shown in fig. 8.

The nonlinear term P is expressed as the second power of the center luminance sample point C and scaled to the sample point value range of the content:

p= (c×c+ midVal) > > bit depth

That is, for 10-bit content, the calculation method is as follows:

P=(C*C+512)>>10

The offset term B represents the scalar offset between the input and output (similar to the offset term in CCLM) and is set to an intermediate chroma value (512 for 10-bit content).

The output of the filter is calculated as the convolution between the filter coefficients ci and the input values and truncated to within the range of valid chroma samples:

predChromaVal=c0C+c1N+c2S+c3E+c4W+c5P+c6B

Calculation of filter coefficients

The filter coefficients ci are calculated by minimizing the MSE between the predicted and reconstructed chroma samples in the reference region. Fig. 11 shows a reference area composed of 6 lines of chroma samples above and to the left of the PU. The reference region extends one PU width to the right of the PU boundary and one PU height below the PU boundary. The region is adjusted to include only available samples. The area shown by the dots needs to be expanded to support the "side-sample" of the plus sign shaped spatial filter and filled in the unavailable areas.

MSE minimization is performed by computing an autocorrelation matrix of the luminance input and a cross-correlation vector between the luminance input and the chrominance output. LDL decomposition is performed on the autocorrelation matrix and the final filter coefficients are calculated using a back-substitution method. This process approximately follows the calculation of ALF filter coefficients in the ECM, but instead of Cholesky decomposition, LDL decomposition is chosen to avoid square root operations.

The autocorrelation matrix is calculated using the reconstructed values of the luminance and chrominance samples. These samples are full-range (e.g., between 0 and 1023 for 10 bits of content), resulting in relatively large values in the autocorrelation matrix. This requires high bit depth operations during model parameter computation. It is proposed to remove a fixed offset from the luminance and chrominance samples in each PU for each model. This reduces the size of the values used in the model creation and allows the reduction of the precision required for fixed point arithmetic operations. Thus, it is proposed to use 16-bit decimal precision instead of 22-bit precision of the original CCCM implementation.

For simplicity, the reference sample value outside the top left corner of the PU is used as the offset (offsetLuma, offsetCb and offsetCr). The sample values used in model creation and final prediction (i.e., luminance and chrominance in the reference region and luminance in the current PU) will both subtract these fixed values as follows:

·C'=C–offsetLuma

·N'=N–offsetLuma

·S'=S–offsetLuma

·E'=E–offsetLuma

·W'=W–offsetLuma

·P'=nonLinear(C')

b= midValue =1 < < (bit depth-1)

And predicting the chroma values using the following equation, wherein offsetChroma is equal to offsetCr and offsetCb for Cr and Cb components, respectively:

predChromaVal=c0C'+c1N'+c2S'+c3E'+c4W'+c5P'+c6B+offsetChroma

to avoid any additional sample level operations, the luminance offset is removed during luminance reference sample interpolation. This may be accomplished, for example, by replacing the rounding term used in the luminance reference sample interpolation with an updated offset that includes both the rounding term and offsetLuma. The chroma offset may be removed by subtracting the chroma offset directly from the reference chroma sample. Alternatively, the effect of the chroma offset may be removed from the cross component vector, resulting in the same result. To add the chroma offset back to the output of the convolution prediction operation, the chroma offset is added to the bias term of the convolution model.

The process of CCCM model parameter calculation requires division operations. Division operations are not always considered to be friendly in implementation. The division operation is replaced with a multiplication (by a scaling factor) and a shift operation, where the scaling factor and the number of shifts are calculated based on a denominator, similar to the method used in calculating the CCLM parameter.

Gradient linear model

For the YUV 4:2:0 color format, chroma samples can be predicted from luma sample gradients using a Gradient Linear Model (GLM) method. Two modes, a dual-parameter GLM mode and a three-parameter GLM mode are supported.

In contrast to CCLM, the bi-parameter GLM does not use downsampled luminance values, but rather uses a luminance sample gradient to derive a linear model. Specifically, when applying the dual parameter GLM, the input of the CCLM process (i.e., the downsampled luminance samples L) is replaced with a luminance sample gradient G. The other parts of the CCLM (e.g., parameter derivation, predicted sample line transformation) remain unchanged.

C=α·G+β

In a three parameter GLM, different parameters may be used to predict chroma samples based on both the luma sample gradient and the downsampled luma values. Model parameters for three-parameter GLM were derived from 6 rows and 6 columns of adjacent samples by the MSE minimization method based on LDL decomposition as used in CCCM.

C=α₀·G+α₁·L+α₂·β

For signaling, when CCLM mode is enabled for the current CU, one flag will be signaled to indicate whether both Cb and Cr components are GLM enabled, if GLM is enabled, the other flag is signaled to indicate which of the two GLM modes is selected, and a syntax element is further signaled to select one of the 4 gradient filters for gradient computation. Four gradient filters of the GLM are shown in fig. 12.

Bit stream signaling

The usage of this mode is signaled by the PU level flag of CABAC codec. This is supported by including a new CABAC context. In terms of signaling, CCCM is considered a sub-mode of the CCLM. That is, the CCCM flag is signaled only when the intra prediction mode is lm_chroma.

While existing intra TMP schemes may provide significant improvements in intra-frame codec in ECM, their design may still be further improved. For example, the present disclosure recognizes that current intra TMP designs suffer from the following drawbacks:

First, in the current intra TMP, only integer pixel based template searching is utilized, while fractional interpolation is ignored, i.e., only integer pixel template matching blocks are used as prediction blocks. It should be appreciated that fractional interpolation can improve the prediction accuracy of inter-prediction, and thus ignoring fractional interpolation makes current intra TMP less efficient.

Second, in the current intra TMP, only one hypothesis is utilized, i.e., the best matching block that results in the smallest matching cost is selected as the final prediction. However, a single hypothesis may be susceptible to noise, making the prediction less accurate. Furthermore, it has been shown that in inter prediction, introducing multiple hypotheses can improve prediction accuracy. Based on the above analysis, it is reasonable to propose intra TMP based on multiple hypotheses.

Third, in JVET-AC0109, the intra TMP block is further filtered using a linear filter model. More specifically, the intra TMP block is the reference block within the search range that has the smallest template matching cost. However, as disclosed above, a single hypothesized intra TMP may not be sufficiently effective. Therefore, it is proposed to combine multiple hypotheses with a linear filter model.

Fourth, in the current ECM, the intra TMP is enabled only for the luminance component, and the chrominance component is not considered. Thus, it is proposed to enable intra TMP for the chrominance components.

Fifth, in the current method of combining intra and intra TMPs, the fused prediction is directly used as the final prediction without further refinement.

In the present disclosure, a method is presented to further increase the compression efficiency of intra-frame TMPs in an ECM. Overall, the main features of the method proposed in the present disclosure are summarized below.

First, to improve the accuracy of intra TMP prediction, fractional pixel template matching is proposed. The integer template matching block may be further refined by using fractional interpolation.

Second, in order to improve the accuracy of intra TMP, a multi-hypothesis based intra TMP scheme is proposed. In the proposed scheme, a plurality of template matching blocks are searched and weighted to generate a final prediction block. The proposed multiple hypothesis intra TMPs can be divided into two classes, fixed multiple hypothesis intra TMPs and adaptive intra TMPs, according to the derivation of the weighting factors.

Third, in order to further improve the prediction accuracy of the intra TMP using the linear filter model, multiple hypotheses are introduced, i.e., intra TMP is performed by the multiple hypotheses and then the linear filter model is performed.

Fourth, to further improve the prediction of the chrominance components, it is proposed to enable intra TMP for the chrominance components.

Fifth, to exploit the cross-component correlation between the luma and chroma components, it is proposed to combine cross-component prediction and chroma component intra TMP.

Sixth, to further improve the prediction of the combined intra and intra TMP, a linear filter model is utilized to refine the fused prediction.

Fractional intra TMP

In the current intra TMP mode, L-shaped templates are used to search for prediction blocks from reconstructed regions in the current frame. In the template matching process, the best prediction block that results in the smallest matching cost is identified and used as the final prediction block. In the current example of intra TMP, template matching is based on integer pixels. To further improve the codec performance of intra TMP mode, in the present disclosure, examples are provided that use fractional pixel template matching for intra TMP mode. The interpolation accuracy of the fractional pixel template matching may be, but is not limited to, half-pel, quarter-pel, etc.

In one or another embodiment, an index value may be signaled in the bitstream to indicate the interpolation accuracy used. The interpolation accuracy index may be signaled at the PPS, SPS, or slice header.

In one or another embodiment, the fractional intra TMP can be split into two steps. In a first step, a template matching process in the current ECM is performed to identify the best integer predictive template. In the second step, the fractional templates around the integer template are interpolated and the fractional pixel template matching is performed. Fig. 13 provides an example of half-pixel template matching, where a _i,j represents integer pixels and b _i,j、h_i,j and j _i,j represent half-pixels. After the best integer prediction template is identified, the eight half-pixel prediction templates are then interpolated. The distances between the current template and the best integer prediction template and the eight half-pixel prediction templates are calculated and compared. The prediction template that results in the smallest template matching cost is identified and used as the best prediction template. The corresponding prediction block of the best prediction template is used as the final prediction of the current block.

Multiple hypothesis intra TMP

In current ECMs, only the best template matching block that results in the smallest template matching cost is used for prediction. In some examples, the selectable template matching blocks are selected based on an ascending order of template matching costs, and the template that results in a lower template matching cost than other candidate template matching costs is selected as the best template matching block/CU. To further improve the accuracy of intra TMP prediction, a multiple hypothesis prediction method is provided in the present disclosure. In a multiple hypothesis intra TMP, more than one prediction block candidate is used and weighted to generate the final prediction of the current block. It is assumed that N prediction block candidates are used.

Prediction block candidate derivation

In one and another embodiment, the prediction block candidates are searched and selected according to criteria that minimizes the template matching cost, i.e., the first N candidates with the smallest template matching cost are selected. The template matching cost may not be limited to SAD (sum of absolute differences) and SSE (sum of squares errors).

Fixed multiple hypothesis intra TMP

In this embodiment, the weighting factors used to generate the final prediction block are predefined and fixed at both the encoder side and the decoder side. As an example, equal weighting factors may be used, i.e. all candidate blocks have a weighting factor of 1/N.

Adaptive multiple hypothesis intra TMP

Examples of adaptive multiple hypothesis intra TMP methods are also provided in order to accommodate different characteristics of video content.

Method 1

In one and another embodiment, the weighting factor may be derived based on template matching costs. The template matching cost for the N candidates is denoted as C ₁,C₂,…,C_N and the weighting factor is calculated as follows.

It should be noted that the template matching costs may be measured using, but not limited to, SAD and SSE.

Method 2

In yet another embodiment, the weighting factors may be extrapolated at the encoder side and then signaled to the decoder in the bitstream. The N prediction block candidates are denoted as P ₁,P₂,…,P_N and the current block is denoted as X, and then the weighting factor can be solved by:

equation (5) can be solved using the Wiener-Hopf equation as ALF. The derived filter coefficients are then quantized to integer types and signaled at the block level.

Method 3

In yet another embodiment, the weighting factor is derived based on a template and the derived weighting factor is applied to the prediction block candidates to generate the final prediction block. The template of the prediction candidate is denoted as T ₁,T₂,…,T_N and the template of the current block is denoted as T, and then the weighting factor can be derived using the following equation:

equation (6) can be solved using the wiener-hopout equation. The final prediction block may then be calculated as Where P _i denotes the i-th prediction block candidate.

Method 4

Intra TMP mode exploits non-local correlation to improve prediction accuracy, where similar blocks are searched for and used to generate the final predicted block. In this embodiment, an example is provided that combines non-local mean filtering with multiple hypothesis intra TMP, as described below. In a first step, N prediction block candidates are searched and identified as done in the intra TMP of the ECM. In the second step, the weighting factor is calculated as follows.

Wherein D _i is used to measure the distance between the template of the ith predicted block candidate and the template of the current block, h is used as the weighted strength, and Z [ i ] is the normalization constant:

in order to calculate the weighting factor in equation (7), the weighted intensity or weighted intensity should be determined first. In this disclosure, several methods are provided to determine the weighting strength.

In the first method, a weighted intensity candidate list composed of some typical weighted intensity values is defined and fixed at the encoder and decoder sides. At the encoder side, the weighted intensity values are checked using rate distortion optimization, the best weighted intensity value is identified and signaled in the bitstream to the decoder side.

In a second method, a weighted intensity value is estimated using a template of the prediction block candidate and a template of the current block. The template of the prediction candidate is denoted as T ₁,T₂,…,T_N and the template of the current block is denoted as T. The weighted intensity values can then be solved using the following equation:

In a third method, the QP value and the variance of the template of the current block may be used to estimate the weighted strength value, i.e., the relationship between the weighted strength value, QP value, and template variance may be fitted offline.

Method 5

To better exploit non-local dependencies in intra TMPs, in this embodiment Singular Value Decomposition (SVD) is utilized to generate the final prediction block from the prediction block candidates. The width and height of the current block are denoted as W and H, and the area of the current block is denoted as d=h×w.

Step 1. Search and identify K prediction block candidates y _i as done in the intra TMP of the ECM.

Step 2. K prediction block candidate building block groups G of the current block y and arranged as a matrix:

Where Y _G is a matrix having a size d x K by arranging each candidate in group G as a column vector.

And 3, performing SVD decomposition on the matrix Y _G.

And 4, applying a soft threshold operation to the singular value matrix _G.

Wherein softTh () is a function of the diagonal elements of Λ _G shrunk with a threshold τ. For the kth diagonal element in Λ _G, it is shrunk on level τ (k) by a nonlinear function D _τ(k):

D_τ(k):λ_k,τ(k)＝max(|λ_k|-τ(k),0) (13)

Λ _G,τ is a matrix of post-shrinkage singular values λ _k,τ(k) located at diagonal positions.

And 5, performing inverse SVD to obtain a filtered block group.

One of the key steps is to determine the threshold value for each diagonal element in step 4. In the present disclosure, the threshold value is calculated as follows. For each group of image blocks, the threshold is estimated using the following equation:

Where σ _n,G is the standard deviation of the noise and σ _x,G,k is the standard deviation of the original block in the k-th dimension of the SVD space of group G. The deviation of the original block in the SVD space is estimated according to the following equation.

Wherein, the Is thatIs the k-th singular value of (c). When σ _x,G,k is zero, the soft threshold operation is skipped. In addition, the deviation of the noise is estimated with the deviation of the prediction block using a power function parameterized with α and β.

σ_n＝α×σ_y ^β (17)

Wherein σ _y is calculated as follows,

Here, y _k (i) denotes the i-th pixel of the prediction block candidate vector y _k.

Multiple hypothesis intra TMP signaling

In the present disclosure, the provided multiple hypothesis intra TMP may be used as a substitute for the current intra TMP mode, or the encoder may adaptively select the intra TMP mode or the multiple hypothesis intra TMP mode.

In one and another embodiment, the provided multiple hypothesis intra TMP is used as a substitute for the current intra TMP mode, i.e. the prediction is always performed using multiple hypotheses.

In yet another embodiment, one of the multiple hypothesized intra TMP methods in the section above is used in combination with the current intra TMP mode in the ECM. A flag is signaled in the bitstream to indicate whether multi-hypothesis intra TMP mode is applied to the CU.

In yet another embodiment, more than one multiple hypothesis intra TMP method in the above section is used in combination with the current intra TMP mode in the ECM. A flag is first signaled in the bitstream to indicate whether multiple hypothesis intra TMP modes are applied. An index is then signaled to indicate which of the multiple hypothesis intra TMP methods is applied to the CU.

Multiple hypothesis intra TMP using linear filter model

The current intra TMP using the linear filter model can be divided into two steps. In a first step, a reference block is first obtained by template matching. In a second step, the reference block is filtered with a linear filter to generate a final prediction block, wherein the filter coefficients are derived using a template. In the current method, only one reference block is used, i.e., only one hypothesis, which is inefficient.

In order to improve the prediction quality of intra TMP using a linear filter model, multiple hypotheses are introduced in the first step. Instead of obtaining only one reference block, a plurality of reference blocks are obtained and used to generate a fusion block. It should be noted that the present disclosure is not limited to the method of generating the fusion block. The fusion block is then used as input to the second step.

For the derivation of the linear filter coefficients, the templates of these reference blocks are fused in the same way as the reference blocks, and the fused templates and the templates of the current block are used to derive the filter coefficients.

Intra TMP of chrominance components

In the current ECM, intra TMP is enabled only for the luma component and not for the chroma component. To take advantage of the intra-frame TMP of the chrominance components, a method of enabling the intra-frame TMP for the chrominance components is proposed.

In one and another embodiment, for inter or intra slices that disable the dual tree, the luma and chroma components share the same partition tree, template matching is performed using only templates for the luma components to identify the block vector of the current block. The reference block of the chrominance component is a co-located block of the reference block of the luminance component. In this embodiment, the luma and chroma coded blocks share the same intra TMP flag to indicate the use of intra TMP mode. The luminance component and the chrominance component are a luminance coding block and a chrominance coding block, respectively.

In yet another embodiment, for inter or intra slices that disable the dual tree, the luma component and the chroma components share the same partition tree, and template matching is performed using templates of the luma component and the two chroma components to identify the block vector of the current block. The total template matching cost is a weighted result of the template matching costs of the luminance component and the chrominance component. In this embodiment, the luma and chroma coded blocks share the same intra TMP flag to indicate the use of intra TMP mode.

cost_total＝cost_Y+ω×(cost_Cb+cost_Cr)

In yet another embodiment, for inter-or intra-slices where dual trees are disabled, the luma and chroma components share the same partition tree, and the template matching process of the luma and two chroma coding blocks is performed separately. The two chroma coding blocks share the same template matching process, i.e., the template matching costs are calculated by summing the template matching costs of the Cb and Cr components. In this embodiment, the luma and chroma coded blocks share the same intra TMP flag to indicate the use of intra TMP mode.

cost_chroma＝cost_Cb+cost_Cr

In yet another embodiment, the template matching process for the luma component and the chroma component share the same partition tree, Y, U and the V-encoded block, all occur separately for inter-or intra-slices that disable the dual-tree. In this embodiment, the luma and chroma coded blocks share the same intra TMP flag to indicate the use of intra TMP mode.

In yet another embodiment, the U and V components are separately template matching searched for dual tree enabled intra slices. A flag is signaled to indicate whether intra TMP is enabled for the chrominance component.

In yet another embodiment, the U and V components are separately template matching searched for dual tree enabled intra slices. One flag is signaled to indicate whether intra TMP is enabled for the U component and another flag is signaled to indicate whether intra TMP is enabled for the V component.

In yet another embodiment, for dual-tree enabled intra slices, a template matching search is performed on the U and V components jointly, i.e., the total template matching cost is the sum of the template matching costs of the U and V components. A flag is signaled to indicate whether intra TMP is enabled for the chrominance component.

Combination of cross-component prediction and chroma component intra TMP

In the current ECM, to take advantage of cross-component correlation, CCLMs and CCCM are used to predict chroma blocks using co-located reconstructed luma blocks. While intra TMP exploits non-local correlations. To exploit both cross-component and non-local dependencies, it is proposed to combine cross-component prediction and chroma component intra TMP. The current chroma block is denoted B, the template for the current chroma block is denoted T, the co-located luma block is denoted B _Y, and the template for the co-located luma block is denoted T _Y.

In one example method of combining cross-component prediction and intra TMP, the proposed method can be divided into several steps as described below.

In a first step, intra template matching is performed to generate reference blocks and reference templates, denoted as P _intraTMP and T _intraTMP. Here, the reference templates include a reference chromaticity template and a reference luminance template.

In the second step, parameters of the CCLM or CCCM are derived using the template T of the current chroma block, the reference chroma template T _intraTMP, and the template T _Y of the co-located luma block obtained in the first step. The CCLM or CCCM model is denoted as F (·) and the following equations are solved to derive the parameters of CCLM or CCCM.

T=T_intraTMP+F(T_Y)

In a third step, the parameters derived in the second step are applied to the co-located luma block to generate the final prediction B _pred for the current chroma block, as follows.

B_pred＝P_intraTMP+F(B_Y)

Combination of intra and intra TMP using linear filter model

In the current combination of intra and intra TMP, the intra prediction block and the intra TMP block are weighted to generate the final prediction block without further refinement. In this disclosure, it is proposed to refine weighted prediction of intra and intra TMPs using a linear filter model. The proposed method can be performed as follows.

In a first step, weighted prediction is generated by weighting the intra prediction block and the intra TMP block.

In a second step, coefficients of the linear filter are derived from templates, including templates of the current block, intra prediction templates and reference templates. An intra prediction template is generated by intra prediction using the current intra mode. The intra-prediction template is fused with a reference template to generate a fused template. The filter coefficients are derived by solving a least squares equation using the fusion template and the current template.

In a third step, the filter coefficients resulting from the second step are applied to the weighted prediction to generate a final prediction.

In this disclosure, several embodiments are presented to apply the proposed combination of intra and intra TMP using a linear filter model.

In one embodiment, the proposed method is always applied to weighted prediction of intra and intra TMPs. No change of syntax elements is required.

In a further embodiment, the use of the proposed method is signaled by a flag. If the flag is 1, the linear filter model is applied, otherwise the linear filter model is not applied. On the encoder side, RDO is performed to decide whether to enable the linear filter model for the block.

In one or another embodiment, the proposed combination of intra and intra TMP using a linear filter model can be applied to both the luminance component and the chrominance component.

Fig. 5 illustrates a computing environment (or computing device) 1610 coupled to a user interface 1650. In some embodiments, computing device 1610 may perform any of the various methods or processes (e.g., decoding/encoding methods or processes) described above according to various examples of the disclosure. The computing environment 1610 may be part of a data processing server. The computing environment 1610 includes a processor 1620, a memory 1630, and an input/output (I/O) interface 1640.

Processor 1620 generally controls overall operation of computing environment 1610, such as operations associated with display, data acquisition, data communication, and image processing. Processor 1620 may include one or more processors for executing instructions to perform all or some of the steps of the methods described above. Further, the processor 1620 may include one or more modules that facilitate interactions between the processor 1620 and other components. The processor may be a Central Processing Unit (CPU), microprocessor, single-chip microcomputer, graphics Processing Unit (GPU), or the like.

Memory 1630 is configured to store various types of data to support the operation of computing environment 1610. The memory 1630 may include predetermined software 1632. Examples of such data include instructions, video data sets, image data, and the like for any application or method operating on computing environment 1610. The memory 1630 may be implemented using any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

I/O interface 1640 provides an interface between processor 1620 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I/O interface 1640 may be coupled with an encoder and decoder.

Fig. 14 is a flowchart illustrating a video decoding method according to an example of the present disclosure. In particular, fig. 14 shows a method used in a multiple hypothesis intra TMP using a linear filter model that improves the prediction quality of the intra TMP using the linear filter model.

In step 1401, the processor 1620 may obtain a plurality of reference blocks of the current block based on the template matching on the decoder side.

In step 1402, the processor 1620 may obtain a fused reference block based on the plurality of reference blocks.

In step 1403, the processor 1620 may obtain a final prediction block for the current block based on the fused reference block and the linear filter.

In some examples, the final prediction block may be obtained by filtering the fused reference block using a linear filter associated with the plurality of filter coefficients.

In one or more examples, filtering the fused reference block using the linear filters associated with the plurality of filter coefficients may be implemented using steps including obtaining a fused template based on the plurality of reference blocks, deriving the plurality of filter coefficients based on the fused template and a current template of the current block, and obtaining a final prediction block of the current block by filtering the fused reference block using the linear filters associated with the plurality of filter coefficients. For example, the plurality of filter coefficients may be derived by minimizing a Mean Square Error (MSE), which may be performed by calculating an autocorrelation matrix that fuses the template input and the current template output. LDL decomposition is performed on the autocorrelation matrix and the final filter coefficients are calculated using a back-substitution method.

In some examples, the fusion template may be obtained by obtaining a plurality of templates of the plurality of reference blocks based on the plurality of reference blocks, and obtaining the fusion template based on the plurality of templates of the plurality of reference blocks. In one or more examples, the fusion template may be obtained in the same manner as a fused reference block is obtained based on multiple reference blocks. For example, for the derivation of linear filter coefficients, templates of these reference blocks are fused in the same way as the reference blocks, and the filter coefficients are obtained using the fused templates and the templates of the current block.

Fig. 15 is a flowchart showing a video encoding method corresponding to the video decoding method shown in fig. 14.

In step 1501, the processor 1620 may obtain a plurality of reference blocks for the current block based on the template matching on the encoder side.

In step 1502, the processor 1620 may obtain a fused reference block based on the plurality of reference blocks.

In step 1503, the processor 1620 may obtain a final prediction block for the current block based on the fused reference block and the linear filter.

In some examples, the final prediction block may be obtained by filtering the fused reference block using a linear filter associated with the plurality of filter coefficients.

In one or more examples, filtering the fused reference block using the linear filters associated with the plurality of filter coefficients may be implemented using steps including obtaining a fused template based on the plurality of reference blocks, deriving the plurality of filter coefficients based on the fused template and a current template of the current block, and obtaining a final prediction block of the current block by filtering the fused reference block using the linear filters associated with the plurality of filter coefficients. For example, the plurality of filter coefficients may be derived by minimizing a Mean Square Error (MSE), which may be performed by calculating an autocorrelation matrix that fuses the template input and the current template output. LDL decomposition is performed on the autocorrelation matrix and the final filter coefficients are calculated using a back-substitution method.

In some examples, the fusion template may be obtained by obtaining a plurality of templates of the plurality of reference blocks based on the plurality of reference blocks, and obtaining the fusion template based on the plurality of templates of the plurality of reference blocks. In one or more examples, the fusion template may be obtained in the same manner as a fused reference block is obtained based on multiple reference blocks. For example, for the derivation of linear filter coefficients, templates of these reference blocks are fused in the same way as the reference blocks, and the filter coefficients are obtained using the fused templates and the templates of the current block.

Fig. 16 is a flowchart illustrating a video decoding method according to an example of the present disclosure. Specifically, fig. 16 shows a method used for the chrominance components in the intra TMP.

In step 1601, processor 1620 may obtain at the decoder side at least one of a luma template for a luma Coded Block (CB) of a template matching or a chroma template of a chroma CB, wherein the chroma templates include a first chroma template of a first chroma CB and a second chroma template of a second chroma CB, wherein the templates may be used for the template matching.

In step 1602, the processor 1620 may calculate a template matching cost between the current template of the current CB and at least one of the luma template, the first chroma template or the second chroma template.

In step 1603, the processor 1620 may obtain a final prediction block for the current CB based on the template matching costs.

In some examples, step 1602 may be implemented by calculating a template matching cost between a luma template and a current template of a current CB when a dual tree is disabled in an encoded CTU including the current CB for an inter-frame band or an intra-frame band, i.e., the luma CB, the first chroma CB, and the second chroma CB share the same partition tree, and obtaining a final prediction block according to the template matching cost.

In some examples, processor 1620 may further receive a flag indicating whether intra Template Matching Prediction (TMP) is enabled for luma CB and chroma CB when the dual tree is disabled for inter-or intra-bands, and obtain a luma template for luma CB based on the template matching in response to determining that the flag indicates that intra-TMP is enabled for luma CB and chroma CB.

In some examples, steps 1602 and 1603 may be implemented by obtaining a luma template cost between the luma template and a current template of the current CB, a first chroma template cost between the first chroma template and the current template, and a second chroma template cost between the second chroma template and the current template when the dual tree is disabled for the inter-frame or intra-frame band, calculating a total template matching cost as a template matching cost by weighting the luma template cost, the first chroma template cost, and the second chroma template cost, and obtaining the final prediction block according to the total template matching cost. For example, the total template matching cost may be obtained by:

cost_total＝cost_Y+ω×(cost_Cb+cost_Cr)

Where ω may be a predefined empirical value.

In some examples, processor 1620 may further receive a flag indicating whether intra-TMP is enabled for luma CB and first and second chroma CBs when the dual tree is disabled for inter-or intra-bands, and obtain a luma template for luma CB, a first chroma template for first chroma CB, and a second chroma template for second chroma CB in response to determining that the flag indicates that intra-TMP is enabled for luma CB and first and second chroma CBs. In one example, the luma and chroma coded blocks may share the same flag, i.e., an intra TMP flag, to indicate the use of intra TMP mode.

In some examples, steps 1602 and 1603 may be implemented by obtaining a luma template cost between the luma template and the current template of the current CB, a first chroma template cost between the first chroma template and the current template, and a second chroma template cost between the second chroma template and the current template when the dual tree is disabled for the inter-frame or intra-frame band, obtaining a chroma template cost by adding the first chroma template cost to the second chroma template cost, and obtaining a final prediction block from the luma template cost and the chroma template cost.

In one or more examples, processor 1620 may further receive a flag indicating whether intra-TMP is enabled for luma CB and chroma CB when the dual tree is disabled for inter-or intra-bands, and obtain a luma template for luma CB, a first chroma template for a first chroma CB, and a second chroma template for a second chroma CB in response to determining that the flag indicates that intra-TMP is enabled for luma CB and chroma CB. In one example, the luma and chroma coded blocks may share the same flag, i.e., an intra TMP flag, to indicate the use of intra TMP mode.

In some examples, steps 1602 and 1603 may be implemented by obtaining a luma template matching cost between a luma template and a current template of a current CB and obtaining a luma prediction block based on the luma template matching cost, obtaining a first chroma template matching cost between a first chroma template and the current template and obtaining a first chroma prediction block based on the first chroma template matching cost, obtaining a second chroma template matching cost between a second chroma template and the current template and obtaining a second chroma prediction block based on the second chroma template matching cost, and obtaining a final prediction block based on the luma prediction block, the first chroma prediction block, and the second chroma prediction block when the dual tree is disabled for the inter-frame or intra-frame band. In these examples, the template matching process for Y, U, V encoded blocks is performed separately.

In one or more examples, processor 1620 may further receive a flag indicating whether intra-TMP is enabled for luma CB and chroma CB when the dual tree is disabled for inter-or intra-bands, and obtain a luma template for luma CB, a first chroma template for a first chroma CB, and a second chroma template for a second chroma CB in response to determining that the flag indicates that intra-TMP is enabled for luma CB and chroma CB. In these examples, the luma and chroma coded blocks share the same flag, i.e., the intra TMP flag, to indicate the use of intra TMP mode.

In some examples, steps 1602 and 1603 may be implemented by obtaining a first chroma template matching cost between a first chroma template and a current template of a current CB and obtaining a first chroma prediction block based on the first chroma template matching cost, obtaining a second chroma template matching cost between a second chroma template and the current template and obtaining a second chroma prediction block based on the second chroma template matching cost, and obtaining a final prediction block based on the first chroma prediction block and the second chroma prediction block when dual trees are enabled for intra slices. In these examples, the template matching search is performed separately for the chrominance components (i.e., the U component and the V component).

In one or more examples, processor 1620 may further receive a flag indicating whether intra Template Matching Prediction (TMP) is enabled for the first chroma CB and the second chroma CB when the dual tree is enabled for the intra stripe, and obtain the first chroma template of the first chroma CB and the second chroma template of the second chroma CB in response to determining that the flag indicates that intra TMP is enabled for the first chroma CB and the second chroma CB.

In one or more examples, processor 1620 may further receive a first flag indicating whether intra Template Matching Prediction (TMP) is enabled for a first chroma CB and a second flag indicating whether intra TMP is enabled for a second chroma CB when dual trees are enabled for intra stripes, and obtain a first chroma template for the first chroma CB and a second chroma template for the second chroma CB in response to determining that the first flag indicates that intra TMP is enabled for the first chroma CB and that the second flag indicates that intra TMP is enabled for the second chroma CB.

In some examples, steps 1602 and 1603 may be implemented by obtaining a first chroma template matching cost between a first chroma template and a current template of a current CB, obtaining a second chroma template matching cost between a second chroma template and the current template, calculating a chroma template matching cost by adding the first chroma template matching cost to the second chroma template matching cost, and obtaining a final prediction block based on the chroma template matching cost when dual trees are enabled for intra slices. In these examples, the template matching search is performed jointly for the chroma components (i.e., the U component and the V component), and the total template matching cost is the sum of the template matching costs of the U component and the V component.

In one or more examples, processor 1620 may further receive a flag indicating whether TMP is enabled for the first chroma CB and the second chroma CB when dual trees are enabled for the intra-band, and obtain the first chroma template of the first chroma CB and the second chroma template of the second chroma CB in response to determining that the flag indicates that intra-TMP is enabled for the first chroma CB and the second chroma CB.

Fig. 17 is a flowchart showing a video encoding method corresponding to the video decoding method shown in fig. 16.

In step 1701, the processor 1620 may obtain, on the encoder side, a luma template of luma CBs or a chroma template of chroma CBs for template matching, wherein the chroma templates include a first chroma template of a first chroma CB and a second chroma template of a second chroma CB, wherein the templates may be used for template matching.

In step 1702, the processor 1620 may calculate a template matching cost between the current template of the current CB and at least one of the luma template, the first chroma template or the second chroma template.

In step 1703, the processor 1620 may obtain a final predicted block for the current CB based on the template matching costs.

In some examples, step 1702 may be implemented by steps including obtaining a luma template for a luma CB based on template matching when a dual tree is disabled in an encoded CTU that includes the current CB for an inter-frame band or an intra-frame band, calculating a template matching cost between the luma template and the current template for the current CB, and obtaining a final prediction block according to the template matching cost.

In one or more examples, processor 1620 may further signal a flag indicating whether intra Template Matching Prediction (TMP) is enabled for luma CB and chroma CB when dual trees are disabled for inter-or intra-bands, and obtain a luma template for luma CB based on template matching in response to the flag indicating that intra-TMP is enabled for luma CB and chroma CB.

In some examples, steps 1702 and 1703 may be implemented by obtaining a luma template cost between the luma template and the current template of the current CB, a first chroma template cost between the first chroma template and the current template, and a second chroma template cost between the second chroma template and the current template when the dual tree is disabled for the inter-frame or intra-frame band, calculating a total template matching cost by weighting the luma template cost, the first chroma template cost, and the second chroma template cost, and obtaining a final prediction block from the total template matching cost. For example, the total template matching cost may be obtained by:

cost_total＝cost_Y+ω×(cost_Cb+cost_Cr)

Where ω may be a predefined empirical value.

In one or more examples, processor 1620 may further signal a flag indicating whether intra TMP is enabled for luma CB and first and second chroma CBs when dual trees are disabled for inter-or intra-bands, and obtain a luma template for luma CB, a first chroma template for first chroma CB, and a second chroma template for second chroma CB in response to the flag indicating that intra TMP is enabled for luma CB and first and second chroma CBs. In one example, the luma and chroma coded blocks may share the same flag, i.e., an intra TMP flag, to indicate the use of intra TMP mode.

In some examples, steps 1702 and 1703 may be implemented by obtaining a luma template cost between the luma template and the current template of the current CB, a first chroma template cost between the first chroma template and the current template, and a second chroma template cost between the second chroma template and the current template when the dual tree is disabled for the inter-frame or intra-frame band, obtaining a chroma template cost by adding the first chroma template cost to the second chroma template cost, and obtaining a final prediction block from the luma template cost and the chroma template cost.

In one or more examples, processor 1620 may further signal a flag indicating whether intra TMP is enabled for luma CB and chroma CB when dual trees are disabled for inter-or intra-bands, and obtain a luma template for luma CB, a first chroma template for first chroma CB, and a second chroma template for second chroma CB in response to the flag indicating that intra TMP is enabled for luma CB and chroma CB. In one example, the luma and chroma coded blocks may share the same flag, i.e., an intra TMP flag, to indicate the use of intra TMP mode.

In some examples, steps 1702 and 1703 may be implemented by obtaining a luma template matching cost between a luma template and a current template of a current CB and obtaining a luma prediction block based on the luma template matching cost, obtaining a first chroma template matching cost between a first chroma template and the current template and obtaining a first chroma prediction block based on the first chroma template matching cost, obtaining a second chroma template matching cost between a second chroma template and the current template and obtaining a second chroma prediction block based on the second chroma template matching cost, and obtaining a final prediction block based on the luma prediction block, the first chroma prediction block, and the second chroma prediction block when the dual tree is disabled for either the inter-frame band or the intra-frame band. In these examples, the template matching process for Y, U, V encoded blocks is performed separately.

In one or more examples, processor 1620 may further signal a flag indicating whether intra TMP is enabled for luma CB and chroma CB when dual trees are disabled for inter-or intra-bands, and obtain a luma template for luma CB, a first chroma template for first chroma CB, and a second chroma template for second chroma CB in response to the flag indicating that intra TMP is enabled for luma CB and chroma CB. In one example, the luma and chroma coded blocks may share the same flag, i.e., an intra TMP flag, to indicate the use of intra TMP mode.

In some examples, steps 1702 and 1703 may be implemented by obtaining a first chroma template matching cost between a first chroma template and a current template of a current CB and obtaining a first chroma prediction block based on the first chroma template matching cost, obtaining a second chroma template matching cost between a second chroma template and the current template and obtaining a second chroma prediction block based on the second chroma template matching cost, and obtaining a final prediction block based on the first chroma prediction block and the second chroma prediction block when dual trees are enabled for intra slices. In these examples, the template matching search is performed separately for the chrominance components (i.e., the U component and the V component).

In one or more examples, processor 1620 may further signal a flag indicating whether intra Template Matching Prediction (TMP) is enabled for the first chroma CB and the second chroma CB when dual trees are enabled for the intra stripes, and obtain the first chroma template of the first chroma CB and the second chroma template of the second chroma CB in response to the flag indicating that intra TMP is enabled for the first chroma CB and the second chroma CB.

In one or more examples, processor 1620 may further signal a first flag indicating whether intra Template Matching Prediction (TMP) is enabled for a first chroma CB and a second flag indicating whether intra TMP is enabled for a second chroma CB when dual trees are enabled for intra stripes, and obtain a first chroma template for the first chroma CB and a second chroma template for the second chroma CB in response to the first flag indicating that intra TMP is enabled for the first chroma CB and determining that the second flag indicates that intra TMP is enabled for the second chroma CB.

In some examples, steps 1702 and 1703 may be implemented by obtaining a first chroma template matching cost between a first chroma template and a current template of a current CB, obtaining a second chroma template matching cost between a second chroma template and the current template, calculating a chroma template matching cost by adding the first chroma template matching cost to the second chroma template matching cost, and obtaining a final prediction block based on the chroma template matching cost when dual trees are enabled for intra slices. In these examples, the template matching search is performed jointly for the chroma components (i.e., the U component and the V component), and the total template matching cost is the sum of the template matching costs of the U component and the V component.

In one or more examples, processor 1620 may further signal a flag indicating whether intra-TMP is enabled for the first chroma CB and the second chroma CB when dual trees are enabled for the intra-stripe, and obtain a first chroma template for the first chroma CB and a second chroma template for the second chroma CB in response to the flag indicating that intra-TMP is enabled for the first chroma CB and the second chroma CB.

Fig. 18 is a flowchart illustrating a video decoding method according to an example of the present disclosure. The method shown in fig. 18 is used in the combination of cross-component prediction and intra TMP of the chrominance components.

In step 1801, the processor 1620 may obtain a reference block of the current block and a reference template of the reference block based on intra template matching at the decoder side, wherein each of the current block and the reference block includes a luminance component and a chrominance component, and the reference template of the reference block includes a reference chrominance template corresponding to the chrominance component of the reference block and a reference luminance template corresponding to the co-located luminance block.

In step 1802, the processor 1620 may derive parameters of a cross-component prediction model based on a chroma template, a reference chroma template, and a reference luma template of a chroma component of the current block at a decoder side.

In some examples, the cross-component prediction model may include one of a convolved cross-component model (CCCM) or a cross-component linear model (CCLM).

In step 1803, processor 1620 may obtain a final prediction of the current chroma block at the decoder side by applying parameters of the cross-component prediction model to the co-located luma block.

Fig. 19 is a flowchart illustrating a video encoding method corresponding to the method illustrated in fig. 18 according to an example of the present disclosure.

In step 1901, the processor 1620 may obtain a reference block of the current block and a reference template of the reference block based on intra template matching at the encoder side, wherein each of the current block and the reference block includes a luminance component and a chrominance component, and the reference template of the reference block includes a reference chrominance template corresponding to the chrominance component of the reference block and a reference luminance template corresponding to the co-located luminance block.

In step 1902, the processor 1620 may derive parameters of the cross-component prediction model at the encoder side based on the chroma templates of the chroma components of the current block, the reference chroma templates, and the reference luma templates.

In some examples, the cross-component prediction model may include one of a convolved cross-component model (CCCM) or a cross-component linear model (CCLM).

In step 1903, processor 1620 may obtain a final prediction of the current chroma block at the encoder side by applying parameters of the cross-component prediction model to the co-located luma block.

Fig. 20 is a flowchart illustrating a video decoding method according to an example of the present disclosure. The method shown in fig. 20 can be used in a combination of intra and intra TMP using a linear filter model.

In step 2001, the processor 1620 may generate weighted prediction at the decoder side by weighting the intra prediction block and the intra Template Matching Prediction (TMP) block. For example, the processor 1620 can apply weights to the intra-prediction block and the intra-TMP block and then generate a weighted prediction based on the weighted sum of the intra-prediction block and the intra-TMP block.

In step 2002, the processor 1620 may obtain coefficients of the linear filter from a plurality of templates including a current template of the current block, an intra prediction template, and a reference template of the reference block at the decoder side.

In some examples, coefficients of the linear filter may be obtained by obtaining an intra prediction template by intra prediction using the current intra mode, obtaining a fused template by fusing the intra prediction template and a reference template, and solving a least squares equation using the fused template and the current template to obtain coefficients of the linear filter.

In some examples, processor 1620 may receive a flag indicating whether to apply the linear filter model and, in response to determining that the flag indicates to apply the linear filter model, derive coefficients of the linear filter from a plurality of templates including a current template, an intra-prediction template, and a reference template of the current block.

In some examples, a combination of intra and intra TMP using a linear filter model may be used for the luminance block and the chrominance block, respectively. Specifically, the coefficients of the linear filter may be obtained by deriving the coefficients of the linear filter from a plurality of templates including a current template of the current block, an intra-prediction template, and a reference template, wherein each of the plurality of templates includes one of a luminance component or a chrominance component.

In step 2003, the processor 1620 may obtain a final prediction by applying coefficients of the linear filter to the weighted prediction at the decoder side.

Fig. 21 is a flowchart illustrating a video encoding method according to an example of the present disclosure.

In step 2101, the processor 1620 may generate weighted predictions at the encoder side by weighting the intra prediction block and the intra Template Matching Prediction (TMP) block.

In step 2002, the processor 1620 may obtain coefficients of the linear filter from a plurality of templates including a current template of the current block, an intra prediction template, and a reference template of the reference block at the encoder side.

In some examples, coefficients of the linear filter may be obtained by obtaining an intra prediction template by intra prediction using the current intra mode, obtaining a fused template by fusing the intra prediction template and a reference template, and solving a least squares equation using the fused template and the current template to obtain coefficients of the linear filter.

In some examples, processor 1620 may signal a flag indicating whether to apply the linear filter model and, in response to the flag indicating that the linear filter model is applied, derive coefficients of the linear filter from a plurality of templates including the current template, the intra-prediction template, and the reference template of the current block. In one or more examples, on the encoder side, RDO is performed to decide whether to enable a linear filter model for the block.

In some examples, a combination of intra and intra TMP using a linear filter model may be used for the luminance block and the chrominance block, respectively. Specifically, the coefficients of the linear filter may be obtained by deriving the coefficients of the linear filter from a plurality of templates including a current template of the current block, an intra-prediction template, and a reference template, wherein each of the plurality of templates includes one of a luminance component or a chrominance component.

In step 2103, the processor 1620 may obtain a final prediction on the encoder side by applying coefficients of the linear filter to the weighted prediction.

In some examples, an apparatus for video encoding and decoding is provided. The apparatus includes a processor 1620 and a memory 1640 configured to store instructions executable by the processor, wherein the processor, when executing the instructions, is configured to perform any of the methods as shown in fig. 14-21.

In an embodiment, there is also provided a non-transitory computer readable storage medium including, for example, a plurality of programs in memory 1630 executable by processor 1620 in computing environment 1610 for performing the above-described methods, and/or storing a bitstream generated by the above-described encoding method or a bitstream to be decoded by the above-described decoding method. In one example, a plurality of programs may be executed by processor 1620 in computing environment 1610 to receive (e.g., from video encoder 20 in fig. 2) a bitstream or data stream comprising encoded video information (e.g., video blocks representing encoded video frames, and/or associated one or more syntax elements, etc.), and may also be executed by processor 1620 in computing environment 1610 to perform the above-described decoding method in accordance with the received bitstream or data stream. In another example, a plurality of programs may be executed by the processor 1620 in the computing environment 1610 for performing the encoding methods described above to encode video information (e.g., video blocks representing video frames, and/or associated one or more syntax elements, etc.) into a bitstream or data stream, and may also be executed by the processor 1620 in the computing environment 1610 for transmitting the bitstream or data stream (e.g., to the video decoder 30 in fig. 3). Alternatively, a non-transitory computer readable storage medium may have stored therein a bitstream or data stream comprising encoded video information (e.g., video blocks representing encoded video frames, and/or associated one or more syntax elements, etc.) generated by an encoder (e.g., video encoder 20 of fig. 2) using, for example, the encoding methods described above, for use by a decoder (e.g., video decoder 30 of fig. 3) in decoding video data. The non-transitory computer readable storage medium may be, for example, ROM, random-access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

In an embodiment, a bitstream generated by the above-described encoding method or a bitstream to be decoded by the above-described decoding method is provided. In an embodiment, a bitstream is provided that includes encoded video information generated by the above-described encoding method or encoded video information to be decoded by the above-described decoding method.

In an embodiment, a computing device is also provided that includes one or more processors (e.g., processor 1620), and a non-transitory computer-readable storage medium or memory 1630 having stored therein a plurality of programs executable by the one or more processors, wherein the one or more processors are configured to perform the above-described methods when executing the plurality of programs.

In an embodiment, there is also provided a computer program product having instructions for storing or transmitting a bitstream comprising encoded video information generated by the above-described encoding method or encoded video information to be decoded by the above-described decoding method. In an embodiment, a computer program product is also provided that includes a plurality of programs, e.g., in memory 1630, executable by processor 1620 in computing environment 1610 for performing the methods described above. For example, the computer program product may include a non-transitory computer readable storage medium.

In an embodiment, the computing environment 1610 may be implemented by one or more ASICs, DSPs, digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the methods described above.

In an embodiment, there is also provided a method of storing a bitstream comprising storing the bitstream on a digital storage medium, wherein the bitstream comprises encoded video information generated by the above-described encoding method or encoded video information to be decoded by the above-described decoding method.

In an embodiment, a method for transmitting a bitstream generated by the above encoder is also provided. In an embodiment, a method for receiving a bitstream to be decoded by the decoder described above is also provided.

The description of the present disclosure is presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosure. Many modifications, variations and alternative embodiments will come to mind to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.

The order of steps of the method according to the present disclosure is intended to be illustrative only, unless otherwise specifically stated, and the steps of the method according to the present disclosure are not limited to the above-described order, but may be changed according to actual circumstances. Furthermore, at least one of the steps of the method according to the present disclosure may be adjusted, combined or pruned as actually needed.

The examples were chosen and described in order to explain the principles of the present disclosure and to enable others skilled in the art to understand the disclosure for various embodiments and with various modifications as are suited to the particular use contemplated. Therefore, it is to be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and that modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims (50)

1. A video decoding method, comprising: Obtaining, by the decoder, a plurality of reference blocks for the current block based on template matching; Obtaining, by the decoder, a fused reference block based on the multiple reference blocks; and The decoder obtains a final prediction block of the current block based on the fused reference block and a linear filter. 2. The method of claim 1 , wherein obtaining a final prediction block of the current block based on the fused reference block and the linear filter further comprises: A final prediction block of the current block is obtained by filtering the fused reference block using the linear filter associated with a plurality of filter coefficients. 3. The method of claim 2 , wherein obtaining the final prediction block of the current block by filtering the fused reference block using the linear filter associated with a plurality of filter coefficients comprises: obtaining a fusion template based on the multiple reference blocks; Obtaining the plurality of filter coefficients based on the fused template and the current template of the current block; and A final prediction block of the current block is obtained by filtering the fused reference block using the linear filter associated with the plurality of filter coefficients. 4. The method of claim 3, wherein obtaining a fusion template based on the multiple reference blocks comprises: obtaining a plurality of templates of the plurality of reference blocks; and The fused template is obtained based on the multiple templates of the multiple reference blocks. 5. A video encoding method, comprising: The encoder obtains a plurality of reference blocks for the current block based on template matching; Obtaining, by the encoder, a fused reference block based on the multiple reference blocks; and The encoder obtains a final prediction block of the current block based on the fused reference block and a linear filter. 6. The method of claim 5 , wherein obtaining a final prediction block of the current block based on the fused reference block and a linear filter further comprises: A final prediction block of the current block is obtained by filtering the fused reference block using the linear filter associated with a plurality of filter coefficients. 7. The method of claim 6 , wherein obtaining the final prediction block of the current block by filtering the fused reference block using the linear filter associated with a plurality of filter coefficients comprises: obtaining a fusion template based on the multiple reference blocks; Obtaining the plurality of filter coefficients based on the fused template and the current template of the current block; and A final prediction block of the current block is obtained by filtering the fused reference block using the linear filter associated with the plurality of filter coefficients. 8. The method of claim 7, wherein obtaining a fusion template based on the plurality of reference blocks comprises: obtaining a plurality of templates of the plurality of reference blocks; and The fused template is obtained based on the multiple templates of the multiple reference blocks. 9. A video decoding method, comprising: Obtaining, by a decoder, at least one of a luma template of a luma coding block (CB) or a chroma template of a chroma CB for template matching, wherein the chroma templates include a first chroma template of the first chroma CB and a second chroma template of the second chroma CB; calculating, by the decoder, a template matching cost between a current template of a current CB and at least one of the luma template or the chroma template; and The decoder obtains the final prediction block of the current CB based on the template matching cost. 10. The method of claim 9, wherein calculating the template matching cost between the current template and at least one of the luma template or the chroma template comprises: Calculate the template matching cost between the luminance template and the current template of the current CB. 11. The method of claim 9, wherein calculating the template matching cost between the current template and at least one of the luma template or the chroma template comprises: Obtaining a luma template cost between the luma template and the current template of the current CB, a first chroma template cost between the first chroma template and the current template, and a second chroma template cost between the second chroma template and the current template; and A total template matching cost is calculated as the template matching cost by weighting the luminance template cost, the first chrominance template cost, and the second chrominance template cost. 12. The method of claim 9, wherein calculating the template matching cost between the current template and at least one of the luma template or the chroma template comprises: Obtaining a luma template cost between the luma template and the current template of the current CB, a first chroma template cost between the first chroma template and the current template, and a second chroma template cost between the second chroma template and the current template; Obtaining a chroma template cost by adding the first chroma template cost and the second chroma template cost; and The template matching cost is obtained according to the luminance template cost and the chrominance template cost. 13. The method of claim 9, wherein calculating the template matching cost between the current template and at least one of the luma template or the chroma template comprises: Obtaining a luminance template matching cost between the luminance template and the current template of the current CB; Obtaining a first chroma template matching cost between the first chroma template and the current template; and Obtaining a second chroma template matching cost between the second chroma template and the current template; Wherein, obtaining the final prediction block of the current CB based on the template matching cost includes: Obtaining a luminance prediction block based on the luminance template matching cost; Obtain a first chroma prediction block based on the first chroma template matching cost; Obtaining a second chroma prediction block based on the second chroma template matching cost; and The final prediction block is obtained based on the luma prediction block, the first chroma prediction block, and the second chroma prediction block. 14. The method of any one of claims 10-13, wherein dual tree is disabled in a coding tree unit (CTU) including the current CB for an inter slice or an intra slice, and the luma CB, the first chroma CB, and the second chroma CB share the same partition tree. 15. The method of any one of claims 10 to 13, wherein obtaining at least one of a luma template of a luma coding block or a chroma template of a chroma CB for template matching comprises: receiving a flag indicating whether intra template matching prediction (TMP) is enabled for the luma CB and the chroma CB; and In response to determining that the flag indicates that the intra TMP is enabled for the luma CB and the chroma CB, a luma template of the luma CB is obtained, or a luma template of the luma CB and a chroma template of the chroma CB are obtained. 16. The method of claim 9, wherein calculating the template matching cost between the current template of the current CB and at least one of the luma template or the chroma template comprises: Obtaining a first chroma template matching cost between the first chroma template and the current template of the current CB; and Obtaining a second chroma template matching cost between the second chroma template and the current template; Wherein, obtaining the final prediction block of the current CB based on the template matching cost includes: Obtain a first chroma prediction block based on the first chroma template matching cost; Obtaining a second chroma prediction block based on the second chroma template matching cost; and The final prediction block is obtained based on the first chroma prediction block and the second chroma prediction block. 17. The method of claim 16 , wherein obtaining at least one of a luma template of a luma coding block or a chroma template of a chroma CB for template matching comprises: receiving a first flag indicating whether intra template matching prediction (TMP) is enabled for the first chroma CB and a second flag indicating whether the intra TMP is enabled for the second chroma CB; and In response to determining that the first flag indicates that the intra TMP is enabled for the first chroma CB and determining that the second flag indicates that the intra TMP is enabled for the second chroma CB, a first chroma template for the first chroma CB and a second chroma template for the second chroma CB are obtained. 18. The method of claim 9, wherein calculating the template matching cost between the current template of the current CB and at least one of the luma template or the chroma template comprises: Obtaining a first chroma template matching cost between the first chroma template and the current template of the current CB; Obtaining a second chroma template matching cost between the second chroma template and the current template; and Calculating a chroma template matching cost by adding the first chroma template matching cost and the second chroma template matching cost; Wherein, obtaining the final prediction block of the current CB based on the template matching cost includes: The final prediction block is obtained based on the chroma template matching cost. 19. The method of any one of claims 16-18, wherein dual-tree is enabled in a coding tree unit (CTU) including the current CB for intra slices. 20. The method of claim 18, wherein obtaining at least one of a luma template of a luma coding block or a chroma template of a chroma CB for template matching comprises: receiving a flag indicating whether intra TMP is enabled for the chroma CB; and In response to determining that the flag indicates that the intra TMP is enabled for the chroma CB, a first chroma template of the first chroma CB and a second chroma template of the second chroma CB are obtained. 21. A video encoding method, comprising: Obtaining, by an encoder, at least one of a luma template of a luma coding block (CB) or a chroma template of a chroma CB for template matching, wherein the chroma templates include a first chroma template of the first chroma CB and a second chroma template of the second chroma CB; calculating, by the encoder, a template matching cost between a current template of a current CB and at least one of the luma template or the chroma template; and The encoder obtains a final prediction block of the current CB based on the template matching cost. 22. The method of claim 21 , wherein calculating a template matching cost between the current template and at least one of the luma template or the chroma template further comprises: Calculate the template matching cost between the luminance template and the current template of the current CB. 23. The method of claim 21 , wherein calculating a template matching cost between the current template and at least one of the luma template or the chroma template further comprises: Obtaining a luma template cost between the luma template and the current template of the current CB, a first chroma template cost between the first chroma template and the current template, and a second chroma template cost between the second chroma template and the current template; and A total template matching cost is calculated as the template matching cost by weighting the luminance template cost, the first chrominance template cost, and the second chrominance template cost. 24. The method of claim 21 , wherein calculating a template matching cost between a current template and at least one of the luma template or the chroma template comprises: Obtaining a luma template cost between the luma template and the current template of the current CB, a first chroma template cost between the first chroma template and the current template, and a second chroma template cost between the second chroma template and the current template; Obtaining a chroma template cost by adding the first chroma template cost and the second chroma template cost; and The template matching cost is obtained according to the luminance template cost and the chrominance template cost. 25. The method of claim 21 , wherein calculating a template matching cost between a current template and at least one of the luma template or the chroma template comprises: Obtaining a luminance template matching cost between the luminance template and the current template of the current CB; Obtaining a first chroma template matching cost between the first chroma template and the current template; and Obtaining a second chroma template matching cost between the second chroma template and the current template; and Wherein, obtaining the final prediction block of the current CB based on the template matching cost includes: Obtaining a luminance prediction block based on the luminance template matching cost; Obtain a first chroma prediction block based on the first chroma template matching cost; Obtaining a second chroma prediction block based on the second chroma template matching cost; and The final prediction block is obtained based on the luma prediction block, the first chroma prediction block, and the second chroma prediction block. 26. The method of any one of claims 21-25, wherein dual trees are disabled in a coding tree unit (CTU) including the current CB for an inter slice or an intra slice, and the luma CB, the first chroma CB, and the second chroma CB share the same partition tree. 27. The method according to any one of claims 21 to 25, wherein obtaining at least one of a luma template of a luma CB or a chroma template of a chroma CB for template matching comprises: signaling a flag indicating whether intra TMP is enabled for the luma CB and the chroma CB; and In response to signaling the flag indicating that the intra TMP is enabled for the luma CB and the chroma CB, a luma template of the luma CB is obtained, or a luma template of the luma CB and a chroma template of the chroma CB are obtained. 28. The method of claim 21 , wherein calculating a template matching cost between a current template of a current CB and at least one of the luma template or the chroma template comprises: Obtaining a first chroma template matching cost between the first chroma template and the current template of the current CB; and Obtaining a second chroma template matching cost between the second chroma template and the current template; and Wherein, obtaining the final prediction block of the current CB based on the template matching cost includes: Obtain a first chroma prediction block based on the first chroma template matching cost; Obtaining a second chroma prediction block based on the second chroma template matching cost; and The final prediction block is obtained based on the first chroma prediction block and the second chroma prediction block. 29. The method of claim 28, wherein obtaining at least one of a luma template of a luma CB or a chroma template of a chroma CB for template matching comprises: signaling a first flag indicating whether intra template matching prediction (TMP) is enabled for the first chroma CB and a second flag indicating whether the intra TMP is enabled for the second chroma CB; and In response to signaling the first flag indicating that the intra TMP is enabled for the first chroma CB and signaling the second flag indicating that the intra TMP is enabled for the second chroma CB, a first chroma template of the first chroma CB and a second chroma template of the second chroma CB are obtained. 30. The method of claim 21 , wherein calculating a template matching cost between a current template of a current CB and at least one of the luma template or the chroma template comprises: Obtaining a first chroma template matching cost between the first chroma template and the current template of the current CB; Obtaining a second chroma template matching cost between the second chroma template and the current template; and Calculating a chroma template matching cost by adding the first chroma template matching cost and the second chroma template matching cost; and Wherein, obtaining the final prediction block of the current CB based on the template matching cost includes: The final prediction block is obtained based on the chroma template matching cost. 31. The method of any one of claims 28-30, wherein dual-tree is enabled in a coding tree unit (CTU) including the current CB for intra slices. 32. The method of claim 30 , wherein obtaining at least one of a luma template of a luma CB or a chroma template of a chroma CB for template matching comprises: signaling a flag indicating whether intra TMP is enabled for the chroma CB; and In response to signaling the flag indicating that the intra TMP is enabled for the chroma CB, a first chroma template of the first chroma CB and a second chroma template of the second chroma CB are obtained. 33. A video decoding method, comprising: Obtaining, by a decoder based on intra-frame template matching, a reference block of a current block and a reference template of the reference block, wherein each of the current block and the reference block includes a luma component and a chroma component, and wherein the reference template of the reference block includes a reference chroma template corresponding to the chroma component of the reference block and a reference luma template corresponding to a co-located luma block; The decoder obtains parameters of a cross-component prediction model based on a chroma template of a chroma component of the current block, the reference chroma template, and the reference luma template; and A final prediction of the current chroma block is obtained by the decoder by applying parameters of the cross-component prediction model to the co-located luma block. 34. The method of claim 33, wherein the cross-component prediction model comprises one of the following models: a convolutional cross-component model (CCCM) or a cross-component linear model (CCLM). 35. A video encoding method, comprising: Obtaining, by an encoder, a reference block of a current block and a reference template of the reference block based on intra-frame template matching, wherein each of the current block and the reference block includes a luma component and a chroma component, and wherein the reference template of the reference block includes a reference chroma template corresponding to the chroma component of the reference block and a reference luma template corresponding to a co-located luma block; The encoder obtains parameters of a cross-component prediction model based on a chroma template of a chroma component of the current block, the reference chroma template, and the reference luma template of a co-located luma block; and A final prediction of the current chroma block is obtained by the encoder by applying parameters of the cross-component prediction model to the co-located luma block. 36. The method of claim 35, wherein the cross-component prediction model comprises one of the following models: a convolutional cross-component model (CCCM) or a cross-component linear model (CCLM). 37. A video decoding method, comprising: The decoder generates a weighted prediction by weighting the intra prediction block and the intra template matching prediction (TMP) block; Obtaining, by the decoder, coefficients of a linear filter from a plurality of templates including a current template of a current block, an intra-frame prediction template, and a reference template of a reference block; and The final prediction is obtained by the decoder by applying the coefficients of the linear filter to the weighted prediction. 38. The method of claim 37, wherein deriving coefficients of a linear filter from a plurality of templates including a current template, an intra prediction template, and a reference template of the current block comprises: Obtaining the intra prediction template by performing intra prediction using the current intra mode; Obtaining a fused template by fusing the intra prediction template and the reference template; and The coefficients of the linear filter are obtained by solving a least square equation using the fused template and the current template. 39. The method of claim 37, wherein deriving coefficients of a linear filter from a plurality of templates including a current template, an intra prediction template, and a reference template of the current block further comprises: receiving a flag indicating whether to apply the linear filter model; and In response to determining that the flag indicates applying the linear filter model, coefficients of the linear filter are obtained from the plurality of templates including the current template of the current block, the intra prediction template, and the reference template. 40. The method of claim 37, wherein deriving coefficients of a linear filter from a plurality of templates including a current template, an intra prediction template, and a reference template of the current block further comprises: A coefficient of the linear filter is obtained from the plurality of templates including a current template of the current block, the intra prediction template, and the reference template, wherein each of the plurality of templates includes one of a luma component or a chroma component. 41. A video encoding method, comprising: The encoder generates a weighted prediction by weighting the intra prediction block and the intra template matching prediction (TMP) block; Obtaining, by the encoder, coefficients of a linear filter from a plurality of templates including a current template of a current block, an intra-frame prediction template, and a reference template of a reference block; and The final prediction is obtained by the encoder by applying the coefficients of the linear filter to the weighted prediction. 42. The method of claim 41 , wherein deriving coefficients of a linear filter from a plurality of templates including a current template, an intra prediction template, and a reference template of the current block comprises: Obtaining the intra prediction template by performing intra prediction using the current intra mode; Obtaining a fused template by fusing the intra prediction template and the reference template; and The coefficients of the linear filter are obtained by solving a least square equation using the fused template and the current template. 43. The method of claim 41 , wherein deriving coefficients of a linear filter from a plurality of templates including a current template, an intra prediction template, and a reference template of the current block further comprises: signaling a flag indicating whether the linear filter model is applied; and In response to the flag indicating application of the linear filter model, coefficients of the linear filter are obtained from the plurality of templates including the current template of the current block, the intra prediction template, and the reference template. 44. The method of claim 41 , wherein deriving coefficients of a linear filter from a plurality of templates including a current template, an intra prediction template, and a reference template of the current block further comprises: A coefficient of the linear filter is obtained from the plurality of templates including a current template of the current block, the intra prediction template, and the reference template, wherein each of the plurality of templates includes one of a luma component or a chroma component. 45. A video decoding device comprising: one or more processors; and a memory coupled to the one or more processors, the memory being configured to store instructions executable by the one or more processors, Wherein, when executing the instructions, the one or more processors are configured to perform the method of any one of claims 1-4, 9-13, 16-18, 20, 33-34 and 37-40. 46. A video encoding apparatus, comprising: one or more processors; and a memory coupled to the one or more processors, the memory being configured to store instructions executable by the one or more processors, Wherein, when executing the instructions, the one or more processors are configured to perform the method of any one of claims 5-8, 21-25, 28-30, 35-36 and 41-44. 47. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform the method of any one of claims 1-4, 9-13, 16-18, 20, 33-34, and 37-40. 48. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform the method of any one of claims 5-8, 21-25, 28-30, 35-36, and 41-44. 49. A non-transitory computer-readable storage medium for storing a bitstream to be decoded by the method of any one of claims 1-4, 9-13, 16-18, 20, 33-34, and 37-40. 50. A non-transitory computer-readable storage medium for storing a bitstream generated by the method of any one of claims 5-8, 21-25, 28-30, 35-36, and 41-44.