KR20250148446A - Method and appratus for video encoding and decoding based on combined reference vector - Google Patents

Abstract

The present invention relates to the field of digital video encoding and decoding, and more particularly, to a method for encoding and decoding digital video, a method for recording such data, and components, devices, and systems for implementing such a method. The present invention provides a method and device for improved combined vector-based video encoding and decoding that can be used in the aforementioned video encoding and decoding field.

Description

METHOD AND APPRATUS FOR VIDEO ENCODING AND DECODING BASED ON COMBINED REFERENCE VECTOR

The present invention relates to the field of encoding and decoding of digital video, and relates to a method for encoding and decoding digital video, a method for recording such data, and components, devices, and systems for realizing such a method.

The present invention may be in the same technical field as at least one of the digital video compression technology standards known by the names of standards such as MPEG-2, MPEG-4 Video, H.263, H.264/AVC, H.265/HEVC, H.266/VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG, or may be in the technical field for improving the inherent efficiency of the standards, or may be in the technical field for improving or replacing the standards.

Digital video encoding and decoding are widely used in various digital video applications. For example, digital television broadcasting, video transmission over communication networks, video calls/video conversations/video chats, recording and providing video content using optical media including video compact discs (VCDs), digital versatile discs (DVDs), and Blu-Rays, all processes for producing, editing, collecting, and distributing video content, and devices such as video recording devices and camcorders for various purposes, including personal, commercial, industrial, and security purposes, all depend on video encoding and decoding technologies.

Accordingly, implementations that may be referred to as digital video encoders and decoders may form part of a wide range of devices related to the generation, recording, and provision of digital video, including digital televisions, digital broadcasting systems, wireless broadcasting systems, computers in the form of notebooks/desktops/tablets, e-book readers, digital cameras, digital recording devices, digital multimedia playback devices, video game devices/terminals/consoles, mobile phones (including smartphones) with multimedia playback capabilities, equipment for video conferencing, and other devices.

The above digital video encoders and decoders can be implemented by a digital video compression standard that is widely used and understood by those skilled in the art. The digital video compression standard may include at least one of compression standards known by a standard name such as MPEG-2, MPEG-4 Video, H.263, H.264/AVC, H.265/HEVC, H.266/VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG.

Video encoders and decoders can be implemented to more efficiently encode or decode digital video information while complying with the above standards, or by improving or modifying the above standards. Attempts to modify the above standards can also lead to the development of new standards. A well-known example is the so-called enhanced compression model (ECM), an attempt to improve and replace the existing H.266/VVC standard, currently being developed by the Joint Video Experts Team (JVET), a joint international standardization group of ISO, IEC, and ITU-T.

Digital video is known to require a significant amount of information to describe its content in its uncompressed state. Therefore, recording or transmitting this information in its original form can be inefficient. Therefore, digital video is compressed using various methods prior to recording or transmission. These compression methods include lossy and lossless encoding. Lossy encoding sacrifices some image quality to achieve high compression performance, while lossless encoding sacrifices some compression performance to prevent image quality degradation. Regardless of the encoding method, there is a need to implement a technology that achieves a high compression ratio while minimizing image quality degradation to meet the growing demand for high-quality digital video within the constraints of limited memory storage capacity and communication transmission bandwidth.

As described above, the encoding process for compression requires various operations, such as spatial segmentation of digital video, segmentation and/or processing in color channels, removal of spatial redundancy, removal of temporal redundancy, tracking of motion vectors within the video, encoding of differential images, quantization, coefficient scan, run-length coding, entropy coding, and loop filtering. These encoding operations generally consume computing resources and take a certain amount of time to complete. Similarly, the decoding operation for the encoding operation also requires certain computing resources and a certain amount of time. The main goal of video encoding and decoding technology is to ensure that the above resource consumption and time consumption do not interfere with the production, recording, distribution, and viewing of digital videos.

Accordingly, the present invention provides a new technology that can contribute to at least one of the following: improvement of encoding efficiency, improvement of decoding efficiency, improvement of video quality, reduction of computational amount, reduction of software size, reduction of hardware size, and improvement of other performances related to encoding and decoding in the technical tasks in the field of video encoding and decoding.

The present invention provides a method and apparatus for improved combined vector-based video encoding and decoding that can be used in the video encoding and decoding area described above.

According to an embodiment of the present invention for solving the above-described technical problem, a video decoding method may include a step of obtaining information about a current block from a bit string, a step of obtaining a first reference vector based on the information about the current block, a step of obtaining information about a first block region based on the first reference vector, a step of obtaining a second reference vector based on the information about the first block region, a step of obtaining a combined vector based on the first reference vector and the second reference vector, and a step of generating a prediction block for the current block using the combined vector.

The above method may be characterized in that the first reference vector is a block vector and the second reference vector is a motion vector.

The method may further include a step of adding a scaled vector to the merge list based on the combined vector.

The above combination vector may be characterized in that it is scaled taking temporal distance into account.

The method may be characterized in that at least one of the first reference vector, the second reference vector, and the combined vector is corrected through template matching.

The above method may further include a step of independently applying a combination vector to each of the divided blocks when the current block is composed of a plurality of divided blocks.

The method may further include a step of generating at least one of a block vector list or a motion vector list based on information about the current block, and a step of obtaining the first reference vector based on at least one of the block vector list and the motion vector list.

The method may further include a step of obtaining information indicating whether at least one of the block vector list or the motion vector list is used from the bit string.

The method may further include a step of obtaining the second reference vector by an operation based on the plurality of reference vectors when a plurality of reference vectors exist in the first block area.

The step of generating a prediction block for the current block using the above combination vector may include a step of performing intra-screen prediction using the above combination vector as a block vector for the current screen.

According to an embodiment of the present invention for solving the above-described technical problem, a video encoding method may include the steps of: obtaining a first reference vector for a current block; obtaining information on a first block region based on the first reference vector; obtaining a second reference vector based on the information on the first block region; obtaining a combined vector based on the first reference vector and the second reference vector; generating a prediction block for the current block using the combined vector; performing prediction encoding on the current block based on the prediction block; and outputting information on the first reference vector and information on obtaining the combined vector as part of a bit string.

The above method may be characterized in that the first reference vector is a block vector and the second reference vector is a motion vector.

The method may further include a step of adding a scaled vector to the merge list based on the combined vector.

The above combination vector may be characterized in that it is scaled taking temporal distance into account.

The method may be characterized in that at least one of the first reference vector, the second reference vector, and the combined vector is corrected through template matching.

The above method may further include a step of independently applying a combination vector to each of the divided blocks when the current block is composed of a plurality of divided blocks.

The method may further include a step of generating at least one of a block vector list or a motion vector list based on information about the current block, and a step of obtaining the first reference vector based on at least one of the block vector list and the motion vector list.

The method may further include a step of including information indicating whether at least one of the block vector list or the motion vector list is used in the bit string.

The step of generating a prediction block for the current block using the above combination vector may include a step of performing intra-screen prediction using the above combination vector as a block vector for the current screen.

According to an embodiment of the present invention for solving the above-described technical problem, a video decoding device may include a receiving unit for receiving a bit stream, a parser unit for obtaining information on a current block from the bit stream, a prediction unit for obtaining a first reference vector based on the information on the current block, obtaining information on a first block region based on the first reference vector, obtaining a second reference vector based on the information on the first block region, and obtaining a combined vector based on the first reference vector and the second reference vector, and a prediction decoding unit for generating a predicted block for the current block using the combined vector.

According to the present invention, at least one effect of improving encoding efficiency, improving decoding efficiency, improving video quality, reducing computational load, reducing software size, reducing hardware size, and improving other performances related to encoding and decoding can be derived in video encoding and decoding.

Figure 1 is a conceptual diagram of a video communication system according to one embodiment of the present invention;
FIG. 2 is a conceptual diagram of the arrangement of an encoder and decoder in a real-time video streaming environment according to one embodiment of the present invention.
Figure 3 is a functional unit conceptual diagram of a video decoder according to one embodiment of the present invention;
Figure 4 is a functional unit conceptual diagram of a video encoder according to one embodiment of the present invention;
Figure 5 is a conceptual diagram of a frame type according to one embodiment of the present invention;
FIG. 6 is a conceptual diagram showing the structure of a video encoder according to another embodiment of the present invention;
Figure 7 is a conceptual diagram illustrating a combined vector prediction technique according to one embodiment of the present invention;
Figure 8 is an example of a combination vector according to one embodiment of the present invention;
Figure 9 is an example of a combination vector according to one embodiment of the present invention;
FIG. 10 is an example diagram of a plurality of motion vectors according to one embodiment of the present invention;
FIG. 11 is an example diagram of a block vector obtained from combined vector inference according to one embodiment of the present invention;
FIG. 12 is an exemplary diagram for generating a prediction block of a block vector obtained from a combined vector inference according to one embodiment of the present invention;
FIG. 13 is an example diagram of a case in which both a block vector list and a motion vector list are used according to one embodiment of the present invention.
FIG. 14 is an example diagram for correction of a combined vector in the form of a block vector according to one embodiment of the present invention, and
FIG. 15 is an exemplary diagram for processing a split block according to one embodiment of the present invention.

The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

Although terms such as “first,” “second,” etc. may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, the first component could be referred to as the second component, and similarly, the second component could also be referred to as the first component. The term “and/or” includes any combination of multiple related listed items or any of multiple related listed items, and is non-exclusive unless otherwise indicated. The listing of items in this specification is merely an exemplary description to easily explain the spirit and possible implementation methods of the invention herein, and therefore is not intended to limit the scope of embodiments of the present invention.

As used herein, "A or B" can mean "only A," "only B," or "both A and B." In other words, as used herein, "A or B" can be interpreted as "A and/or B." For example, as used herein, "A, B or C" can mean "only A," "only B," "only C," or "any combination of A, B and C."

As used herein, a slash (/) or a comma can mean "and/or." For example, "A/B" can mean "A and/or B." Accordingly, "A/B" can mean "only A," "only B," or "both A and B." For example, "A, B, C" can mean "A, B, or C."

In this specification, "at least one of A and B" may mean "only A", "only B" or "both A and B". Additionally, in this specification, the expressions "at least one of A or B" or "at least one of A and/or B" may be interpreted identically to "at least one of A and B".

Additionally, in this specification, “at least one of A, B and C” can mean “only A,” “only B,” “only C,” or “any combination of A, B and C.” Additionally, “at least one of A, B or C” or “at least one of A, B and/or C” can mean “at least one of A, B and C.”

When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

The terminology used herein is merely used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

In describing the invention herein, embodiments may be described or illustrated in terms of unit blocks that perform the described function or functions. The blocks may be expressed herein as one or more devices, units, modules, parts, etc. The blocks may be implemented in hardware by one or more logic gates, integrated circuits, processors, controllers, memories, electronic components, or information processing hardware implementation methods, but not limited thereto. Alternatively, the blocks may be implemented in software by application software, operating system software, firmware, or information processing software implementation methods, but not limited thereto. A single block may be implemented by being separated into multiple blocks that perform the same function, or conversely, a single block may be implemented to perform the functions of multiple blocks simultaneously. The blocks may also be implemented by being physically separated or combined according to any criteria. The blocks may be implemented to operate in an environment where their physical locations are not specified and are separated from each other by a communication network, the Internet, a cloud service, or a communication method, but not limited thereto. All of the above implementation methods are within the scope of various embodiments that can be taken by a person skilled in the field of information and communication technology to implement the same technical idea, and therefore, any detailed implementation method should be interpreted as being included within the scope of the technical idea of the invention in this specification.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. To facilitate a comprehensive understanding of the present invention, the same reference numerals will be used for identical components in the drawings, and redundant descriptions of identical components will be omitted. Furthermore, the multiple embodiments are not mutually exclusive, and it is assumed that some embodiments may be combined with one or more other embodiments to form new embodiments.

digital video codec

Figure 1 is a conceptual diagram of a video communication system according to one embodiment of the present invention. The video communication system (100) may be configured to include at least two terminals (110, 120) connected to each other via a network (105).

In one embodiment of the present invention, the above-described FIG. 1 may refer to a block diagram for configuring a unidirectional video communication network. A first terminal (110) among the terminals may encode video data in order to transmit (111) the video data via a network (105). A second terminal (120) among the terminals may be configured to receive (121) the encoded video data via a network and decode and display the same.

In another embodiment of the present invention, the above-described FIG. 1 may refer to a block diagram for configuring a two-way video communication network. For the two-way video communication, each terminal (110, 120) may be configured to encode video data acquired by itself for video transmission (112, 122) to each other terminal via the network. Each terminal may also be configured to receive (113, 123) video data transmitted by another terminal via the network, decode the same, and display the decoded video data.

The terminals (110, 120) shown in Fig. 1 may be exemplified as devices such as server computers, personal computers, portable computers, and smartphones, depending on the embodiment, but are not limited thereto. The present invention is applicable to all environments for establishing a one-way or two-way video communication network, and it should be understood that the network (105) may be established by any means for transporting encoded video data between the terminals (110, 120).

In one embodiment of the present invention, the network (105) may refer to a wired or wireless communication network. Depending on the embodiment, the network may be configured to communicate information using any communication standard, which may include packet-based communication. The packet communication may be understood to include packets, for example, known as TCP or UDP.

However, in another embodiment of the present invention, the network (105) may be understood to include a process of information transmission using a recording medium. In this case, the configuration of the network is not limited to a communication medium, and should be understood to include a process of temporarily storing and physically transporting information on a hard disk, a solid state disk (SSD), a flash memory, a compact disc (CD), a digital versatile disc (DVD), a Blu-ray disc, and other mechanical, electronic, or optical recording media.

Any other means of information communication or transport, regardless of the method employed, can be considered within the scope of the present invention as long as it has a structure that supports the transmission and decoding of video data in an encoded state. Therefore, in addition to the examples listed above, any means of information communication or transport, whether known in the past or newly available, can fall within the scope of application of the present invention.

FIG. 2 is a conceptual diagram illustrating the arrangement of an encoder and decoder in a real-time video streaming environment according to one embodiment of the present invention. The streaming system (200) illustrated in FIG. 2 can be applied to video data communication networks, including, for example, digital broadcasting, video telephony, and video conferencing. However, it should be noted that technical structures identical or similar to the streaming system can be equally applied even when information is transmitted via a recording medium, as described above.

According to one embodiment of the present invention, the streaming system may include a video source (210) that generates a video stream. The video source may include a digital video acquisition means (212), which may be, for example, a digital camera or other device, for acquiring uncompressed raw video. The raw video stream (215) may have a large capacity and may therefore be compressed by a video encoder (217) coupled or connected to the video source.

The above encoder (217) may be configured as a means including hardware, software, or a combination of the two, configured to implement an image encoding method and/or an implementation method thereof according to one embodiment of the present invention.

Through the encoder (217), an encoded bitstream (219) having a reduced capacity compared to the original video stream can be output. The bitstream (219) can be provided in real time via a relay device, which may be referred to as a streaming server (220), and/or can be stored in a recording medium (225) of the streaming server (220) for subsequent use.

The streaming system (200) may include at least one streaming client (230, 240) that connects to the streaming server (220) to receive the encoded bit string (229) in real time or obtain it later. The streaming client may include a video decoder (232) that obtains the encoded bit string (229) (which may also be considered as a copy of the bit string (219) received by the streaming server), decodes the bit string (229), and outputs the resulting video data as video data in a form that can be displayed by a display (235) or other visual, auditory, or other sensory display means.

As described above, the functions for encoding and decoding video data are collectively called a coder-and-decoder, or video codec.

FIG. 3 is a conceptual diagram of a functional unit of a video decoder according to an embodiment of the present invention. As shown in FIG. 3, a receiving unit (310) can receive at least one encoded video data to be decoded by a decoder (305). In an embodiment of the present invention, the encoded video data may be independent for each reception, and the decoding procedure of each independent video data may be independent from the decoding procedure of other video data. The encoded video data may be received by the receiving unit (310) through a hardware or software connection (315) to a device storing the same, and as described above, the storing device may be a type of streaming server located at the other end of a communication network, or may mean a physical recording medium, but is not limited thereto.

The above-described receiving unit (310) can receive the encoded video data together with other data accompanying it, such as encoded audio data or other auxiliary data, and each of the data can be separated from the video data and provided to an appropriate processing function unit (312) other than the video decoder.

When the video data is provided through a communication network, a buffer memory (320) may be coupled between the receiving unit (310) and the decoder (305) to minimize delay and disconnection according to the network environment. The buffer memory (320) may refer to a computer-readable recording medium that temporarily stores the received video data and stably supplies it to a parser (330) corresponding to the input terminal of the decoder (305). However, if the bandwidth of the communication network is sufficient, if video data is read from a recording medium in a local location that is not physically separated, or if the possibility of communication delay is not predicted in other environments, the buffer memory may be unnecessary.

The video decoder (305) may include the parser (330) as its input terminal to interpret the encoded video data. The parser may perform a function of separating (parsing) a plurality of pieces of information stored in the form of a bit string in the encoded video data according to a predetermined rule, and, if necessary, performing an entropy decoding (335) of entropy-coded video data, thereby performing a function of reconstructing symbols (338), which are paragraphs of video encoding information. The symbols (338) may include all information for controlling the operation of the decoder (305), and/or may further include information for controlling a device that is attached to and operable with the decoder (305), such as a display device. Control information for controlling the above display device may include information in a format called supplementary enhancement information (SEI) or video usability information (VUI).

As described above, the parser (330) may be configured to perform entropy decoding (335) of the encoded video data. The entropy encoding method of the encoded video data may vary depending on the encoding standard, and decoding may be performed accordingly. Representative examples of the entropy encoding standard may include variable length coding, Huffman coding, and arithmetic coding, and each of the encoding methods may be a context-adaptive or context-sensitive method depending on the standard, and may also be based on principles widely known to those skilled in the art.

The parser (330) may be configured to extract at least one picture from the encoded video data. The definition of the picture may vary depending on the encoding standard, and depending on the standard, one or more of the examples listed below may correspond simultaneously and overlappingly. The picture may be grouped, defined, and/or divided into encoding/decoding units such as, for example, groups of pictures (GOPs), pictures/frames, tiles, slices, macroblocks, blocks, subblocks, transform units (TUs), and prediction units (PUs).

The parser (330) may be configured to extract encoding information, such as transform coefficients, quantization parameters (QPs), and/or motion vectors, from the encoded video data. The parser (330) may be configured to perform entropy decoding (335) and parsing operations on the video data received from the buffer memory, and to selectively decode symbols (338) representing the encoding information. In addition, the parser (330) may be configured to selectively supply a specific symbol (338) to a specific decoding function unit within the decoder (305), such as an inverse quantization and inverse transform unit (340), an intra prediction unit (350), an inter prediction unit (355), or a loop filter unit (360). Control of such information supply can be determined by the information sequence contained in the encoded video, and may vary depending on the encoding standard, and is not limited within the scope of the embodiments of the present invention, and is not described in detail in this conceptual diagram.

The decoder (305) may be comprised of a number of conceptual functional units that receive and process the encoded information provided by the parser (330). It should be readily apparent that these conceptual functional units may be combined or further subdivided, depending on implementation needs. For example, they may be further separated for ease of implementation, or integrated into one for operational efficiency. In any case, each functional unit may be configured to perform close interaction with each other. However, despite the possibility of such integration or separation, the following description will be given as a combination of conceptual functional units to illustrate the video data decoding procedure applied as an embodiment of the present invention.

The decoder may include an inverse quantization and inverse transformation unit (340). The inverse quantization and inverse transformation unit (340) may be configured to receive encoding information including a method to be used for numerical transformation (transform), a block size, quantization coefficients for recovering quantized information, and distinction information of a quantization matrix that simplifies and represents the quantized coefficients from the parser (330), and may be configured to output block values (341) that can be input to an aggregator (370) as a result of processing the encoding information.

In one embodiment of the present invention, the output values of the inverse quantization and inverse transformation unit (340) may include a predicted encoded block value within the screen. The predicted block value within the screen may mean a value that can be decoded using prediction information within the picture currently being decoded, for example, the current frame, but without using prediction information from a previously decoded picture, for example, a previous frame.

The prediction information within the current picture may be provided by the intra-screen prediction unit (350). According to an embodiment of the present invention, the intra-screen prediction unit (350) generates a block value of the same form as the block being decoded as the prediction information by using picture information of a spatially adjacent area derived from a picture currently being decoded and of which decoding has been partially completed. The picture information may be provided (381) from a buffer for the current picture, a so-called line buffer (380). The merging unit (370), according to an embodiment, may be configured to merge the prediction information (351) generated by the intra-screen prediction unit (350) with the block values (341) provided by the inverse quantization and inverse transformation unit (340).

In another embodiment, the output values of the inverse quantization and inverse transformation unit (340) may include block values that have undergone inter-screen prediction encoding, and in some cases, block values that have undergone motion compensation. In this case, the inter-screen prediction unit (355) may extract and use sample information (386) used for motion-based prediction from a reference picture buffer (385). The information (356) derived by performing motion compensation on the sample information based on the symbols (338) included in the block values as the output values may be configured to be merged with the block values (341) provided by the inverse quantization and inverse transformation unit (340) by the merger unit (370). In this case, the block values (341) may be referred to as so-called differential or residual values.

The position information within the memory used by the inter-screen prediction unit (355) to extract the sample information from the reference picture may be determined by a motion vector provided to the inter-screen prediction unit (355) and composed of a combination of symbols (338) for indicating, for example, X, Y, and other specific points of the reference picture. The inter-screen prediction unit (355) may also include a function for interpolating and using the sample values when a so-called 'subsampling'-capable motion vector is provided, and may further include a function for predicting and reinforcing the value of the motion vector.

The output values (371) of the above merging unit (370) may be provided to the loop filter unit (360) and processed by various loop filtering methods. The loop filter unit (360) may be configured to receive not only the block unit output (371) of the merging unit (370) but also the symbol (338) provided from the parser (330) to control its operation. The output of the loop filter unit (360) may be output to an external display means such as the display device through an output connection (390), but may be stored (361) in a line buffer (380) for use in prediction for interpreting a post-screen or inter-screen prediction encoding block value, and may also be stored in a reference picture buffer (385) through this.

Certain pictures, such as frames, after their decoding is completed, can be utilized as reference pictures for performing predictive decoding in a subsequent decoding process. A picture (or frame) can be gradually accumulated in a line buffer (380) and decoded. When the decoding of a frame is completed, the contents of the line buffer (380) are transferred (383) to the reference picture buffer (385), and a new line buffer (380) can be allocated for decoding the new frame.

The above video decoder (305) may be configured to perform a decoding operation according to a predetermined video compression technique that may be documented by various international standards or commercial standards. The standards may include, for example, international standard recommendations such as H.264, H.265, and H.266 defined by the International Telecommunication Union Standardization Sub-Division (ITU-T). Those skilled in the art will understand that each of the above recommendations is equivalent to an international standard jointly defined by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). The encoded video data may comply with a specific bitstream syntax defined by the relevant standards, as defined and required by the video compression standard documents and standard documents, and specifically by the profiles and levels specified within such documents. In addition, the complexity of the encoded video data may be limited to a certain level to comply with the profiles and levels. For example, a profile or level may be configured to limit a maximum picture size, a maximum decoding speed, and a maximum reference picture size. These limitations may, in some embodiments, also be further limited by metadata signals for a hypothetical reference decoder (HRD) and HRD buffer management included in the encoded video data.

According to one embodiment of the present invention, the receiver (310) may receive additional redundant data along with the encoded video. The additional data may be considered as part of the encoded video data. The additional data may include information that may be used by the decoder (305) to properly decode the data or to more accurately reconstruct an image that approximates the original image. The additional data may be provided in the form of, for example, layers for temporal, spatial, or signal-to-noise ratio (SNR) enhancement, redundant slices, redundant pictures, and forward error correction codes.

Figure 4 is a functional unit conceptual diagram of a video encoder according to one embodiment of the present invention. The encoder (405) may be configured to receive original video information (402) from a video source (401) and perform encoding.

The original video information (402) may have any suitable bit depth, for example, 8 bits, 10 bits, 12 bits, etc. In addition, the original video information (402) may have any suitable color space, for example, R/G/B, Y/U/V, Y/Cb/Cr, etc. In addition, the original video information (402) may have any suitable sampling structure corresponding to the color space, for example, Y/Cb/Cr 4:2:0, Y/Cb/Cr 4:4:4, etc. The original video information (402) having such a predetermined format may be provided to the encoder in the form of a digital video stream.

In a one-way video communication network, the original video information (402) can be obtained from a recording medium storing a previously prepared video source. In a two-way video communication network, the original video information (402) can be obtained from an image acquisition device, such as a camera, that generates at least one video transmission stream included in the two-way video communication.

The video data including the above original video information (402) may be configured as a plurality of pictures configured to simulate motion by playing them in chronological order. The picture may also be expressed as a concept such as a frame in addition to a picture. The picture may include one or more samples depending on the type of sampling structure, color space, etc. being used. Those skilled in the art will understand that the sample and the pixel and/or pixel in a digital image are closely related terms. The operation of the encoder will be described below with reference to such samples.

According to one embodiment of the present invention, the encoder (405) may be configured to encode and compress pictures (and/or grouped or segmented information) constituting the original video information (402) in real time (or according to other temporal requirements required according to the implementation method) into the form of encoded video information.

In the encoder (405), the control unit (450) may be a functional unit configured to control an appropriate encoding speed. The control unit (450) may be configured to control other functional units and be functionally coupled to the following functional units as described below. The parameters set by the control unit (450) may include parameters related to bitrate control, such as picture skip, quantizer, and variable values for applying image quality optimization techniques, and may also include values such as picture size, group of pictures (GOP) structure, and maximum search range of motion vectors. A person skilled in the art will be able to understand various other functions that the control unit (450) may have, and such other functions may be added or removed according to the design of a video encoder optimized for individual system design.

According to an embodiment of the present invention, the encoder (405) may be configured to operate in a structure such as a "coding loop" well known to those skilled in the art. To simplify the description by way of example, the coding loop may be configured with an internal encoder (so-called "source coder") (410) which is responsible for receiving a picture to be encoded and generating symbols based on at least one reference picture that has been encoded in the past, and a local decoder (420) configured to be connected to the internal encoder. The local decoder (420) may be configured to perform an operation to reproduce sample data to be generated by a decoder (490) located at an actual remote location that will receive encoded video information from the encoder (405) by receiving an output of the internal encoder (410).

Video data composed of sample data reconstructed by the internal decoder (420) may be configured to be input to the reference picture buffer of the encoder (405). As described above, the internal decoder (420) is implemented to reproduce the result output by the encoder (405) and to be decoded by a remote decoder, so the video data recorded in the reference picture buffer may also be identical in bit unit to the information of the reference picture buffer of the remote decoder. That is, the prediction function unit that may be included in the encoder (405) may read the same values as the sample values of the previous frame that the decoder will later refer to in the decoding process from the reference picture buffer of the encoder (405).

As described above, the principle of achieving matching of the reference picture buffers between the encoder (405) and the decoder (490) by means of the internal decoder (420) on the encoder (405) side is well known to those skilled in the art, and a method of responding to an environment in which such an environment is not guaranteed (e.g., information loss due to communication failure, etc.) can also follow what is known to those skilled in the art.

An embodiment of the operation method of the internal decoder (420) has been described in detail above with reference to FIG. 3. The decoder of FIG. 3 may be regarded as the aforementioned "remote" decoder (490). The internal decoder (420) may be implemented excluding lossless encoding and decoding sections such as the parser (330) or entropy decoding (335). This is because the internal encoder (405) is implemented to simply reproduce the operation of a decoder located at a remote location, and thus may directly decode symbols without requiring a process of compressing and then decompressing symbols. Accordingly, the functional units preceding the parser and entropy decoder as shown in FIG. 3 may not be provided or may be implemented at least partially.

As described above, according to a preferred embodiment of the present invention, any decoder function (excluding a parser and an entropy decoder) present in the decoder can naturally exist as a substantially identical function in the corresponding encoder (405).

The operation of the encoding function unit that may be included in the above encoder (405) can be considered as the reverse operation of the decoder function unit. Therefore, the embodiment can be explained by performing the operation of the decoder function unit in reverse. For example, a quantization and transform function unit corresponding to the inverse quantization and inverse transform unit may be provided, and an inter-screen prediction encoding unit corresponding to the inter-screen prediction unit may be provided. In addition, some additional explanations will be added.

The internal encoder (410) may be configured to perform encoding on input picture information, for example, an input frame, by a prediction encoding method executed by a prediction encoding unit (440) that operates by referencing at least one reference picture information, for example, at least one temporally previous encoded picture (or frame) from a reference picture buffer (430) from video data designated as a reference frame. In this case, the encoder (405) may be configured to encode a differential between blocks of samples constituting the input picture and blocks of samples constituting the reference picture.

The internal decoder (420) can decode video data that can be designated as the reference picture from symbols generated by the internal encoder (410). As described above, since the video data is identical to the decoding operation performed by the remote decoder, the video data used as the reference picture may be provided to the encoder (405) in a form that has undergone lossy compression and has been partially damaged, and this operation may be intended to ensure operational consistency with the decoder.

The prediction encoding unit (440) may be configured to perform a prediction search operation within the encoder (405). The prediction search operation may refer to an operation corresponding to the inter-screen prediction or intra-screen prediction described in the description of the decoder. For picture information that is input and scheduled to be newly encoded, the prediction unit may access the reference picture buffer (430) to retrieve information such as a motion vector, a block shape, and metadata that may include the same, which are information indicating a point of a reference picture that can function as prediction reference information suitable for the new picture information, and a sample block to be actually referenced. The above prediction encoding unit (440) may operate on the basis of the so-called "sample block by pixel block" criteria in order to obtain appropriate prediction reference information. According to one embodiment of the present invention, at least one prediction reference information designating at least one reference picture information stored in the reference picture buffer (430) may be designated for the input picture, as determined based on the search results obtained by the prediction encoding unit (440).

In one embodiment of the present invention, the control unit (450) may be configured to manage the overall encoding operation of the internal encoder (410), including setting parameters used to encode video data.

All outputs of the above-described functional units may be subjected to entropy encoding (460) in order to be finally output. The entropy encoding (460) may include various entropy coding techniques, such as variable length coding, Huffman coding, and arithmetic coding, for the symbols generated by the various functional units as described above, and each encoding method may be a context-adaptive or context-sensitive method according to the standard, or may be based on principles widely known to those skilled in the art. Such entropy encoding (460) can typically achieve lossless compression, and thus can be configured to convert at least one symbol generated by the functional units into encoded video data.

The above control unit (450) may, when controlling the operation of the encoder (405), apply to each picture (or frame) the type of encoding in which a specific picture is encoded during the encoding period. Depending on the type, the method by which the picture is encoded may be affected. Depending on the embodiment, the type may include what is categorized as the following "frame type."

Fig. 5 is a conceptual diagram of a frame type according to one embodiment of the present invention. The following description will be made with reference to Fig. 5.

An intra-picture ("I") picture (510) may refer to a picture that can be encoded and decoded using only its own information without referring to other picture information in video data through predictive encoding. The "I" picture may be designated by names such as a key frame, an independent/instantaneous decoder referh (IDR) frame, and a clean random-access (CRA) frame, depending on the video encoding standard, and the "I" pictures designated by the various names as described above may have various modifications and application methods as permitted by each standard and may be partially different from each other. In addition to those listed above, various application methods for implementing the "I" picture may be by various methods that are already known to those skilled in the art or may be newly provided.

An inter-prediction (“P”) picture (520) may refer to a picture that can be encoded and decoded through intra- or inter-prediction based on at least one prediction information and/or a motion vector that designates at least one reference picture to predict sample values of a block constituting the picture. The “P” picture may be configured to refer to only one reference frame, or may be configured to refer to one or more reference frames, according to a video encoding standard. When referring to more than one reference frame, sample information and/or associated metadata derived from multiple reference pictures may be used to reconstruct a single block. However, in common cases, a picture designated as a “P” picture may be understood as a picture that performs reference only to a temporally preceding picture.

A bidirectional prediction ("B") picture (530) may refer to a picture that can be encoded and decoded through intra-screen or inter-prediction based on at least one piece of prediction information and/or a motion vector that designates at least two reference pictures in order to predict sample values of blocks constituting the picture. In a common case, a picture designated as the "B" picture is distinguished from a picture designated as the "P" picture, and may be understood as a picture that performs a reference without being limited to a temporally preceding picture.

Video data may be spatially divided into a plurality of sample blocks during the encoding and decoding process, and encoding may be performed in units of the blocks. The block units may include, but are not limited to, sizes such as 4x4, 8x8, 4x8, or 16x16 in units of horizontal/vertical pixels, as is widely known. The block may be encoded using a predictive encoding method with reference to any other (already encoded) blocks, as permitted and/or restricted by the type specified for each picture in which the block is included. For example, the blocks of the "I" picture (510) may be encoded without using a predictive encoding method, or with reference to blocks that have already been encoded within the same partial picture. That is, only the so-called intra-picture prediction method may be used. In contrast, the "P" picture (520) may further reference a reference picture encoded in at least one previous time unit, and thus, inter-prediction as well as intra-picture prediction may be used for encoding. In the case of a "B" picture (530), reference can be made not only to a previously encoded picture in the encoding order but also to a later reference picture in terms of time unit. However, it is widely known that there may be blocks within a "P" picture or a "B" picture that are encoded without relying on predictive encoding.

The above video encoder (405) may be configured to perform encoding operations according to a predetermined video compression technique that may be documented by various international standards or commercial standards. Examples of the above standards may include all of those described in the above decoder.

According to one embodiment of the present invention, the transmitter (470) may buffer the encoded video data generated by the entropy encoding in order to provide/transmit the video data (ultimately to a remote decoder (490)) to a device storing the encoded video data via a hardware or software connection (495). According to an embodiment, when providing/transmitting the encoded video data from the video encoder (405), the transmitter (470) may receive and merge other data accompanying the encoded video data, for example, encoded audio data or other auxiliary data, from a separate source (480).

According to one embodiment of the present invention, the transmitter (470) may be configured to transmit additional data along with the encoded video. The additional data may be considered part of the encoded video data. The additional data may include information that can be used by a decoder to properly decode the data or to more accurately reconstruct an image that approximates the original image. Examples of the additional data may include all of the examples previously presented with respect to the receiver (310) of the decoder.

The present invention can be implemented by a digital video compression standard that is widely used and understood by those skilled in the art as described above. The digital video compression standard may include at least one of compression standards known by the standard name such as MPEG-2, MPEG-4 Video, H.263, H.264/AVC, H.265/HEVC, H.266/VVC, VC-1, AV1, QuickTime, VP-9, VP-10, and Motion JPEG.

Fig. 6 is a conceptual diagram illustrating the structure of a video encoder according to another embodiment of the present invention. What is depicted in Fig. 6 may be a rough structure of a video encoder widely known as a standard code such as ITU-T H.266 and ISO/IEC 23090-3, and also known as MPEG-I Part 3 or versatile video coding (VVC).

According to FIG. 6, a video encoder (605) may be configured to receive uncompressed and unencoded original video data (601) as input and output an encoded bit stream (602). The video data (601) may be directly supplied to a luma mapping unit (610a) when intra-screen prediction encoding is performed, or may be supplied to a luma mapping unit (610b) via an inter-prediction unit (620) including motion vector extraction. In the case of intra-screen prediction encoding, the mapped luma signal may be supplied to an output merger (606) by selecting (608) at least one of an intra-screen prediction encoding signal via the intra-screen prediction unit (625) or an inter-prediction encoding signal output from the luma mapping unit (610b) via the inter-prediction unit (620). The result of the above output merger can be applied to a chroma scaling unit (615). (The operation of the luminance signal mapping unit (610) and the operation of the chroma scaling unit (615) are collectively referred to as a luma mapping/chroma scalaing (LMCS) process.) The reduced chroma signal can be provided to a transform unit (630), and the transform unit (630) can perform an adaptive color transform, particularly on the chroma signal. The coefficients derived as a result of the transform are applied to a quantization unit (640) and quantized. As a result, lossy compression is achieved, and the result of the lossy compression can be output as a bit string (602) through a multi-hypothesis CABAC (650), which is a lossless compression method.

Meanwhile, the result of the lossy compression may actually enter the decoding process by going through the processes of inverse quantization (645), inverse transform (635), and luminance signal expansion (617) to generate a coding loop. The result of the luminance signal expansion may be supplied to the internal merger (607) together with the result of selecting (608) at least one of the previously generated intra-screen prediction encoding signal or inter-prediction encoding signal. The result of the internal merger may go through inverse luma mapping (618), and then may go through processing such as a deblocking filter (660), sample adaptive offset (SAO) (670), and an adaptive loop filter (ALF) to reproduce the image quality improvement process in the decoder. The result of reproducing the operation in the decoder as described above is applied to the reference picture buffer (690) and can be reused for prediction encoding by the inter prediction unit (620).

The present invention can also be utilized by or in combination with an enhanced compression model (ECM), which is an implementation of a next-generation video codec currently being developed by the Joint Video Experts Team (JVET), an international standardization expert organization. The enhanced compression model can include an improved intra-picture prediction coding method, an improved inter-prediction coding method, an improved transform and transform coefficient coding method, an improved adaptive loop filtering method, a bilateral filtering method, a new sample adaptive offset (SAO) method for improving picture quality, an extended entropy coding method, and an improved gradual decoding refresh (GDR) technique.

Composition of the present invention

Fig. 7 is a conceptual diagram illustrating a combined vector prediction technique according to one embodiment of the present invention. As illustrated in Fig. 7, a combined vector (760) according to one embodiment of the present invention may be configured to derive a combined vector (760) for final prediction by starting from a motion vector (MV) (751) of a current block (711) and/or its surrounding blocks within a current frame (710), deriving a block area (721) indicated by the vector (751) in a previous frame (720), deriving a block area (722) indicated by the vector (752) using a block vector (BV) (752) obtained from the area (721), and deriving a motion vector (753) obtained from the area (722). The above restoration area (715) may refer to an area that belongs to the same slice, frame, etc. as the current block (711) currently being encoded/decoded and has already been restored after encoding/decoding has been completed. The prediction block (730) may refer to a reference block that can be obtained by the combined vector (760) and may refer to a block used as a prediction block or used in its generation.

The continuous derivation of the combined vector (760) illustrated in FIG. 7 corresponds to an example through an embodiment of the present invention, and as permitted by the technical idea of the present invention, the combined vector may mean a result of any type of reference vector including a motion vector and a block vector being continuously acquired multiple times in any order. According to an embodiment of the present invention, the continuation of the combined vector may start from a block vector (BV) and continue to a motion vector (MV). According to another embodiment of the present invention, the continuation of the combined vector may start from a motion vector (MV) and continue to a block vector (BV). According to another embodiment of the present invention, the derivation of the combined vector may be continued by a plurality of block vectors (BV). According to yet another embodiment of the present invention, the derivation of the combined vector may be continued by a plurality of motion vectors (MV). According to an embodiment of the present invention, the derivation of the combined vector may be performed based on a block vector or a motion vector of a neighboring block in the reconstructed area (715) of the current block (711).

The following problems may exist in the conventional combined vector prediction technology. According to one conventional implementation method, only an inference method starting with a motion vector is used when inferring a combined vector. In addition, an efficient method for storing motion vectors or block vectors may be required to construct a combined vector. In addition, when multiple combined vectors exist, a method for supporting efficient selection and use of at least one of them may be required. In addition, restrictions on the use of combined vectors may be required to reduce complexity. In addition, a method for encoding/decoding the reference picture index of the combined vector during the motion search process may be required. In addition, a method for reusing the block vector obtained in the combined vector inference step in within-screen prediction may be required.

FIG. 8 is an exemplary diagram of a combined vector according to an embodiment of the present invention. As illustrated in FIG. 8, the combined vector of the present invention may be configured to start from a block vector (851) of a current block (811) and/or its surrounding blocks within a current frame (810), derive a block region (812) pointed to by the vector (851) within a reconstructed region (815) of the current frame (810), derive a block region (821) pointed to by the vector (852) in a previous frame (820) using a motion vector (852) obtained from the region (812), and derive a motion vector (853) obtained from the region (821), thereby deriving a combined vector (860) corresponding to a final prediction block (830). This represents an example in which, in contrast to the configuration illustrated in FIG. 7, the continuation of the combined vector starts from a block vector and continues to a motion vector.

According to one embodiment of the present invention, when applying the combined vector to a prediction mode based on list-based reference vector determination including a merge mode, when inferring the combined vector and adding it to a reference vector suffix list (which may include, for example, a block vector list, a motion vector list, a merge list, and/or an affine list), the combined vector may be inferred starting from a block vector in a block vector list constructed from surrounding blocks. Alternatively, reference vector lists having at least one different type may be configured to be used respectively. For example, a motion vector list and a block vector list in which motion vectors are stored may be configured to be used respectively. Alternatively, a reference vector candidate list construction method including but not limited to the examples described above may be adaptively determined and used. According to one embodiment of the present invention, the combined vector can be used cross-wise between various reference vectors and reference vector candidate lists, for example, a combined vector inferred from a motion vector can be used as a block vector or added to the candidate list, and similarly, a combined vector inferred from a block vector can be used as a motion vector or added to the candidate list, and in other cases, the combined vector can be configured to be cross-inferred and used cross-wise between various types of reference vectors.

According to one embodiment of the present invention, a method of using a combined vector can be applied by extending the Advanced Motion Vector Prediction (AMVP) method. The AMVP can be understood as a technique that uses the motion vectors of surrounding blocks to efficiently encode a motion vector, and the present invention can be configured to introduce the concept of a combined vector to the AMVP structure. In addition, according to one embodiment of the present invention, similar to the method of applying a scaling factor to a vector in Temporal Motion Vector Prediction (TMVP), scaling according to the temporal distance between reference frames can also be applied to the combined vector. In this case, the reference target of TMVP itself can be the combined vector, and therefore, it should be understood that the combined vector according to the present invention can be applied to both TMVP and the like.

According to one embodiment of the present invention, a method for inferring a combined vector based on a vector of a temporal block (TMVP) may be provided. Specifically, a co-located block, which is spatially identical to a current block but temporally adjacent, may be targeted, and a motion vector or block vector stored in the co-located block may be used as a first reference vector to initiate combined vector inference according to the present invention. The co-located block may refer to a block in the same or similar position as the current block within a reference frame temporally associated with the current frame. When inferring combined vector based on TMVP, the vector may be scaled by considering the temporal distance between the reference frames. For example, a scaling factor may be calculated based on the difference in picture order count (POC) between the current frame and the reference frame, and this may be applied to scaling the vector. In addition, TMVP itself may be utilized as a part of another combined vector, in which case the combined vector may be inferred according to various orders and combinations, such as TMVP-BV, TMVP-MV, or TMVP-BV-MV.

FIG. 9 is an exemplary diagram of a combined vector according to one embodiment of the present invention. According to one embodiment of the present invention, when a combined vector is inferred and added to a merge list or a motion vector list, both or at least one of a combined vector (combined vector 1) (961) inferred based on a block vector list and a combined vector (combined vector 2) (962) inferred based on a motion vector list may be used. Each combined vector (961, 962) may mean a vector that ultimately reaches a reference block (931, 932) through at least one vector continuation derivation starting from the block area (912, 921) that each of them points to from the current block (911). In one embodiment, if both of the above are used, it may be necessary to transmit information such as a flag or index that distinguishes them or specifies each of them. Accordingly, a corresponding encoder may be configured to transmit information on whether a block vector list or a motion vector list is used. The bitstream syntax for transmitting the above information can be explicitly included in the bitstream, and a corresponding decoder can be configured to read the bitstream syntax and use the correct vector list. For example, a 1-bit flag can be used to indicate whether a block vector list or a motion vector list is used. Alternatively, a list index can be used to specify one of several possible lists. This information can be transmitted in various bitstream syntax layers, such as a slice header, a sequence parameter set (SPS), a picture parameter set (PPS), or an adaptive parameter set (APS).

According to one embodiment of the present invention, the motion vector list and/or the block vector list may be composed of motion vectors and/or block vectors of blocks spatially adjacent to the current block and/or temporally adjacent blocks. The spatially adjacent blocks may refer to blocks spatially adjacent to the current block, such as to the left, top, top right, top left, bottom left, etc. of the current block, and may be selected from blocks that have already been encoded/decoded. The temporally adjacent blocks may refer to blocks of another time (frame) located at the same spatial location as or around the current block, and may be selected from at least one of the reference frames of the current frame. The reference vectors obtained from the spatially adjacent blocks and temporal blocks may be utilized for the combined vector inference.

According to one embodiment of the present invention, when storing a motion vector or a block vector or a combined vector for the combined vector, the current block may be divided into an arbitrary block size such as 4×4, 8×8, 16×16, 32×32, 64×64 and stored, or the current slice or frame or an arbitrary unit may be divided and encoded, and then divided again into an arbitrary block size and stored. For example, while encoding the current frame, the block vector may be stored for every 4×4 block, and after encoding, it may be stored for every 16×16 block. In addition, when storing multiple motion vectors or block vectors as one block, a representative vector among them may be calculated and then stored. The resolution and/or storage unit of the stored vector may be set differently according to a frame for intra-screen prediction or inter-screen prediction and stored, or the current slice or frame or an arbitrary unit may be compressed and then stored again at an arbitrary vector resolution.

According to one embodiment of the present invention, when using the stored motion vector, block vector, or combined vector in subsequent encoding and decoding processes, the resolution of the stored vector may be taken into account and then scaled for use. For example, if the resolution of the stored vector is relatively high, it may be lowered using a shift operator. For another example, if the resolution of the stored vector is relatively low, it may be increased using a shift operator.

FIG. 10 is an exemplary diagram of a plurality of motion vectors according to one embodiment of the present invention. According to one embodiment of the present invention, when a plurality of motion vectors and/or block vectors exist within a block vector or a block area indicated by a motion vector, a combined vector can be inferred by comprehensively considering the plurality of vectors. Referring to FIG. 10, FIG. 10 illustrates a case where a block area (1012) is pointed out by a block vector (1051) from a current block (1011) within a current frame (1010), and a plurality of motion vectors (1052) exist within the area (1012). According to one embodiment, a combined vector can be inferred using a vector obtained through a weighted average of vectors at the center and four corners of the block area (1012). According to another embodiment, a combined vector can be inferred using a result vector obtained through an arbitrary calculation formula for an arbitrary number of vectors. For example, a median filter can be applied, or a weighting according to the magnitude of the motion can be applied. In another embodiment, the combined vector may be inferred using only the vectors stored in the region closest to the pixel pointed to by the vector. In another embodiment, all of the inferred combined vectors may be used, or a vector obtained by weighting the combined vectors may be used. In one embodiment, when considering the plurality of vectors in an integrated manner, at least one motion vector and at least one block vector may be calculated together.

According to one embodiment of the present invention, whether to use a combined vector can be determined by considering the horizontal or vertical size (which may be in pixel units) of a block, the area (which may be in pixel units), the number of pixel samples within a block, or information about surrounding blocks. For example, a combined vector can be used only when the horizontal size is greater than A and the vertical size is greater than B. Or, for example, a combined vector can be used only when the horizontal size is less than A and the vertical size is less than B. Or, for example, a combined vector can be used only when the number of pixel samples within a block is greater than or less than C. Or, for example, a combined vector can be used only when the block area is greater than or less than D. According to one embodiment of the present invention, the restriction on the use of a combined vector based on the size, area, number of samples, or information about surrounding blocks of the block can be applied selectively and/or based on various criteria depending on the type of prediction mode. For example, in the case of an affine prediction mode, a combined vector can be configured to be used only when the horizontal or vertical size of the block is 8 or more. Of course, the scope of implementation of the present invention is not limited by the examples described above.

According to one embodiment of the present invention, the resolution of the combined vector may be limited, or bidirectional prediction of the combined vector may be limited, considering the horizontal or vertical size of the block, the area, the number of pixel samples within the block, or information about the surrounding blocks. For example, when the width is smaller than A and the height is smaller than B, only the combined vector in integer pixel units may be configured to be used. Or, for example, when the width is smaller than A and the height is smaller than B, only the combined vector in unidirectional units may be configured to be used. Or, for example, when the width is larger than A and the height is larger than B, the combined vector in bidirectional and unidirectional units may be configured to be used. Or, for example, when the number of pixel samples within the block is greater than or less than C, only the combined vector in integer pixel units may be configured to be used. Or, for example, when the number of pixel samples within the block is greater than or less than C, the combined vector in integer and subpixel units may be configured to be used. Or, for example, when the area of the block is greater than or less than D, only the combined vector in integer pixel units may be configured to be used.

According to one embodiment of the present invention, when performing motion search using a combined vector, if the reference image index of the combined vector is different from the reference image to be motion searched, the combined vector may not be used for the corresponding block. Alternatively, the combined vector may be inferred from the motion vector of the matching reference image within the block pointed to by the motion vector or block vector. Alternatively, the combined vector may be scaled to the reference image to be motion searched. Alternatively, the combined vector may be scaled by considering the distance between the reference image of the combined vector and the reference image to be motion searched.

According to one embodiment of the present invention, the process of scaling the combined vector into a reference image to be motion searched may be implemented by adding the scaled vector to the merge list. This may mean a method of adjusting the arrival point of the vector by considering the temporal distance between reference frames, similar to vector scaling in AMVP or TMVP. FIG. 11 is an exemplary diagram of a block vector obtained from combined vector inference according to one embodiment of the present invention. As illustrated in FIG. 11, according to one embodiment of the present invention, when adding a block vector obtained in the combined vector inference step to a block vector list, the inference may be made first starting with a motion vector (1151) in a motion vector list constructed from the current block (1111) in the current frame (1110) and/or neighboring blocks of the current block (1111) belonging to the restoration area (1115). The motion vector (1151) is connected to a block area (1121) to which the motion vector points, and a block vector (1152) may be inferred from the block area (1121). The above block vector (1152), or a vector continuously inferred based thereon, can be used for intra-screen or inter-screen prediction based on the current block (1111). That is, according to one embodiment, the block vector (1152) of the block (1152) pointed to by the motion vector (1151) in the motion vector list can be used. According to another embodiment, if there is no block vector in the block pointed to by the motion vector, the block vector can be identified at another location within the block or adjacent thereto. It should be understood that the above-described implementation method can be used in a manner of crossing, overlapping, and combining with various other implementation methods disclosed herein.

FIG. 12 is an exemplary diagram for generating a prediction block of a block vector obtained from a combined vector inference according to an embodiment of the present invention. As illustrated in FIG. 12, according to an embodiment of the present invention, when a block vector obtained in the combined vector inference step is added to a block vector list, the added vector can be utilized for intra-screen prediction. For example, when a block vector (1252) exists in a block area (1221) obtained by a motion vector (1251) based on a current block (1211) of a current frame (1210), such block vector (1252) can be cited and used as a block vector (1253) or a candidate thereof for intra-screen prediction in the current frame (1210). The intra-screen prediction includes a template matching (IntraTMP) mode, an intra-block copy (IBC) mode, and can be used in other intra-screen prediction methods based on block vectors to generate a prediction block (1230). According to one embodiment of the present invention, the derived block vector may be corrected through template matching based on the current block (1211). Alternatively, a vector inferred again based on the block vector of the block pointed to by the block vector may be recursively searched, i.e., the combined vector may be obtained based on the results of continuous N-th search.

FIG. 13 is an exemplary diagram illustrating a case in which both a block vector list and a motion vector list are used according to one embodiment of the present invention. Referring to FIG. 13, a method of using both a combined vector (combined vector 1) (1361) inferred based on a block vector list and a combined vector (combined vector 2) (1362) inferred based on a motion vector list is illustrated. For example, when performing prediction on a current block (1311) in a current frame (1310), the combined vector can be inferred in various ways. One way may be to start from a block area (1312) within a reconstructed area (1315) pointed to by the current block (1311) and/or its surrounding block vectors (1351) and infer combined vector 1 (1361) and predicted block 1 (1331). Another way to do this could be to access a block region (1321) of a previous frame (1320) pointed to by the current block (1311) and/or its surrounding motion vector (1332), and then infer a combined vector 2 (1362) pointing to the current frame (1310) and predicted block 2 (1332) based on the block vector (1352) indicated by the block region.

According to one embodiment of the present invention, when adding a block vector obtained in a combined vector inference step to a block vector list, if both a block vector list (combined vector 1) and a motion vector list (combined vector 2) are used, the corresponding encoder can transmit information such as a flag or index that distinguishes them or specifies each of them through a bit string syntax or the like. Alternatively, information on whether a block vector list or a motion vector list is used can be transmitted. The corresponding decoder can be configured to perform accurate predictive decoding based on the above information.

FIG. 14 is an exemplary diagram illustrating the correction of a combined vector in the form of a block vector according to one embodiment of the present invention. As illustrated in FIG. 14, according to one embodiment of the present invention, a motion vector, a block vector, or a combined vector obtained in the combined vector inference step can be corrected through template matching. The corrected combined vector can be ultimately used for intra-screen or inter-screen prediction, and/or stored for encoding/decoding of the next block.

Referring to FIG. 14, when a reference vector (1451) corresponding to a motion vector, a block vector, or a combination vector is derived by any method including those described in the present specification in the current block (1411), the reference vector (1451) can be corrected (1452) based on a template (1412) of the current block (1411).

According to one embodiment of the present invention, the template matching may refer to a method of finding the location of the most similar block area (1413) by using some of the already restored samples (1415) around the current block (1411) as templates (1412) and comparing them with other templates (e.g., templates (1414)) located around the point indicated by the reference vector (1451). In this process, the initial reference vector (1451) is used to define a starting point of the search, and may be configured to find the location of the block area (1413) corresponding to the template (1414) that best matches the template (1412) of the current block (1411) within a specified search range, and correct (1452) the vector. At this time, the surrounding search range of the template matching may be set in units of integer pixels or sub-pixels. For example, a search range of 1 pixel in each of the upper, lower, left, and right directions centered on the reference vector (1451), or in units of 1 or less sub-pixels, may be used.

According to one embodiment of the present invention, correction may be applied at each step of combined vector inference. Specifically, after obtaining a first reference vector, it may be corrected through template matching, and a first block area may be obtained using the corrected first reference vector. Subsequently, correction through template matching may also be applied to a second reference vector obtained from the first block area. Template matching in each of the above correction steps may be performed by finding an optimal matching position based on templates surrounding the current block, as described above with reference to FIG. 14, and the correction range may be set in integer pixel or sub-pixel units. Furthermore, according to one embodiment of the present invention, correction methods in each of the above correction steps, such as the template matching method and/or correction range, may be designated differently, and the different correction methods may be configured to be shared by the encoder and decoder through a standard, explicitly signaled through an encoded bit stream, or implicitly derived depending on the decoding aspect.

Fig. 15 is an exemplary diagram for processing a partition block according to one embodiment of the present invention. According to one embodiment of the present invention, when a motion vector or block vector, or a combination vector obtained in a combination vector inference step is used for a non-rectangular partition block and/or prediction based thereon, comparison and calculation may be performed only for positions within the partition block. Alternatively, a motion vector or block vector stored at a position within the partition block may be checked. Alternatively, the motion vector or block vector at the boundary position of the partition block may be stored as an arbitrary vector or a weighted average vector. The non-rectangular partition block may include, for example, a partition block to be diagonally partitioned by a geometric partitioning mode (GPM), but is not limited to this example.

Referring to FIG. 15, there is illustrated an example in which a current block (1511) is divided into a left divided block (1511-1) and a right divided block (1511-2) by a boundary line (1517) according to a geometrical division mode. According to one embodiment of the present invention, when a reference vector such as a motion vector, a block vector, or a combined vector obtained in a combined vector inference step is used for prediction encoding/decoding based on the divided blocks, vector processing can be performed independently for each of the left divided block (1511-1) and the right divided block (1511-2). For example, for the left split block (1511-1), the left split block corresponding reference vector (1551-1) may be applied and configured to designate the first reference area (1512-1), and for the right split block (1511-2), the right split block corresponding reference vector (1551-2) may be separately applied and configured to designate the second reference area (1512-2). According to one embodiment of the present invention, comparison and calculation may be performed only for positions within each split block, and for example, only the left area (1541) of the first reference area (1512-1) and the right area (1542) of the second reference area (1512-2) may be configured to be used for prediction encoding/decoding of the current block (1511), respectively.

According to one embodiment of the present invention, the combined vector can be utilized in various prediction modes. For example, in Merge mode, the combined vector can be added to the merge list and selected as the motion information of the current block. In Advanced Motion Vector Prediction (AMVP) mode, the combined vector can be added to the AMVP list and used as the motion vector predictor of the current block. In addition, the combined vector can be effectively utilized when the reference image of the combined vector and the reference image for motion search are the same.

According to one embodiment of the present invention, in the process of constructing an AMVP list in the AMVP mode, the index of a reference image to be motion searched and the reference image index of a combined vector may be compared to determine whether and how to add a combined vector. Specifically, the AMVP list may be constructed by sequentially examining motion vectors of surrounding blocks such as a left block and an upper block, and in this process, it may be confirmed whether the reference image index of each block matches the index of the reference image to be current motion searched. Similarly, for the combined vector, the reference image index may be examined, and if the reference image index of the combined vector is identical to the index of the reference image to be current motion searched, the combined vector may be configured to be added to the AMVP list. If there is no suitable motion vector, the zero (0) vector may be configured to be stored in the AMVP list.

According to one embodiment, the additional process of the combination vector related to the above-described AMVP mode can be expressed by a pseudo-code as shown in Table 1 below.

if(ref_idx_me==ref_idx_of_blk_a) amvp_list[i] = mv_blk_a;
else if(ref_idx_me==ref_idx_of_blk_b) amvp_list[i] = mv_blk_b;
…
else if(ref_idx_me==ref_idx_of_cmvp) amvp_list[i] = mv_cmvp;
…
else amvp_list[i] = 0;

In the pseudocode of Table 1, ref_idx_me represents the reference image index for motion search of the current block, ref_idx_of_blk_a and ref_idx_of_blk_b represent the reference image indices of the left block and the upper block, respectively, and ref_idx_of_cmvp represents the reference image index of the combined vector. In addition, mv_blk_a, mv_blk_b, and mv_cmvp represent the motion vector of the left block, the motion vector of the upper block, and the combined vector, respectively. In addition, any other conditional statements and/or processing steps, including the content of adding any reference vector derived from other surrounding blocks to the AMVP list, may be incorporated into the omitted (…) portion of the pseudocode of Table 1.

According to one embodiment of the present invention, if the reference image index of the combined vector is different from the index of the reference image to be currently motion searched, the combined vector can be expanded by scaling it to match the reference image to be motion searched and then adding it to the AMVP list. At this time, the scaling can be performed by considering the temporal distance between the reference images.

According to various embodiments of the present invention, in affine mode, a combined vector can be used to represent motion at multiple control point locations within a block, thereby efficiently expressing complex motions such as rotation, enlargement, and reduction of the block. In one embodiment, in the affine mode, the combined vector can be restricted to be applied only when both the horizontal and vertical dimensions of the current block are 8 or greater. In the geometric segmentation mode, the combined vector can be independently applied to each segmented area, as described above with reference to FIG. 15. In the intra block copy (IBC) mode, the combined vector can be utilized as a block vector within the current screen to perform intra-screen prediction. By utilizing the combined vector in various prediction modes in this way, the motion expression accuracy of complex images can be increased and the encoding efficiency can be improved. In addition, the combined vector according to the present invention can be applied to various prediction modes configured to use at least one motion vector, block vector, and other reference vector, regardless of intra-screen prediction or inter-screen prediction.

In addition, according to one embodiment of the present invention, the terms merge list, block vector list, motion vector list, etc. mentioned in this specification should be understood as comprehensively compatible terms, rather than expressions limited to a specific prediction mode. For example, the above lists may mean a list of candidate vectors and/or a merge list according to any method including reference vector candidates for predicting the current block in the encoding/decoding process, and are not limited to a specific prediction mode. For example, the merge list in the merge mode, the AMVP list in the advanced motion vector prediction (AMVP) mode, the affine merge list in the affine mode, the block vector list in the intra block copy mode, etc., can be interchangeably applied with candidate lists used in various prediction modes. Therefore, the combined vector proposed in the present invention should be understood as a technical configuration that is commonly and/or interchangeably applicable to these various types of candidate lists and prediction modes.

Encoder and decoder

It is self-evident that the method according to the present invention can be applied equally to both encoders and decoders. Additionally, as illustrated in FIG. 4 through the internal decoder (420) and the coding loop including it, this decoding process can be implemented identically within the encoder to predict the state of the decoder.

The encoding method according to the present invention described above can be implemented through an encoder as a device. The encoder as a device can be implemented in a form that maintains the conventional encoder structure exemplified through FIGS. 1 to 6 above or applies a predetermined change thereto. However, the form of implementation is not necessarily limited to that exemplified, and any form of encoder structure that can function as a video encoder should be considered an encoder established by the present invention as long as it implements the technical idea of the present invention.

In addition, the method for decoding the encoding result according to the present invention described above can be implemented through a decoder as a device. The decoder as a device can be implemented in a form that maintains the conventional decoder structure exemplified through FIGS. 1 to 6 above or applies a predetermined change thereto. However, the form of implementation is not necessarily limited to that exemplified, and any form of decoder structure that can function as a video decoder should be considered a decoder established according to the present invention as long as it implements the technical idea of the present invention.

A person skilled in the art will readily understand that a bit string encoded by the above-described method and device can be decoded by applying a method symmetrical and/or reverse to the encoding method. In one embodiment, when reading information for decoding from the encoded bit string, at least one variable length coded phrase included in the encoded bit string can be interpreted, and furthermore, in one embodiment, the variable length coding can be performed by an entropy coding method. The technical details and application method of implementing such a decoding procedure can be readily understood from the above-described encoding procedure.

The processor that may be included in the encoder and/or decoder described herein may mean one or more general purpose computers or special purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding.

Even if the processor is expressed singularly for ease of understanding, those skilled in the art will appreciate that the processor may include multiple processing elements and/or multiple types of processing elements. For example, a device according to one embodiment of the present invention may include multiple processors or one processor and one controller as the processor. Furthermore, the processor may be implemented using various processing configurations, such as a parallel processor or a multi-core processor.

The processor may be configured to execute an operating system (OS) and one or more software programs running on the operating system. Furthermore, the processor may access, store, manipulate, process, and generate data in response to the execution of the software.

The software may include a computer program, code, instructions, or a combination of one or more of these, and may be configured to control the processor to perform a desired operation and to issue commands to the processor, either independently or collectively. The software may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processor or for providing commands or data to the processor. The software may also be distributed over networked computer systems, and stored or executed in a distributed manner.

The software may also be implemented in the form of program commands that can be executed through various computer means and recorded or stored in the memory. The memory may be a computer-readable recording medium, and program commands, data files, data structures, etc. may be recorded singly or in combination in the computer-readable recording medium. The program commands stored in the memory may be based on a command system specifically designed and configured for the embodiment of the present invention, or may follow a command system known and available to those skilled in the art of computer software, such as a command system exemplified by the assembly language, C, C++, Java, Python, etc. It should be understood that the command system and the program commands therefrom include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by the device and/or the processor according to an embodiment of the present invention using an interpreter or the like.

The computer-readable recording medium constituting the device according to one embodiment of the present invention, including the memory described in this specification, may include a temporary or volatile recording medium that is maintained only while the processor is operating, such as a processor cache, a RAM, a flash memory, or a relatively non-volatile or long-term recording medium, such as a magnetic media such as a hard disk, a floppy disk, and a magnetic tape, an optical media such as a CD-ROM, a DVD, a magneto-optical media such as a floptical disk, or a solid state memory, or may include a read-only recording medium, such as a ROM arranged on hardware, and further, the hardware itself configured to perform an operation equivalent to a series of program commands by a hard-wired structure by circuit wiring, and each step for performing the operation for implementing the embodiment of the present invention can be viewed as being recorded by the connection and arrangement of the hardware components, and thus the connection and arrangement method is the memory and It is obvious to a person skilled in the art that they can be considered equivalent.

The embodiments described above with respect to the processor and the memory are not mutually exclusive and may be selected or combined and implemented as needed. For example, a single hardware device may be configured to operate as a module composed of one or more of the software to perform the operations of an embodiment of the present invention, and vice versa. As another example, in the present specification, all or part of the operations assigned to a certain functional unit may be implemented by one or more of the software stored in a device according to an embodiment of the present invention (preferably, in any one of the recording media belonging to the category of the memory) and configured to be executed by the processor. In such a case, such a functional unit may be referred to as a functional unit "included" in the processor.

Although the present invention has been described with reference to drawings and embodiments, as already mentioned above, it does not mean that the scope of protection of the present invention is limited to the drawings or embodiments presented above, and it will be understood that a person skilled in the relevant technical field can modify and change the present invention in various ways within a scope that does not depart from the spirit and scope of the present invention described in the claims of the present invention patent.

Claims (20)

In the video decryption method,
A step of obtaining information about the current block from a bit string;
A step of obtaining a first reference vector based on information about the current block;
A step of obtaining information about a first block area based on the first reference vector;
A step of obtaining a second reference vector based on information about the first block area;
A step of obtaining a combined vector based on the first reference vector and the second reference vector;
An image decoding method, comprising: a step of generating a prediction block for the current block using the combined vector.
In the first paragraph,
An image decoding method, characterized in that the first reference vector is a block vector and the second reference vector is a motion vector.
In the first paragraph,
An image decoding method, further comprising: a step of adding a scaled vector based on at least one of a temporal distance and a spatial distance based on the above-mentioned combined vector to a list of reference vector candidates.
In the first paragraph,
A method for decoding an image, characterized in that, when the above-mentioned combined vector is used for an affine prediction mode, whether or not to use it is determined based on at least one of the horizontal and vertical sizes of the block.
In the first paragraph,
An image decoding method, characterized in that at least one of the first reference vector, the second reference vector, and the combined vector is corrected through template matching.
In the first paragraph,
An image decoding method, further comprising: a step of independently applying a combination vector to each of the divided blocks when the current block is composed of a plurality of divided blocks.
In the first paragraph,
generating at least one reference vector candidate list based on information about the current block; and
An image decoding method, further comprising: a step of obtaining the first reference vector based on the list of reference vector candidates.
In paragraph 7,
A video decoding method, characterized in that the reference vector candidate list includes a reference vector candidate list based on at least one of a block vector list, a motion vector list, a merge list, an AMVP list, and an affine list.
In the first paragraph,
An image decoding method further comprising: a step of obtaining the second reference vector by an operation based on the plurality of reference vectors when a plurality of reference vectors exist in the first block area.
In the first paragraph,
The step of generating a prediction block for the current block using the above combination vector is as follows:
A method for decoding an image, comprising: performing intra-screen prediction using the above combined vector as a block vector for the current screen.
In the image encoding method,
A step of obtaining a first reference vector for the current block;
A step of obtaining information about a first block area based on the first reference vector;
A step of obtaining a second reference vector based on information about the first block area;
A step of obtaining a combined vector based on the first reference vector and the second reference vector;
A step of generating a prediction block for the current block using the above combination vector;
A step of performing prediction encoding for the current block based on the prediction block; and
An image encoding method, comprising: a step of outputting information about the first reference vector and information about obtaining the combined vector as part of a bit string.
In paragraph 11,
A video encoding method, characterized in that the first reference vector is a block vector and the second reference vector is a motion vector.
In paragraph 11,
A method for encoding an image, further comprising: adding a scaled vector based on at least one of a temporal distance and a spatial distance based on the above-mentioned combined vector to a list of reference vector candidates.
In paragraph 11,
A video encoding method, characterized in that, when the above-mentioned combined vector is used for an affine prediction mode, whether or not to use it is determined based on at least one of the horizontal and vertical sizes of the block.
In paragraph 11,
An image encoding method, characterized in that at least one of the first reference vector, the second reference vector, and the combined vector is corrected through template matching.
In paragraph 11,
An image encoding method, further comprising: a step of independently applying a combination vector to each of the divided blocks when the current block is composed of a plurality of divided blocks.
In paragraph 11,
A video encoding method, further comprising: a step of generating at least one reference vector candidate list based on information about the current block; and a step of obtaining the first reference vector based on the reference vector candidate list.
In paragraph 17,
A video encoding method, characterized in that the reference vector candidate list includes a reference vector candidate list based on at least one of a block vector list, a motion vector list, a merge list, an AMVP list, and an affine list.
In paragraph 11,
The step of generating a prediction block for the current block using the above combination vector is as follows:
A method for encoding an image, comprising: performing intra-screen prediction using the above combined vector as a block vector for the current screen.
In the video decoding device,
A receiving unit that receives a bit string;
A parser unit that obtains information about the current block from the above bit string;
A prediction unit that obtains a first reference vector based on information about the current block, obtains information about a first block area based on the first reference vector, obtains a second reference vector based on information about the first block area, and obtains a combined vector based on the first reference vector and the second reference vector; and
An image decoding device, comprising a prediction decoding unit that generates a prediction block for the current block using the above combination vector.