HK1189734A - Method and apparatus for decoding video data - Google Patents

Description

Motion vector prediction in video coding

Priority is claimed for U.S. provisional application No. 61/477,561, filed on.4/20/2011 and U.S. provisional application No. 61/512,765, filed on.7/28/2011, the entire contents of both of which are incorporated herein by reference.

Technical Field

The present disclosure relates to video coding.

Background

Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, Personal Digital Assistants (PDAs), laptop or desktop computers, tablet computers, electronic book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video gaming consoles, cellular or satellite radio telephones, so-called "smart phones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video compression techniques, such as those described in: standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4 part 10 (advanced video coding (AVC)), the High Efficiency Video Coding (HEVC) standard currently under development, and extensions of these standards. Video devices may transmit, receive, encode, decode, and/or store digital video information more efficiently by implementing these video compression techniques.

Video compression techniques perform spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (i.e., a picture or a portion of a picture) may be partitioned into multiple video blocks, which may also be referred to as treeblocks, Coding Units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture may be encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures.

Spatial or temporal prediction generates a predictive block for the block to be coded. The residual data represents pixel differences between the original block to be coded and the predictive block. An inter-coded block is encoded according to a motion vector that points to a block of reference samples that forms a predictive block and residual data that indicates a difference between the coded block and the predictive block. The intra-coded block is encoded according to the intra-coding mode and the residual data. For further compression, the residual data may be transformed from the pixel domain to the transform domain, producing residual transform coefficients that may then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, may be scanned in order to generate a one-dimensional vector of transform coefficients, and entropy coding may be applied to achieve even more compression.

Disclosure of Invention

In general, techniques are described for coding video data. This disclosure describes techniques for performing motion vector prediction, motion estimation, and motion compensation when inter-mode coding (i.e., coding a current block relative to blocks of other pictures) in Multiview Video Coding (MVC). In general, MVC is a video coding standard for encapsulating video data for multiple views. Each view may correspond to a different perspective or angle at which corresponding video data of a common scene was captured. The techniques of this disclosure generally include predicting motion prediction data in the context of multiview video coding. That is, for example, in accordance with the techniques of this disclosure, a disparity motion vector from a block that is in the same or different view as the block currently being coded may be used to predict a motion vector for the current block. In another example, in accordance with the techniques of this disclosure, temporal motion vectors from blocks that are in the same or different view as the block currently being coded may be used to predict the motion vector of the current block.

In an example, an aspect of this disclosure relates to a method of coding video data, the method comprising: identifying a first block of video data in a first temporal location from a first view, wherein the first block is associated with a first disparity motion vector; determining a motion vector predictor for a second motion vector associated with a second block of video data, wherein the motion vector predictor is based on the first disparity motion vector; wherein when the second motion vector comprises a disparity motion vector, determining the motion vector predictor comprises scaling the first disparity motion vector to produce a scaled motion vector predictor, wherein scaling the first disparity motion vector comprises applying a scaling factor to the first disparity motion vector, the scaling factor comprising a line of sight of the second disparity motion vector divided by a line of sight of the first motion vector; and coding prediction data for the second block using the scaled motion vector predictor.

In another example, aspects of this disclosure relate to an apparatus for coding video data, the apparatus comprising one or more processors configured to: identifying a first block of video data in a first temporal location from a first view, wherein the first block is associated with a first disparity motion vector; determining a motion vector predictor for a second motion vector associated with a second block of video data, wherein the motion vector predictor is based on the first disparity motion vector; wherein when the second motion vector comprises a disparity motion vector, the one or more processors are configured to determine the motion vector predictor by scaling the first disparity motion vector to generate a scaled motion vector predictor, wherein scaling the first disparity motion vector comprises applying a scaling factor to the first disparity motion vector, the scaling factor comprising a line of sight of the second disparity motion vector divided by a line of sight of the first motion vector; and code prediction data for the second block based on the scaled motion vector predictor.

In another example, aspects of this disclosure relate to an apparatus for coding video data, the apparatus comprising: means for identifying a first block of video data in a first temporal location from a first view, wherein the first block is associated with a first disparity motion vector; means for determining a motion vector predictor for a second motion vector associated with a second block of video data, wherein the motion vector predictor is based on the first disparity motion vector; wherein when the second motion vector comprises a disparity motion vector, the means for determining the motion vector predictor is configured to determine the motion vector predictor by scaling the first disparity motion vector to produce a scaled motion vector predictor, wherein scaling the first disparity motion vector comprises applying a scaling factor to the first disparity motion vector, the scaling factor comprising a line of sight of the second disparity motion vector divided by a line of sight of the first motion vector; and code prediction data for the second block based on the scaled motion vector predictor.

In another example, aspects of this disclosure relate to a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: identifying a first block of video data in a first temporal location from a first view, wherein the first block is associated with a first disparity motion vector; determining a motion vector predictor for a second motion vector associated with a second block of video data, wherein the motion vector predictor is based on the first disparity motion vector; wherein when the second motion vector comprises a disparity motion vector, the instructions cause the one or more processors to determine the motion vector predictor by scaling the first disparity motion vector to produce a scaled motion vector predictor, wherein scaling the first disparity motion vector comprises applying a scaling factor to the first disparity motion vector, the scaling factor comprising a line of sight of the second disparity motion vector divided by a line of sight of the first motion vector; and code prediction data for the second block based on the scaled motion vector predictor.

In another example, aspects of this disclosure relate to a method of coding video data, the method comprising: identifying a first block of video data in a first temporal location from a first view, wherein the first block of video data is associated with a first temporal motion vector; when a second motion vector associated with a second block of video data includes a temporal motion vector and the second block is from a second view, determining a motion vector predictor for the second motion vector based on the first temporal motion vector; and coding prediction data for the second block using the motion vector predictor.

In another example, aspects of this disclosure relate to an apparatus for coding video data, the apparatus comprising one or more processors configured to: identifying a first block of video data in a first temporal location from a first view, wherein the first block of video data is associated with a first temporal motion vector; when a second motion vector associated with a second block of video data includes a temporal motion vector and the second block is from a second view, determining a motion vector predictor for the second motion vector based on the first temporal motion vector; and coding prediction data for the second block using the motion vector predictor.

In another example, aspects of this disclosure relate to an apparatus that codes video data, the apparatus comprising: means for identifying a first block of video data in a first temporal location from a first view, wherein the first block of video data is associated with a first temporal motion vector; means for determining, when a second motion vector associated with a second block of video data includes a temporal motion vector and the second block is from a second view, a motion vector predictor for the second motion vector based on the first temporal motion vector; and code prediction data for the second block using the motion vector predictor.

In an example, aspects of this disclosure relate to a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: identifying a first block of video data in a first temporal location from a first view, wherein the first block of video data is associated with a first temporal motion vector; when a second motion vector associated with a second block of video data includes a temporal motion vector and the second block is from a second view, determining a motion vector predictor for the second motion vector based on the first temporal motion vector; and coding prediction data for the second block using the motion vector predictor.

The details of one or more aspects of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.

Drawings

FIG. 1 is a block diagram illustrating an example video encoding and decoding system that may utilize the techniques described in this disclosure.

FIG. 2 is a block diagram illustrating an example video encoder that may implement the techniques described in this disclosure.

FIG. 3 is a block diagram illustrating an example video decoder that may implement the techniques described in this disclosure.

Fig. 4 is a conceptual diagram illustrating example multi-view video coding (MVC) prediction modes.

Fig. 5 is a block diagram illustrating example locations of motion vector predictor candidates.

Fig. 6 is a conceptual diagram illustrating generating and scaling a motion vector predictor according to an aspect of this disclosure.

Fig. 7 is another conceptual diagram illustrating generating and scaling a motion vector predictor according to an aspect of this disclosure.

Fig. 8 is another conceptual diagram illustrating generating and scaling a motion vector predictor according to an aspect of this disclosure.

Fig. 9 is a flow diagram illustrating an example method of coding prediction information for a block of video data.

Fig. 10 is a conceptual diagram illustrating generation of a motion vector predictor from a block in a different view than a current block.

Fig. 11 is a flow diagram illustrating an example method of generating a motion vector predictor from a block in a different view than a current block.

Detailed Description

According to some video coding systems, motion estimation and motion compensation may be used to reduce temporal redundancy in video sequences in order to achieve data compression. In this case, a motion vector may be generated that identifies a predictive block of video data (e.g., a block from another video picture or slice), which may be used to predict the value of the current video block being coded. Values of the predictive video block are subtracted from values of the current video block to generate a block of residual data. Motion information, such as motion vectors, motion vector indices, prediction directions, or other information, is communicated from a video encoder to a video decoder along with residual data. The decoder may locate the same predictive block (based on the motion vector) and reconstruct the encoded video block by combining the residual data with the data of the predictive block.

In some cases, predictive coding of motion vectors is also applied to further reduce the amount of data required to convey the motion vectors. When a motion vector is established, the motion vector is established from the target picture to the reference picture. The motion vector may be predicted spatially or temporally. The spatially predicted motion vectors are associated with available spatial blocks (blocks of the same temporal instant). The temporally predicted motion vectors are associated with available temporal blocks (blocks of different temporal instants). In the case of motion vector prediction, rather than encoding and communicating the motion vectors themselves, the encoder encodes and communicates the Motion Vector Difference (MVD) relative to a known (or knowable) motion vector. In h.264/AVC, the known motion vectors that can be used by the MVD to define the current motion vector can be defined by a so-called Motion Vector Predictor (MVP). As a valid MVP, the motion vector must point to the same picture as the motion vector currently coded by MVP and MVD.

A video coder may build a motion vector predictor candidate list that includes several neighboring blocks in spatial and temporal directions as candidates for MVP. In this case, the video encoder may select the most accurate predictor from the candidate set based on an analysis of the encoding rate and distortion (e.g., using a rate-distortion cost analysis or other coding efficiency analysis). The motion vector predictor index (MVP _ idx) may be transmitted to the video decoder to tell the decoder where to locate the MVP. The MVD is also communicated. The decoder may combine the MVD and MVP (which are defined by the motion vector predictor index) in order to reconstruct the motion vector.

So-called "merge modes" may also be available in which the current video block being coded inherits motion information (e.g., motion vectors, reference picture indices, prediction directions, or other information) of neighboring video blocks. The index value may be used to identify the neighbor from which the current video block inherits its motion information.

Multiview Video Coding (MVC) is a video coding standard for encapsulating video data for multiple views. In general, each view corresponds to a different perspective or angle at which corresponding video data of a common scene is captured. MVC provides a set of metadata, i.e., descriptive data (collectively and individually) of views.

The coded views may be used for three-dimensional (3D) display of video data. For example, two views (e.g., left and right eye views of a human viewer) may be displayed simultaneously or near simultaneously using different polarizations of light, and the viewer may wear passive polarized glasses such that each of the viewer's eyes receives a respective one of the views. Alternatively, the viewer may wear active glasses with shutters (shutter) independently for each eye, and the display may rapidly alternate between images for each eye in synchronization with the glasses.

In MVC, a particular picture of a particular view is referred to as a view component. That is, the view component of a view corresponds to a particular temporal instant of the view. Typically, the same or corresponding objects of the two views are not co-located. The term "disparity vector" may be used to refer to a vector that indicates a displacement of an object in a picture of a view relative to a corresponding object in a different view. This vector may also be referred to as a "shift vector". Disparity vectors may also apply to pixels or blocks of video data of a picture. For example, pixels in a picture of a first view may be shifted by a particular disparity with respect to different camera positions from which the first view and second view are captured relative to corresponding pixels in a picture of a second view. In some examples, disparity may be used to predict a motion vector from a view to another view.

In the context of MVC, pictures of one view may be predicted from pictures of another view. For example, a block of video data may be predicted with respect to a block of video data in a reference picture of the same temporal instant but a different view. In an example, the block currently being coded may be referred to as the "current block". Predicting a motion vector of a current block from blocks in different views but at the same temporal instant is referred to as a "disparity motion vector". Disparity motion vectors are typically applicable in the context of multiview video coding where more than one view is available. According to this disclosure, the "line of sight" of a disparity motion vector may refer to a translation difference between a view of a reference picture and a view of a target picture. That is, the line of sight may be represented as a view identifier difference between the view identifier of the reference picture and the view identifier of the target picture.

Another type of motion vector is a "temporal motion vector". In the context of multiview video coding, a temporal motion vector refers to a motion vector that predicts a current block from blocks in different temporal instances, but within the same view. According to this disclosure, the "temporal distance" of a temporal motion vector may refer to a Picture Order Count (POC) distance from a reference picture to a target picture.

Certain techniques of this disclosure are directed to using motion information (e.g., motion vectors, motion vector indices, prediction directions, or other information) associated with blocks of video data in a multiview setting to predict motion information for a block currently being coded. For example, according to aspects of this disclosure, motion vectors predicted from different views may be added as candidates for one or more motion vector lists for motion vector prediction of a current block. In some examples, a video coder may use a disparity motion vector associated with a block in a different view than the block currently being coded to predict a motion vector for the current block, and may add the predicted disparity motion vector to a list of candidate motion vectors. In other examples, a video coder may use a temporal motion vector associated with a block in a different view than the block currently being coded to predict a motion vector for the current block, and may add the predicted temporal motion vector to the list of candidate motion vectors.

According to aspects of this disclosure, a disparity motion vector may be scaled before the disparity motion vector is used as a motion vector predictor for a block currently being coded. For example, if a disparity motion vector identifies a reference picture that has the same view identifier as the current motion vector being predicted, and the disparity motion vector has a target picture that has the same view identifier as the current motion vector being predicted, the disparity motion vector may not be scaled before it is used to predict the motion vector for the current block. In other examples, the disparity motion vector may be scaled before it is used to predict the motion vector of the current block.

In another example, a disparity motion vector may be predicted from disparity motion vectors associated with spatially neighboring blocks. In this example, scaling may not be required if the view identifier of the reference picture for the disparity motion vector is the same as the view identifier of the reference picture for the motion vector to be predicted (e.g., the motion vector associated with the block currently being predicted). Otherwise, the disparity motion vector may be scaled based on the camera position of the camera used to capture the video data. That is, for example, a disparity motion vector for prediction may be scaled according to a difference between a view identifier of a reference picture of the disparity motion vector and a view identifier of a target picture of the motion vector. In some examples, the scaling of the disparity motion vector may be scaled based on the translation of the view.

In another example, the disparity motion vector may be predicted from disparity motion vectors associated with temporally neighboring blocks. In this example, if the view identifier of the reference picture of the disparity motion vector is the same as the view identifier of the reference picture of the motion vector to be predicted, and the view identifier of the target picture of the disparity motion vector is the same as the view identifier of the reference picture of the motion vector to be predicted, then scaling may not be needed. Otherwise, the disparity motion vector may be scaled based on the difference of the view identifiers, as described with respect to the previous example.

With respect to temporal motion vector prediction, according to aspects of this disclosure, a temporal motion vector of a target picture in a first view may be used to predict a temporal motion vector of the target picture in a second, different view. In some examples, a block in the target picture of the temporal motion vector used for prediction may be co-located with the block currently being predicted in a different view. In other examples, due to disparity between the two views, blocks in the target picture of the temporal motion vector used for prediction may be offset from the current block.

In some examples, when a motion vector predicted from a different view is a temporal motion vector, the motion vector may be scaled based on a difference in Picture Order Count (POC) distances. For example, according to aspects of this disclosure, a motion vector for prediction may not be scaled if the reference picture for the temporal motion vector for prediction has the same POC value as the reference picture of the current motion vector being predicted, and the target picture for the temporal motion vector for prediction has the same POC value as the reference picture of the current motion vector being predicted. Otherwise, the motion vector used for prediction may still be scaled based on the difference in POC values between the reference picture of the motion vector used for prediction and the reference picture of the motion vector currently being predicted.

According to some aspects of this disclosure, temporal and/or disparity motion vectors from different views may be used as MVP candidates. For example, the temporal and/or disparity motion vector may be used to calculate the MVD for the current block. According to other aspects of this disclosure, temporal and/or disparity motion vectors from different views may be used as merge candidates. For example, the current block may inherit a temporal and/or disparity motion vector. In these examples, the index value may be used to identify the neighbor from which the current video block inherits its motion information. In any case, the disparity and/or temporal motion vector from a different view used as an MVP or merge candidate may be scaled before the disparity and/or temporal motion vector is used as an MVP or merge candidate.

Fig. 1 is a block diagram illustrating an example video encoding and decoding system 10 that may utilize techniques for motion vector prediction in multiview coding. As shown in fig. 1, system 10 includes a source device 12, source device 12 providing encoded video data to be later decoded by a destination device 14. Specifically, source device 12 provides video data to destination device 14 via computer-readable medium 16. Source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets (e.g., so-called "smart" phones), so-called "smart" tablets (smart pads), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

Destination device 14 may receive encoded video data to be decoded via computer-readable medium 16. Computer-readable medium 16 may include any type of medium or device capable of moving encoded video data from source device 12 to destination device 14. In an example, computer-readable medium 16 may include a communication medium to enable source device 12 to transmit encoded video data directly to destination device 14 in real-time.

The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium may include any wireless or wired communication medium, such as a Radio Frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the internet. The communication medium may comprise a router, switch, base station, or any other apparatus that may be used to facilitate communications from source device 12 to destination device 14.

In some examples, the encoded data may be output from output interface 22 to a storage device. Similarly, encoded data may be accessed from a storage device through an input interface. The storage device may comprise any of a variety of distributed or locally accessed data storage media such as a hard disk, blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device that may store the encoded video generated by source device 12.

Destination device 14 may access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing encoded video data and transmitting the encoded video data to destination device 14. Example file servers include web servers (e.g., for a website), FTP servers, Network Attached Storage (NAS) devices, or local disk drives. Destination device 14 may access the encoded video data over any standard data connection, including an internet connection. This data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on a file server. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

The techniques of this disclosure are not necessarily limited to wireless applications or environments. The techniques may be applied to video coding in support of any of a variety of multimedia applications, such as: wireless television broadcasts, cable television transmissions, satellite television transmissions, internet streaming video transmissions such as dynamic adaptive streaming over HTTP (DASH), digital video encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

In the example of fig. 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Destination device 14 includes input interface 28, video decoder 30, and display device 32. In accordance with this disclosure, video encoder 20 of source device 12 may be configured to apply the techniques for motion vector prediction in multiview coding. In other examples, the source device and the destination device may include other components or arrangements. For example, source device 12 may receive video data from an external video source 18 (e.g., an external camera). Likewise, destination device 14 may interface with an external display device, rather than comprising an integrated display device.

The illustrated system 10 of fig. 1 is merely an example. Techniques for motion vector prediction in multiview coding may be performed by any digital video encoding and/or decoding device. Although the techniques of this disclosure are generally performed by a video encoding device, the techniques may also be performed by a video encoder/decoder (commonly referred to as a "CODEC"). Furthermore, the techniques of this disclosure may also be performed by a video preprocessor. Source device 12 and destination device 14 are merely examples of such coding devices, where source device 12 generates encoded video data for transmission to destination device 14. In some examples, devices 12, 14 may operate in a substantially symmetric manner such that each of devices 12, 14 includes video encoding and decoding components. Thus, system 10 may support one-way or two-way video transmission between video devices 12, 14, such as for video streaming, video playback, video broadcasting, or video telephony.

Video source 18 of source device 12 may include a video capture device such as a video camera, a video archive containing previously captured video, and/or a video feed interface to receive video from a video content provider. As another alternative, video source 18 may generate computer graphics-based data as the source video, or generate a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a video camera, source device 12 and destination device 14 may form so-called camera phones or video phones. However, as mentioned above, the techniques described in this disclosure may be generally applicable to video coding, and may be applied to wireless and/or wired applications. In each case, the captured video, the pre-captured video, or the computer-generated video may be encoded by video encoder 20. The encoded video information may then be output onto computer-readable medium 16 through output interface 22.

Computer-readable medium 16 may include transitory media such as a wireless broadcast or a wired network transmission; or a storage medium (i.e., a non-transitory storage medium) such as a hard disk, flash drive, compact disc, digital video disc, blu-ray disc, or other computer-readable medium. In some examples, a network server (not shown) may receive encoded video data from source device 12 and provide the encoded video data to destination device 14, e.g., via a network transmission. Similarly, a computing device of a media production facility (e.g., a disc stamping facility) may receive encoded video data from source device 12 and generate a disc containing the encoded video data. Thus, in various examples, computer-readable medium 16 may be understood to include one or more computer-readable media in various forms.

Input interface 28 of destination device 14 receives information from computer-readable medium 16. The information of computer-readable medium 16 may include syntax information defined by video encoder 20 that is also used by video decoder 30 and includes syntax elements that describe characteristics and/or processing of blocks and other coded units (e.g., GOPs). Display device 32 displays the decoded video data to a user, and may comprise any of a variety of display devices, such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or another type of display device.

Video encoder 20 and video decoder 30 may operate according to a video coding standard, such as the High Efficiency Video Coding (HEVC) standard currently under development, and may conform to the HEVC test model (HM). Alternatively, video encoder 20 and video decoder 30 may operate according to, for example, the ITU-T H.264 standard or other proprietary or industrial standards known as MPEG-4 part 10 (advanced video coding (AVC) or extensions of these standards, however, the techniques of this disclosure are not limited to any particular coding standard other examples of video coding standards include MPEG-2 and ITU-T H.263, although not shown in FIG. 1, in some aspects, however, video encoder 20 and video decoder 30 may each be integrated with an audio encoder and decoder, and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software, to handle encoding of both audio and video in a common data stream or in separate data streams, if applicable, the MUX-DEMUX unit may conform to the ITU h.223 multiplexer protocol or other protocols such as the User Datagram Protocol (UDP).

The ITU-T H.264/MPEG-4(AVC) standard is formulated by the ITU-T Video Coding Experts Group (VCEG) together with the ISO/IEC Moving Picture Experts Group (MPEG) as a product of collective collaboration called Joint Video Team (JVT). In some aspects, the techniques described in this disclosure may be applied to devices that substantially conform to the h.264 standard. The h.264 standard is described by the ITU-T research group in the ITU-T international standard h.264 "Advanced Video Coding for general audio visual services" at 3 months 2005, which may be referred to herein as the h.264 standard or the h.264 specification, or the h.264/AVC standard or specification. The Joint Video Team (JVT) is still working on extensions to H.264/MPEG-4 AVC.

JCT-VC is dedicated to the development of the HEVC standard. HEVC standardization efforts are based on an evolution model of the video coding device, referred to as the HEVC test model (HM). The HM assumes several additional capabilities of the video coding device relative to existing devices in accordance with, for example, ITU-T H.264/AVC. For example, h.264 provides nine intra-prediction encoding modes, while HM may provide up to thirty-three intra-prediction encoding modes.

In general, the working model of the HM describes that a video picture (or "frame") may be divided into a sequence of tree blocks or Largest Coding Units (LCUs) that include both luma and chroma samples. Syntax data within the bitstream may define a size of an LCU, which is the largest coding unit in terms of the number of pixels. A slice includes a number of consecutive treeblocks in coding order. A picture may be partitioned into one or more slices. Each treeblock may be split into Coding Units (CUs) according to a quadtree. In general, a quadtree data structure includes one node per CU, where the root node corresponds to a tree-type block. If a CU is split into four sub-CUs, the node corresponding to the CU includes four leaf nodes, each of which corresponds to one of the sub-CUs.

Each node of the quadtree data structure may provide syntax data for the corresponding CU. For example, a node in a quadtree may include a split flag that indicates whether a CU corresponding to the node is split into sub-CUs. Syntax elements of a CU may be defined recursively and may depend on whether the CU is split into sub-CUs. If a CU does not split further, it is called a leaf CU. In the present invention, four sub-CUs of a leaf CU will be referred to as leaf CUs even if there is no explicit split of the original leaf CU. For example, if a CU of size 16 × 16 is not further split, then although the 16 × 16CU is never split, the four 8 × 8 sub-CUs will also be referred to as leaf CUs.

The meaning of a CU is similar to that of a macroblock of the h.264 standard, except that the CU has no size difference. For example, a tree-type block may be split into four child nodes (also referred to as child CUs), and each child node may in turn be a parent node and split into another four child nodes. The last non-split child node (referred to as a leaf node of the quadtree) contains a coding node (also referred to as a leaf CU). Syntax data associated with a coded bitstream may define a maximum number of times a treeblock may be split (referred to as a maximum CU depth), and may also define a minimum size of a coding node. Thus, the bitstream may also define a minimum coding unit (SCU). This disclosure uses the term "block" to refer to any of a CU, PU, or TU in the context of HEVC, or similar data structures in the context of other standards (e.g., macroblocks and sub-blocks in h.264/AVC).

A CU includes a coding node and a number of Prediction Units (PUs) and Transform Units (TUs) associated with the coding node. The size of a CU corresponds to the size of the coding node, and the shape must be square. The size of a CU may range from 8 × 8 pixels up to the size of a tree-type block with a maximum of 64 × 64 pixels or larger than 64 × 64 pixels. Each CU may contain one or more PUs and one or more TUs. Syntax data associated with a CU may describe, for example, partitioning of the CU to one or more PUs. The partition mode may differ depending on whether the CU is skipped or direct mode encoded, intra prediction mode encoded, or inter prediction mode encoded. The shape of the PU may be segmented into non-squares. Syntax data associated with a CU may also describe, for example, partitioning of the CU into one or more TUs according to a quadtree. The TU may be square or non-square (e.g., rectangular) in shape.

The HEVC standard allows for a transform according to a TU, which may be different for different CUs. TU sizes are typically set based on the size of PUs within a given CU defined for a partitioned LCU, but this may not always be the case. TUs are typically the same size as a PU, or smaller than a PU. In some examples, a residual sample corresponding to a CU may be subdivided into a number of smaller units using a quadtree structure referred to as a "residual quadtree" (RQT). The leaf nodes of the RQT may be referred to as Transform Units (TUs). The pixel difference values associated with the TUs may be transformed to produce quantifiable transform coefficients.

A leaf-CU may comprise one or more Prediction Units (PUs). In general, a PU represents a spatial region corresponding to all or a portion of a corresponding CU, and may include data used to retrieve reference samples for the PU. In addition, the PU includes data related to prediction. For example, when a PU is intra-mode encoded, data for the PU may be included in a Residual Quadtree (RQT), which may include data describing an intra-prediction mode for a TU corresponding to the PU. As another example, when a PU is inter-mode encoded, the PU may include data defining one or more motion vectors for the PU. The data defining the motion vector for the PU may describe, for example, a horizontal component of the motion vector, a vertical component of the motion vector, a resolution of the motion vector (e.g., one-quarter pixel precision or one-eighth pixel precision), a reference picture to which the motion vector points, and/or a reference picture list (e.g., list 0, list 1, or list C) of the motion vector.

A leaf-CU having one or more PUs may also include one or more Transform Units (TUs). Transform units may be specified using RQTs (also referred to as TU quadtree structures), as discussed above. For example, the split flag may indicate whether a leaf CU is split into four transform units. Each transform unit may then be further split into other sub-TUs. When a TU is not further split, it may be referred to as a leaf-TU. In general, for intra coding, all leaf-TUs belonging to a leaf-CU share the same intra prediction mode. That is, the same intra prediction mode is generally applied to calculate prediction values for all TUs of a leaf-CU. For intra coding, video encoder 20 may calculate a residual value for each leaf-TU using the intra-prediction mode as a difference between the portion of the CU corresponding to the TU and the original block. TUs are not necessarily limited to the size of a PU. Therefore, TU may be larger or smaller than PU. For intra coding, a PU may be arranged with corresponding leaf-TUs of the same CU. In some examples, the maximum size of a leaf-TU may correspond to the size of the corresponding leaf-CU.

Furthermore, the TUs of a leaf CU may also be associated with a respective quadtree data structure, referred to as a Residual Quadtree (RQT). That is, a leaf-CU may include a quadtree that indicates how the leaf-CU is partitioned into TUs. The root node of a TU quadtree generally corresponds to a leaf CU, while the root node of a CU quadtree generally corresponds to a tree block (or LCU). The non-split TU of the RQT is called a leaf-TU. In general, unless otherwise noted, this disclosure uses the terms CU and TU to refer to leaf-CU and leaf-TU, respectively.

A video sequence typically comprises a series of pictures. As described herein, "picture" and "frame" may be used interchangeably. That is, pictures containing video data may be referred to as video frames, or simply "frames". A group of pictures (GOP) generally includes one or more of a series of video pictures. The GOP may include syntax data describing the number of pictures included in the GOP in a header of the GOP, a header of one or more of the pictures, or elsewhere. Each slice of a picture may include slice syntax data that describes an encoding mode of the respective slice. Video encoder 20 typically operates on video blocks within individual video slices in order to encode the video data. The video block may correspond to a coding node within a CU. Video blocks may have fixed or varying sizes, and may have different sizes according to a specified coding standard.

As an example, the HM supports prediction with various PU sizes. Assuming that the size of a particular CU is 2N × 2N, the HM supports intra prediction with PU sizes of 2N × 2N or N × N, and inter prediction with symmetric PU sizes of 2N × 2N, 2N × N, N × 2N, or N × N. The HM also supports asymmetric partitioning for inter prediction with PU sizes of 2N × nU, 2N × nD, nL × 2N, and nR × 2N. In asymmetric partitioning, one direction of a CU is not partitioned, while the other direction is partitioned into 25% and 75%. The portion of the CU corresponding to 25% split is indicated by an indication of "n" followed by "up", "down", "left", or "right". Thus, for example, "2N × nU" refers to a 2N × 2N CU partitioned in the horizontal direction with a top 2N × 0.5N PU and a bottom 2N × 1.5N PU.

In this disclosure, "nxn" and "N by N" are used interchangeably to refer to pixel dimensions of a video block in terms of vertical and horizontal dimensions, e.g., 16 x 16 pixels or 16 by 16 pixels. In general, a 16 × 16 block will have 16 pixels in the vertical direction (y ═ 16) and 16 pixels in the horizontal direction (x ═ 16). Likewise, an nxn block typically has N pixels in the vertical direction and N pixels in the horizontal direction, where N represents a non-negative integer value. The pixels in a block may be arranged in rows and columns. Further, the block does not necessarily need to have the same number of pixels in the horizontal direction as in the vertical direction. For example, a block may include N × M pixels, where M is not necessarily equal to N.

After intra-predictive or inter-predictive coding using PUs of the CU, video encoder 20 may calculate residual data for the TUs of the CU. A PU may include syntax elements that describe a method or mode of generating predictive pixel data in the spatial domain, also referred to as the pixel domain, and a TU may include coefficients in the transform domain after applying a transform, such as a Discrete Cosine Transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform, to residual video data. The residual data may correspond to pixel differences between pixels of the unencoded picture and prediction values corresponding to the PU. Video encoder 20 may form TUs that comprise residual data of the CU and then transform the TUs to generate transform coefficients for the CU.

After applying any transform to generate transform coefficients, video encoder 20 may perform quantization of the transform coefficients. Quantization generally refers to the process of: transform coefficients are quantized to possibly reduce the amount of data used to represent the coefficients, providing further compression. The quantization process may reduce the bit depth associated with some or all of the coefficients. For example, an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

After quantization, the video encoder may scan the transform coefficients, producing a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) coefficients at the front of the array and lower energy (and therefore higher frequency) coefficients at the back of the array. In some examples, video encoder 20 may utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector that may be entropy encoded. In other examples, video encoder 20 may perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, video encoder 20 may entropy encode the one-dimensional vector, e.g., according to Context Adaptive Variable Length Coding (CAVLC), Context Adaptive Binary Arithmetic Coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), Probability Interval Partitioning Entropy (PIPE) coding, or another entropy encoding method. Video encoder 20 may also entropy encode syntax elements associated with the encoded video data for use by video decoder 30 in decoding the video data.

To perform CABAC, video encoder 20 may assign contexts within the context model to symbols to be transmitted. The context may relate to, for example, whether adjacent values of a symbol are non-zero. To perform CAVLC, video encoder 20 may select a variable length code for a symbol to be transmitted. Codewords in VLC may be constructed such that relatively shorter codes correspond to more probable symbols, while longer codes correspond to less probable symbols. In this way, using VLC may achieve bit savings compared to, for example, using codewords of equal length for each symbol to be transmitted. The probability determination may be made based on the context assigned to the symbol.

Video encoder 20 may further send syntax data (e.g., block-based syntax data, picture-based syntax data, and GOP-based syntax data) to video decoder 30 in, for example, a picture header, a block header, a slice header, or a GOP header. The GOP syntax data may describe the number of pictures in a respective GOP, and the picture syntax data may indicate the encoding/prediction mode used to encode the corresponding picture.

Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder or decoder circuits, such as: one or more microprocessors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), discrete logic circuitry, software, hardware, firmware or any combinations thereof. Each of video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined video encoder/decoder (CODEC). A device that includes video encoder 20 and/or video decoder 30 may include an integrated circuit, a microprocessor, and/or a wireless communication device (e.g., a cellular telephone).

Fig. 2 is a block diagram illustrating an example video encoder 20 that may implement the techniques described in this disclosure for predicting motion vectors in multi-view coding. Video encoder 20 may perform intra-coding and inter-coding of video blocks within a video slice. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy of video within a given picture. Inter-coding relies on temporal prediction to reduce or remove temporal redundancy of video within adjacent pictures or within pictures of a video sequence. Intra-mode (I-mode) may refer to any of a number of space-based compression modes. An inter mode, such as uni-directional prediction (P-mode) or bi-directional prediction (B-mode), may refer to any of a number of time-based compression modes.

As shown in fig. 2, video encoder 20 receives video data to be encoded. In the example of fig. 2, video encoder 20 includes a mode select unit 40, a summer 50, a transform unit 52, a quantization unit 54, an entropy encoding unit 56, and a reference picture memory 64. Mode select unit 40, in turn, includes motion estimation unit 42, motion compensation unit 44, intra-prediction unit 46, and partition unit 48. For video block reconstruction, video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and a summer 62. A deblocking filter (not shown in fig. 2) may also be included to filter block boundaries, removing blockiness artifacts from the reconstructed video. The deblocking filter will typically filter the output of summer 62 if necessary. In addition to deblocking filters, additional loop filters (in-loop or post-loop) may be used. These filters are not shown for simplicity, but may filter the output of summer 50 (as an in-loop filter) if desired.

During the encoding process, video encoder 20 receives a picture or slice to be coded. A picture or slice may be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-predictive coding of the received video block relative to one or more blocks in one or more reference pictures to provide temporal compression. Alternatively, intra-prediction unit 46 may perform intra-predictive coding of the received video block relative to one or more neighboring blocks in the same picture or slice as the block to be coded to provide spatial compression. Video encoder 20 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

Furthermore, partitioning unit 48 may partition a block of video data into sub-blocks based on an evaluation of previous partitioning schemes in previous coding passes. For example, partitioning unit 48 may initially partition a picture or slice into LCUs based on rate-distortion analysis (e.g., rate-distortion optimization), and partition each of the LCUs into sub-CUs. Mode select unit 40 may further generate a quadtree data structure that indicates partitioning of LCUs into sub-CUs. Leaf-node CUs of a quadtree may comprise one or more PUs and one or more TUs.

Mode select unit 40 may select one of the coding modes (intra or inter), e.g., based on the error results, and provide the resulting intra or inter coded block to summer 50 to generate residual block data and to summer 62 to reconstruct the encoded block for use as a reference picture. Mode select unit 40 also provides syntax elements such as motion vectors, intra-mode indicators, partition information, and other such syntax information to entropy encoding unit 56.

Motion estimation unit 42, motion vector prediction unit 43, and motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation, performed by motion estimation unit 42, is the process of generating motion vectors that estimate the motion of video blocks. For example, a motion vector may indicate a displacement of a PU of a video block within a current picture relative to a predictive block (or other coded unit) within a reference picture that is related to the current block (or other coded unit) being coded within the current picture.

Predictive blocks are blocks that are found to closely match the block to be coded in terms of pixel differences, which may be determined by Sum of Absolute Differences (SAD), Sum of Squared Differences (SSD), or other difference metrics. In some examples, video encoder 20 may calculate values for piecewise integer (sub-integer) pixel positions of reference pictures stored in reference picture memory 64, which may also be referred to as a reference picture buffer. For example, video encoder 20 may interpolate values for one-quarter pixel positions, one-eighth pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform a motion search relative to the full pixel position and the fractional pixel position and output a motion vector with fractional pixel precision.

Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the location of the PU to the location of a predictive block of a reference picture. Thus, in general, the data for a motion vector may include a reference picture list, an index into the reference picture list (ref _ idx), a horizontal component, and a vertical component. The reference picture may be selected from a first reference picture list (list 0), a second reference picture list (list 1), or a combined reference picture list (list c), each of which identifies one or more reference pictures stored in reference picture memory 64.

Motion estimation unit 42 may generate a motion vector that identifies a predictive block of the reference picture and send the motion vector to entropy encoding unit 56 and motion compensation unit 44. That is, motion estimation unit 42 may generate and send motion vector data that identifies the following to locate the predictive block within the identified picture: a reference picture list containing the predictive block, an index into the reference picture list identifying the picture of the predictive block, and horizontal and vertical components.

In some examples, rather than sending the actual motion vector for the current PU, motion vector prediction unit 43 may predict the motion vector to further reduce the amount of data needed to convey the motion vector. In this case, rather than encoding and conveying the motion vectors themselves, motion vector prediction unit 43 may generate Motion Vector Differences (MVDs) relative to known (or knowable) motion vectors. The known motion vector that may be used by the MVD to define the current motion vector may be defined by a so-called Motion Vector Predictor (MVP). In general, as a valid MVP, the motion vector used for prediction must point to the same reference picture as the motion vector currently being coded.

In some examples, as described in more detail below with respect to fig. 5, motion vector prediction unit 43 may build a motion vector predictor candidate list that includes several neighboring blocks in the spatial and/or temporal direction as candidates for MVP. According to aspects of this disclosure, as described in more detail below, motion vector predictor candidates may also be identified in pictures of different views (e.g., in multiview coding). When multiple motion vector predictor candidates are available (from multiple candidate blocks), motion vector prediction unit 43 may determine the motion vector predictor for the current block according to a predetermined selection criterion. For example, motion vector prediction unit 43 may select the most accurate predictor from the candidate set based on an analysis of the encoding rate and distortion (e.g., using a rate-distortion cost analysis or other coding efficiency analysis). In other examples, motion vector prediction unit 43 may generate an average of the motion vector predictor candidates. Other methods of selecting a motion vector predictor are also possible.

When a motion vector predictor is selected, motion vector prediction unit 43 may determine a motion vector predictor index (MVP flag) that may be used to inform a video decoder (e.g., video decoder 30) where to locate the MVP in a reference picture list containing MVP candidate blocks. Motion vector prediction unit 43 may also determine an MVD between the current block and the selected MVP. The MVP index and MVD may be used to reconstruct a motion vector.

In some examples, motion vector prediction unit 43 may alternatively implement a so-called "merge" mode, in which motion vector prediction unit 43 may "merge" the motion information (e.g., motion vector, reference picture index, prediction direction, or other information) of the predictive video block and the current video block. Thus, with respect to merge mode, the current video block inherits motion information from another known (or knowable) video block. Motion vector prediction unit 43 may build a merge mode candidate list that includes several neighboring blocks in the spatial and/or temporal direction as candidates for a merge mode. Motion vector prediction unit 43 may determine an index value (e.g., merge idx) that may be used to inform a video decoder (e.g., video decoder 30) where to locate the merge video block in a reference picture list containing the merge candidate blocks.

According to aspects of this disclosure, motion vector prediction unit 43 may identify, for example, motion vector predictors for use in generating MVDs or merges in multi-view coding. For example, motion vector prediction unit 43 may identify a disparity motion vector from a block in a different view component than the current block to predict a motion vector for the current block. In other examples, motion vector prediction unit 43 may identify a temporal motion vector from a block in a different view component than the current block to predict the motion vector for the current block.

With respect to disparity motion vector prediction, motion vector prediction unit 43 may identify disparity motion vector candidates from the candidate blocks to predict a motion vector for a video block (referred to as a "current block") currently being coded. The current block may be located in the same picture as the candidate block (e.g., spatially adjacent to the candidate block), or may be located in another picture within the same view as the candidate block. In some examples, motion vector prediction unit 43 may identify a motion vector predictor that refers to a reference picture that is in a different view than the motion vector of the current block. In these examples, motion vector prediction unit 43 may scale the motion vector predictor based on a difference in camera position between two views (e.g., the view referenced by the motion vector predictor, and the view referenced by the current motion vector), in accordance with the techniques of this disclosure. For example, motion vector prediction unit 43 may scale the disparity motion vector predictor according to the difference between the two views. In some examples, the difference between two views may be represented by the difference between the view identifiers (view _ ids) associated with the views.

With respect to temporal motion vector prediction, motion vector prediction unit 43 may identify a temporal motion vector candidate from a candidate block in a different view from the current block to predict a motion vector of the current block. For example, motion vector prediction unit 43 may identify a temporal motion vector predictor candidate in a first view that refers to a block in a picture at another temporal position of the first view. According to aspects of this disclosure, motion vector prediction unit 43 may use the identified temporal motion vector predictor candidate to predict a motion vector associated with the current block in a second, different view. The candidate block (which includes the motion vector predictor candidate) may be co-located with the current block. However, due to disparity between the two views, the relative position of the candidate block may be offset from the current block.

According to aspects of this disclosure, motion vector prediction unit 43 may generate an MVP index (MVP _ flag) and an MVD, or may generate a merge index (merge _ idx). For example, motion vector prediction unit 43 may generate a list of MVPs or merge candidates. According to aspects of this disclosure, MVP and/or merge candidates comprise one or more video blocks located in a different view than the video block currently being decoded.

The motion compensation performed by motion compensation unit 44 may involve taking or generating a predictive block based on the motion vector determined by motion estimation unit 42 and/or information from motion vector prediction unit 43. Furthermore, in some examples, motion estimation unit 42, motion vector prediction unit 43, and motion compensation unit 44 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference picture lists.

Summer 50 forms a residual video block by subtracting pixel values of the predictive block from pixel values of the current video block being coded, forming pixel difference values, as discussed below. In general, motion estimation unit 42 performs motion estimation with respect to luma components, and motion compensation unit 44 uses motion vectors calculated based on luma components for both chroma and luma components. Mode select unit 40 may also generate syntax elements associated with the video blocks and the video slice for use by video decoder 30 in decoding the video blocks of the video slice.

In lieu of inter-prediction (as described above) performed by motion estimation unit 42 and motion compensation unit 44, intra-prediction unit 46 may intra-predict the current block. In particular, intra-prediction unit 46 may determine to use an intra-prediction mode to code the current block. In some examples, intra-prediction unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction unit 46 (or mode select unit 40 in some examples) may select an appropriate intra-prediction mode to use from the tested modes.

For example, intra-prediction unit 46 may calculate rate-distortion values using rate-distortion analysis for various tested intra-prediction modes, and select the intra-prediction mode having the best rate-distortion characteristics among the tested modes. Rate-distortion analysis generally determines an amount of distortion (or error) between a coded block and an original, uncoded block, and a bitrate (i.e., number of bits) used to generate the encoded block, the original, uncoded block being encoded to generate the encoded block. Intra-prediction unit 46 may calculate the ratios of the various encoded blocks from the distortion and rate to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

After selecting the intra-prediction mode for the block, intra-prediction unit 46 may provide information to entropy encoding unit 56 indicating the selected intra-prediction mode for the block. Entropy encoding unit 56 may encode information indicating the selected intra-prediction mode. Video encoder 20 may include definitions of the encoding contexts for the various blocks and indications of the most probable intra-prediction modes, intra-prediction mode index tables, and modified intra-prediction mode index tables to be used for each of the contexts in transmitted bitstream configuration data, which may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables).

Video encoder 20 forms a residual video block by subtracting the prediction data from mode select unit 40 from the original video block being coded. Summer 50 represents the component(s) that perform this subtraction operation. Transform processing unit 52 applies a transform, such as a Discrete Cosine Transform (DCT) or a conceptually similar transform, to the residual block, producing a video block that includes residual transform coefficient values. Transform processing unit 52 may perform other transforms that are conceptually similar to DCT. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. In any case, transform processing unit 52 applies a transform to the residual block, producing a block of residual transform coefficients. The transform may convert the residual information from a pixel value domain to a transform domain (e.g., frequency domain).

Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The quantization level may be modified by adjusting the quantization parameter. In some examples, quantization unit 54 may then perform a scan of a matrix comprising quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scanning.

After quantization, entropy encoding unit 56 entropy codes the quantized transform coefficients. For example, entropy encoding unit 56 may perform Context Adaptive Variable Length Coding (CAVLC), Context Adaptive Binary Arithmetic Coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), Probability Interval Partition Entropy (PIPE) coding, or another entropy coding technique. In the case of context-based entropy coding, the contexts may be based on neighboring blocks. Following entropy coding by entropy encoding unit 56, the coded bitstream may be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, e.g., for later use as a reference block. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the pictures of reference picture memory 64. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate segmented integer pixel values for use in motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reconstructed video block for storage in reference picture memory 64. The reconstructed video block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block to inter-code a block in a subsequent picture.

Fig. 3 is a block diagram illustrating an example video decoder 30 that may implement the techniques described in this disclosure for predicting motion vectors in multi-view coding. In the example of fig. 3, video decoder 30 includes an entropy decoding unit 80, a prediction unit 81, an inverse quantization unit 86, an inverse transform unit 88, a summer 90, and a reference picture memory 92. The prediction unit 81 includes a motion compensation unit 82 and an intra prediction unit 84.

During the decoding process, video decoder 30 receives an encoded video bitstream from video encoder 20, the stream representing video blocks of an encoded video slice and associated syntax elements. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors, and other syntax elements. Entropy decoding unit 80 forwards the motion vectors and other syntax elements to prediction unit 81. Video decoder 30 may receive syntax elements at the video slice level and/or the video block level.

For example, by way of background, video decoder 30 may receive compressed video data that has been compressed into so-called "network abstraction layer units" or NAL units for transmission over a network. Each NAL unit may include a header that identifies the type of data stored to the NAL unit. There are two types of data that are typically stored to NAL units. The first type of data stored to a NAL unit is Video Coding Layer (VCL) data, which includes compressed video data. The second type of data stored to a NAL unit is referred to as non-VCL data, which includes additional information such as parameter sets and Supplemental Enhancement Information (SEI) defined as header data common to a large number of NAL units.

For example, parameter sets may contain sequence level header information (e.g., in a Sequence Parameter Set (SPS)) and infrequently changing picture level header information (e.g., in a Picture Parameter Set (PPS)). Infrequently changing information contained in the parameter sets need not be repeated for each sequence or picture, thereby improving coding efficiency. In addition, the use of parameter sets enables out-of-band transmission of header information, thereby avoiding the need for redundant transmission to enable error recovery.

When a video slice is coded as an intra-coded (I) slice, intra-prediction unit 84 of prediction unit 81 may generate prediction data for video blocks of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current picture. When the picture is coded as an inter-coded (i.e., B, P or GPB) slice, motion compensation unit 82 of prediction unit 81 generates predictive blocks for video blocks of the current video slice based on the motion vectors and other syntax elements received from entropy decoding unit 80. The predictive block may be generated from one of the reference pictures within one of the reference picture lists. Video decoder 30 may construct the reference picture list using preset construction techniques based on the reference pictures stored in reference picture memory 92: list 0 and list 1.

Motion compensation unit 82 determines prediction information for video blocks of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to generate predictive blocks for the current video block being decoded. For example, motion compensation unit 82 uses some received syntax elements to determine construction information for one or more of a prediction mode (e.g., intra-prediction or inter-prediction) used to code video blocks of a video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), a reference picture list for a slice, a motion vector for each inter-coded video block of a slice, an inter-prediction state for each inter-coded video block of a slice, and other information used to decode video blocks in a current video slice. In some examples, motion compensation unit 82 may receive certain motion information from motion vector prediction unit 83.

According to aspects of this disclosure, motion vector prediction unit 83 may receive prediction data indicating where to retrieve motion information for the current block. For example, motion vector prediction unit 83 may receive motion vector prediction information such as an MVP index (MVP _ flag), MVD, merge flag (merge _ flag), and/or merge index (merge _ idx), and use this information to identify motion information used to predict the current block. That is, as mentioned above with respect to video encoder 20, according to aspects of this disclosure, motion vector prediction unit 83 may receive an MVP index (MVP _ flag) and an MVD and use this information to determine a motion vector used to predict the current block. Motion vector prediction unit 83 may generate a list of MVPs or merge candidates. According to aspects of this disclosure, MVP and/or merge candidates may comprise one or more video blocks located in a different view than the video block currently being decoded.

The motion vector prediction unit 83 may use MVP or a merge index to identify motion information used to predict a motion vector of the current block. That is, for example, the motion vector prediction unit 83 may identify MVPs from a list of reference pictures using an MVP index (MVP _ flag). Motion vector prediction unit 83 may combine the identified MVP with the received MVD to determine the motion vector for the current block. In other examples, motion vector prediction unit 83 may identify a merge candidate from a list of reference pictures using a merge index (merge idx) to determine motion information for the current block. In any case, after determining the motion information for the current block, motion vector prediction unit 83 may generate a predictive block for the current block.

According to aspects of this disclosure, motion vector prediction unit 83 may determine a motion vector predictor in multi-view coding. For example, motion vector prediction unit 83 may receive particular information that specifies a disparity motion vector from a block in a different view component than the current block, which is used to predict a motion vector for the current block. In other examples, motion vector prediction unit 83 may receive particular information that identifies a temporal motion vector from a block in a different view component than the current block, which is used to predict the motion vector of the current block.

Regarding the disparity motion vector prediction, the motion vector prediction unit 83 may predict a disparity motion vector of the current block from the candidate block. The candidate block may be located in the same picture as the current block (e.g., spatially adjacent to the candidate block), or may be located in another picture within the same view as the current block. The candidate block may also be located in a picture of a different view, but at the same temporal instant as the current block.

For example, with respect to MVP or merge mode, the target picture and the reference picture of disparity motion vector "a" for the current block to be predicted are known (previously determined). For purposes of explanation, assume that the motion vector from the candidate block is "B". According to aspects of this disclosure, if motion vector B is not a disparity motion vector, motion vector prediction unit 83 may consider the candidate block unavailable (e.g., unavailable to predict motion vector a). That is, motion vector prediction unit 83 may disable the ability to use candidate blocks for motion vector prediction purposes.

If motion vector B is a disparity motion vector and the reference picture of motion vector B belongs to the same view as the reference picture of disparity motion vector a and the target picture of motion vector B belongs to the same view as the target picture of disparity motion vector a, motion vector prediction unit 83 may use motion vector B directly as a candidate predictor for motion vector a. Otherwise, the motion vector prediction unit 83 may scale the disparity motion vector B before the disparity motion vector B may be used as a candidate predictor for the motion vector a. In these examples, in accordance with the techniques of this disclosure, motion vector prediction unit 83 may scale the disparity motion vector based on the line of sight of motion vector a and the line of sight of motion vector B. For example, motion vector prediction unit 83 may scale disparity motion vector B by a particular scaling factor that is equal to the line of sight of motion vector a divided by the line of sight of motion vector B. In some examples, motion vector prediction unit 83 may perform such scaling using view identifiers of the reference picture and the target picture.

Regarding the temporal motion vector prediction, the motion vector prediction unit 83 may predict a temporal motion vector of the current block from a candidate block in a view different from that of the current block. For example, motion vector prediction unit 83 may identify a temporal motion vector predictor candidate whose target picture is in a first view and refers to a block in a reference picture at another temporal location of the first view. According to aspects of this disclosure, motion vector prediction unit 83 may use the identified temporal motion vector predictor candidate to predict a motion vector associated with the current block in a second, different view.

For example, with respect to MVP or merge mode, the target picture and the reference picture of temporal motion vector "a" of the current block to be predicted are known (previously determined). For purposes of explanation, assume that the motion vector from the candidate block is "B". According to aspects of this disclosure, if motion vector B from the candidate block is not a temporal motion vector, motion vector prediction unit 83 may consider the candidate block unavailable (e.g., unavailable to predict motion vector a). That is, in some examples, motion vector prediction unit 83 may disable the ability to use candidate blocks for motion vector prediction purposes.

If motion vector B is a temporal motion vector, and the POC of the reference picture of motion vector B is the same as the reference picture of motion vector a, and the POC of the target picture of motion vector B is the same as the target picture of motion vector B, motion vector prediction unit 83 may use motion vector B directly as a candidate predictor for motion vector a. Otherwise, motion vector prediction unit 83 may scale temporal motion vector B based on the temporal distance. The candidate block in a different view, which includes the motion vector predictor candidate, may be co-located with the current block. However, due to disparity between the two views, the relative position of the candidate block may be offset from the current block.

Inverse quantization unit 86 inverse quantizes (i.e., dequantizes) the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 80. The inverse quantization process may include using quantization parameters calculated by video encoder 20 for each video block in the video slice to determine the degree of quantization and the degree of inverse quantization that should (likewise) be applied.

The inverse transform unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to generate a residual block in the pixel domain. According to aspects of this disclosure, inverse transform unit 88 may determine the manner in which the transform is applied to the residual data. That is, for example, inverse transform unit 88 may determine an RQT that represents a manner in which transforms (e.g., DCTs, integer transforms, wavelet transforms, or one or more other transforms) are applied to residual luma samples and residual chroma samples associated with a block of received video data.

After motion compensation unit 82 generates the predictive block for the current video block based on the motion vector and other syntax elements, video decoder 30 forms a decoded video block by summing the residual block from inverse transform unit 88 with the corresponding predictive block generated by motion compensation unit 82. Summer 90 represents the component(s) that perform this summation operation. If necessary, a deblocking filter may also be applied to filter the decoded block in order to remove blockiness artifacts. Other in-loop filters (in or after the coding loop) may also be used to smooth pixel transitions, or otherwise improve video quality. The decoded video blocks in a given picture are then stored in reference picture memory 92, reference picture memory 92 storing reference pictures for subsequent motion compensation. Reference picture memory 92 also stores decoded video for later presentation on a display device (e.g., display device 32 of fig. 1).

Fig. 4 is a conceptual diagram illustrating an example MVC prediction mode. In the example of FIG. 4, eight views are illustrated, and twelve time positions are illustrated for each view. Generally, each row in FIG. 4 corresponds to a view, while each column indicates a temporal location. Each of the views may be identified using a view identifier ("view _ id") that may be used to indicate a relative camera position with respect to the other views. In the example shown in FIG. 4, the view IDs are indicated as "S0" through "S7," although digital view IDs may also be used. In addition, each of the temporal locations may be identified using a Picture Order Count (POC) value that indicates a display order of the pictures. In the example shown in fig. 4, POC values are indicated as "T0" through "T11".

Although MVC has a so-called base view that can be decoded by an h.264/AVC decoder and MVC can support a stereoscopic view pair, MVC can support more than two views as a 3D video input. Thus, a renderer of a client with an MVC decoder may expect 3D video content with multiple views.

Pictures are indicated in fig. 4 using colored blocks that include letters that specify whether the corresponding picture is intra-coded (i.e., an I-frame) or inter-coded in one direction (i.e., as a P-frame) or in multiple directions (i.e., as a B-frame). Generally, prediction is indicated by an arrow, where a pointed-to picture uses the pointed-from object for prediction reference. For example, a P frame at temporal position T0 of view S2 is predicted from an I frame at temporal position T0 of view S0.

As with single-view video coding, pictures of a multi-view video sequence may be predictively coded with respect to pictures at different temporal locations. For example, the b frame at temporal position T1 of view S0 has an arrow pointing thereto from the I frame at temporal position T0 of view S0, indicating that the b frame is predicted from an I frame. However, in addition, in the context of multi-view video coding, pictures can be predicted in an inter-view manner. That is, a view component may use a view component in other views for reference. For example, in MVC, inter-view prediction is achieved as if the view component in another view is an inter-prediction reference. Possible inter-view references are signaled in the Sequence Parameter Set (SPS) MVC extension and may be modified by a reference picture list construction process, which enables flexible ordering of inter-prediction or inter-view prediction references.

Fig. 4 provides various examples of inter-view prediction. In the example of fig. 4, the pictures of view S1 are illustrated as being predicted from pictures of view S1 at different temporal positions, and being inter-view predicted from pictures of the pictures of views S0 and S2 at the same temporal position. For example, the B-frame at temporal position T1 of view S1 is predicted from each of the B-frames at temporal positions T0 and T2 of view S1 and the B-frames at temporal positions T1 of views S0 and S2.

In the example of fig. 4, the capital "B" and lowercase "B" letters are intended to indicate different hierarchical relationships between pictures rather than different encoding methods. In general, capital "B" frames are relatively high in the prediction hierarchy compared to lower case "B" frames. Fig. 4 also illustrates the variation of the prediction hierarchy using different degrees of shading, where a larger number of shaded (i.e., relatively deeper) pictures are higher in the prediction hierarchy than those pictures with less shading (i.e., relatively lighter). For example, all I frames in FIG. 4 are illustrated in full shading, while P frames have a slightly lighter shading, and B frames (and lower case B frames) have various degrees of shading relative to each other, but always lighter than the shading of P and I frames.

In general, a prediction hierarchy is related to view order indices because pictures that are relatively higher in the hierarchy should be decoded before pictures that are relatively lower in the prediction hierarchy are decoded so that those pictures that are relatively higher in the hierarchy can be used as reference pictures during decoding of pictures that are relatively lower in the hierarchy. The view order index is an index indicating a decoding order of view components in an access unit. A view order index may be implied in a parameter set (e.g., SPS).

In this way, a picture used as a reference picture may be decoded prior to decoding a picture encoded with reference to the reference picture. The view order index is an index indicating a decoding order of view components in an access unit. For each view order index i, a corresponding view _ id is signaled. The decoding of the view components follows the ascending order of the view order indices. If all views are presented, the set of view order indices contains a continuously ordered set from zero to one less than the total number of views.

In MVC, a subset of the entire bitstream may be extracted to form a sub-bitstream that still conforms to MVC. There are many possible sub-bitstreams that a particular application may require based on, for example, the following: a service provided by a server, capabilities, support and capabilities of decoders of one or more clients, and/or preferences of one or more clients. For example, a client may only need three views, and there may be two cases. In an example, one client may need a smooth viewing experience and may prefer views with view _ id values S0, S1, and S2, while another other client may need view scalability and prefer views with view _ id values S0, S2, and S4. Note that these two sub-bitstreams may be decoded as independent MVC bitstreams, and may be supported simultaneously.

Fig. 5 is a block diagram illustrating possible motion vector predictor candidates when performing motion vector prediction, including merge mode. That is, for block 100 currently being coded, from neighboring block A₀、A₁、B₀、B₁And B₂May be used to predict the motion information of the block 100, e.g., a motion vector including a horizontal component and a vertical component, a motion vector index, a prediction direction, or other information. In addition, the motion information associated with the co-located block COL may also be used to predict the motion information of the block 100. In the context of motion vector prediction, neighboring block A₀、A₁、B₀、B₁And B₂And co-located block COL may be generally referred to as a motion vector predictor candidate hereinafter.

In some examples, the motion vector predictor candidates shown in fig. 5 may be identified when performing motion vector prediction (e.g., whether to generate MVDs or perform merge mode). In other examples, different candidates may be identified when performing merge mode and motion vector prediction. That is, the video coder may identify a set of motion vector predictor candidates for performing merge mode that is different from the set of motion vector predictor candidates for performing motion vector prediction.

To perform merge mode, in an example, a video encoder (e.g., video encoder 20) may initially determine which motion vectors from the motion vector predictor candidates are available for merging with block 100. That is, in some examples, the motion information of the motion vector predictor candidates may be made unavailable due to, for example, one or more of the motion vector predictor candidates being intra-coded, not yet coded, or not present (e.g., one or more of the motion vector predictor candidates being located in another picture or slice). Video encoder 20 may construct a motion vector predictor candidate list that includes each of the available motion vector predictor candidate blocks.

After constructing the candidate list, video encoder 20 may select a motion vector from the candidate list to use as the motion vector for current block 100. In some examples, video encoder 20 may select the motion vector that best matches the motion vector of block 100 from the candidate list. That is, video encoder 20 may select a motion vector from the candidate list according to a rate-distortion analysis.

Video encoder 20 may provide an indication that block 100 was encoded using the merge mode. For example, video encoder 20 may set a flag or other syntax element that indicates that the motion vector of block 100 is predicted using merge mode. In an example, video encoder 20 may indicate: the inter prediction parameters of the block 100 are inferred from the motion vector predictor candidates by setting merge _ flag x0 y 0. In this example, the array indices x0, y0 may specify the position of the upper left luma sample of the prediction block relative to the upper left luma sample of the picture (or slice) (x0, y 0).

Additionally, in some examples, video encoder 20 may provide an index that identifies a particular merge candidate from which block 100 inherits its motion vector. For example, merge _ idx [ x0] [ y0] may specify a merge candidate index that identifies a picture in the merge candidate list, and where x0, y0 specify the position of the upper-left luma sample of the prediction block relative to the upper-left luma sample of the picture (or slice) (x0, y 0).

A video decoder, such as video decoder 30, may perform similar steps when decoding block 100 to identify appropriate merge candidates. For example, video decoder 30 may receive an indication that block 100 was predicted using merge mode. In an example, video decoder 30 may receive merge _ flag [ x0] [ y0], where (x0, y0) specifies the position of the top-left luma sample of the prediction block relative to the top-left luma samples of the picture (or slice).

In addition, video decoder 30 may construct a merge candidate list. For example, video decoder 30 may receive one or more syntax elements (e.g., flags) indicating video blocks that may be used for motion vector prediction. Video decoder 30 may construct a merge candidate list based on the received flag. According to some examples, video decoder 30 may construct a merge candidate list (e.g., a mergeCandList) according to the following sequence:

1.A₁if availableFlaga₁Is equal to 1

2.B₁If availableFlagB₁Is equal to 1

3.B₀If availableFlagB₀Is equal to 1

4.A₀If availableFlaga₀Is equal to 1

5.B₂If availableFlagB₂Is equal to 1

Col if availableFlagCol equals 1

If several merge candidates have the same motion vector and the same reference index, the merge candidate may be removed from the list.

Video decoder 30 may identify appropriate merge candidates based on the received index. For example, video decoder 30 may receive an index that identifies a particular merge candidate from which block 100 inherits its motion vector. In an example, merge _ idx [ x0] [ y0] may specify a merge candidate index that identifies a picture in the merge candidate list, and wherein x0, y0 specify the position of the top-left luma sample of the prediction block relative to the top-left luma sample of the picture (or slice) (x0, y 0).

In some examples, video decoder 30 may scale the motion vector predictor prior to merging the motion information of the candidate block with block 100. For example, with respect to a temporal motion vector predictor, if the motion vector predictor refers to a particular predictive block in a reference picture, video decoder 30 may scale the motion vector predictor, which is located in a different temporal location than the predictive block referred to by block 100 (e.g., the actual motion vector of block 100). For example, video decoder 30 may scale the motion vector predictor such that it refers to the same reference picture as that of block 100. In some examples, video decoder 30 may scale the motion vector predictor according to a difference in Picture Order Count (POC) values. That is, video decoder 30 may scale the motion vector predictor based on the difference between: the POC distance between the candidate block and the predictive block referred to by the motion vector predictor, and the POC distance between block 100 and the current reference picture (e.g., referred to by the actual motion vector of block 100). After selecting the appropriate motion vector predictor, video decoder 30 may merge the motion information associated with the motion vector predictor with the motion information of block 100.

A similar process may be implemented by video encoder 20 and video decoder 30 to perform motion vector prediction for a current block of video data. For example, video encoder 20 may initially determine which motion vectors from the motion vector predictor candidates are available as MVPs. Motion information from a motion vector predictor candidate may not be available due to, for example, one or more of the motion vector predictor candidates being intra coded, not yet coded, or not present.

In order to determine which motion vector predictor candidates are available, video encoder 20 may, in turn, analyze each of the motion vector predictor candidates according to a priority-based predetermined scheme. For example, for each motion vector predictor candidate, video encoder 20 may determine whether the motion vector predictor refers to the same reference picture as the actual motion vector of block 100. If the motion vector predictor refers to the same reference picture, video encoder 20 may add the motion vector predictor candidate to the MVP candidate list. If the motion vector predictor does not refer to the same reference picture, the motion vector predictor may be scaled (e.g., based on POC distance, as discussed above) before being added to the MVP candidate list.

With respect to a co-located block COL, if the co-located block includes more than one motion vector predictor (e.g., COL is predicted as a B-frame), video encoder 20 may select one of the temporal motion vector predictors from the current list and the current reference picture (for block 100). Video encoder 20 may then add the selected temporal motion vector predictor to the motion vector predictor candidate list.

Video encoder 20 may signal the following information: one or more motion vector predictors are made available by setting enable temporal mvp flag. After building the candidate list, video encoder 20 may select a motion vector from the candidates to be used as a motion vector predictor for block 100. In some examples, video encoder 20 may select candidate motion vectors according to a rate-distortion analysis.

Video encoder 20 may signal the selected motion vector predictor using an MVP index (MVP _ flag) that identifies the MVPs in the candidate list. For example, video encoder 20 may set mvp _10_ flag [ x0] [ y0] to specify the motion vector predictor index of list 0, where x0, y0 specify the position of the upper-left luma sample of the candidate block relative to the upper-left luma sample of the picture (x0, y 0). In another example, video encoder 20 may set mvp _11_ flag [ x0] [ y0] to specify the motion vector predictor index of list 1, where x0, y0 specify the position of the upper-left luma sample of the candidate block relative to the upper-left luma sample of the picture (x0, y 0). In yet another example, video encoder 20 may set mvp _ lc _ flag [ x0] [ y0] to specify the motion vector predictor index of list c, where x0, y0 specify the position of the upper-left luma sample of the candidate block relative to the upper-left luma sample of the picture (x0, y 0).

Video encoder 20 may also generate motion vector difference values (MVDs). The MVD may constitute the difference between the selected motion vector predictor and the actual motion vector of block 100. Video encoder 20 may signal the MVDs with the MVP indices.

Video decoder 30 may perform similar operations using the motion vector predictor to determine the motion vector for the current block. For example, video decoder 30 may receive an indication in a parameter set (e.g., a Picture Parameter Set (PPS)) indicating that motion vector prediction is enabled for one or more pictures. That is, in an example, video decoder 30 may receive enable temporal mvp flag in the PPS. When a particular picture refers to a PPS having an enable temporal mvp flag equal to zero, the reference picture in the reference picture memory may be marked as "unused for temporal motion vector prediction.

If motion vector prediction is implemented, upon receiving block 100, video decoder 30 may construct a MVP candidate list. Video decoder 30 may use the same scheme discussed above with respect to video encoder 20 to construct the MVP candidate list. In some examples, video decoder 30 may perform motion vector scaling similar to that described above with respect to video encoder 20. For example, if a motion vector predictor does not refer to the same reference picture as block 100, the motion vector predictor may be scaled (e.g., based on POC distance, as discussed above) before being added to the MVP candidate list. Video decoder 30 may identify an appropriate motion vector predictor for block 100 using a received MVP index (MVP _ flag) that identifies MVPs in the candidate list. Video decoder 30 may then generate motion vectors for block 100 using the MVPs and the received MVDs.

Fig. 5 generally illustrates merge mode and motion vector prediction in a single view. It should be understood that the motion vector predictor candidate blocks shown in fig. 5 are provided for purposes of example only, and that more, fewer, or different blocks may be used for purposes of achieving prediction of motion information. According to aspects of this disclosure, merge mode and motion vector prediction may also be applied when more than one view is coded (e.g., in MVC), as described below. In these examples, the motion vector predictor and the predictive block may be located in a different view than block 100.

Fig. 6 is a conceptual diagram illustrating the generation and scaling of motion vector predictors in multi-view coding. For example, according to aspects of this disclosure, a video coder (e.g., video encoder 20 or video decoder 30) may scale a disparity motion vector 120(mv) from a disparity motion vector predictor candidate block 122 ("candidate block") to generate a motion vector predictor 124 (mv') for a current block 126. Although fig. 6 is described with reference to video decoder 30, it should be understood that the techniques of this disclosure may be implemented by a variety of other video coders, including other processors, processing units, hardware-based coding units such as encoders/decoders (CODECs), and the like.

In the example of fig. 6, in view component 2(view _ id2), the candidate block 122 is spatially adjacent to the current block 126. The candidate block 122 is inter-predicted and includes a motion vector 120 that refers to (or "points to") the predictive block in view component 0(view id 0). For example, the target picture of motion vector 120 is in view 2(view _ id2) and its reference picture is in view 0(view _ id 0). The current block 126 is also inter-predicted and includes the actual motion vector (not shown in the figure) that refers to the predictive block in view component 1(view id 1). That is, for example, the target picture of the actual motion vector of the current block 126 is in view 2(view _ id2) and its reference block is in view 1(view _ id 1).

According to aspects of this disclosure, video decoder 30 may use the scaled version of motion vector 120 to generate motion vector predictor 124 for current block 126. For example, video decoder 30 may scale motion vector 120 based on a difference in line-of-sight between motion vector 120 and the actual motion vector of current block 126. That is, video decoder 30 may scale motion vector 120 based on the difference between: the camera position of the camera used to capture the predictive block (in the reference picture) of candidate block 122, and the camera position of the camera used to capture the predictive block (in the reference picture) of current block 126. Thus, video decoder 30 may scale disparity motion vector 120 (e.g., the motion vector used for prediction) according to the difference between: the view component, which is referred to by the motion vector 120 of the candidate block 122, and the view component, which is referred to by the actual motion vector of the current block 126.

In an example, video decoder 30 may generate a scaled motion vector predictor for the current block according to equation (1) shown below:

where viewdistance (mv) is equal to the difference between the view ID of the reference picture of motion vector 120 (e.g., ViewId (refpic (mv)) and the view ID of the target picture of motion vector 120 (e.g., ViewId (targetpic (mv)), and the ViewDistance (mv ') is equal to the difference between the view ID of the reference picture of the motion vector predictor 124, e.g., ViewId (RefPic (mv ')), and the view ID of the target picture of the motion vector predictor 124, e.g., ViewId (TargetPic (mv '))), accordingly, in this example, the reference picture RefPic (mv') of the motion vector predictor 124 belongs to the new target view, and the target picture TargetPic (mv') of motion vector predictor 124 belongs to the current view similarly, the reference picture refpic (mv) of motion vector 120 belongs to the view that the candidate motion vector points to, and the target picture targetpic (mv) of motion vector 120 belongs to the current view video decoder 30 may therefore generate a scaled motion vector predictor according to equation (2) below:

where mv' represents the scaled motion vector predictor of the current block, mv represents the motion vector of the candidate block, ViewID (new target) is the view component referred to by the actual motion vector of the current block, ViewID (current) is the view component of the current block, and ViewID (candidate) is the view component of the candidate block.

In case equation (2) is applied to the example in fig. 6, mv' represents the scaled motion vector predictor of the current block 126, mv represents the motion vector 120, ViewID (new target) is the view component referred to by the motion vector 124, ViewID (current) is the view component of the current block 126, and ViewID (candidate) is the view component of the candidate block 122. In the example shown in fig. 4, therefore, motion vector predictor 124 is a motion vector 120 scaled to half (e.g.,that is, video decoder 30 may scale to half both the horizontally-shifted component and the vertically-shifted component of motion vector 120, generating motion vector predictor 124 for current block 126.

The motion vector scaling described with respect to fig. 6 may be performed for both merging and motion vector prediction. That is, for example, video decoder 30 may scale motion vector 120 prior to merging motion vector 120 with the motion information of current block 126. In another example, video decoder 30 may scale motion vector 120 before calculating a motion vector difference value (MVD) from the difference between motion vector predictor 124 and the actual motion vector of current block 126.

As shown in the example of fig. 6, the candidate block 122 and the current block 126 may be located in the same view component. However, in other examples, as described in more detail with respect to fig. 7 and 8, the candidate block may be located in a different view component than the current block.

Fig. 7 is another conceptual diagram illustrating the generation and scaling of motion vector predictors. For example, according to aspects of this disclosure, a video coder (e.g., video encoder 20 or video decoder 30) may scale a disparity motion vector 130(mv) from a disparity motion vector predictor candidate block 132(x ', y ') to generate a motion vector predictor 134(mv ') for a current block 136(x, y), where the candidate block 132 belongs to a different view component than the current block 136. Thus, the process shown and described with respect to fig. 7 may be generally referred to as inter-view disparity motion vector prediction. Although fig. 7 is described with respect to video decoder 30, it should be understood that the techniques of this disclosure may be implemented by a variety of other video coders, including other processors, processing units, hardware-based coding units such as encoders/decoders (CODECs), and the like.

In the example shown in fig. 7, candidate block 132 is located in view component 1(view _ id 1). The candidate block 132 is inter-predicted and includes a motion vector 130(mv) that refers to the predictive block in view component 0(view id 0). For example, the target picture of motion vector 130 is in view 1(view _ id1) and its reference picture is in view 0(view _ id 0). The current block 136 is co-located with the candidate block 132 and is located in view component 2(view _ id 2). As described in more detail below, in some examples, current block 136 may include an actual motion vector (not shown in the figures) that identifies a block in a first reference view (view id 1). That is, for example, the target picture of the actual motion vector of the current block 136 is in view 2(view _ id2) and its reference block may be in view 1(view _ id 1). In other examples, the current block may include an actual motion vector that identifies the block in the second reference view (view _ id 0). That is, for example, the target picture of the actual motion vector of the current block 136 is in view 2(view _ id2) and its reference block may be in view 0(view _ id 0). Thus, the motion vector predictor 134 (mv') may refer to a block in the first reference view (view _ id 1). In another example, the second motion vector predictor 138(mv ") may refer to a block in a second reference view (view _ id 0).

In some examples, the second motion vector predictor 138 may not be available for purposes of achieving motion vector prediction. For example, if a predictive block in a second reference view may be used for direct inter-view prediction, only the second motion vector predictor 138 may be generated. The availability of the predictive block in the second reference view may be specified, for example, in a parameter set (e.g., a Sequence Parameter Set (SPS) or a Picture Parameter Set (PPS)) or a slice header associated with the current block 136.

According to aspects of this disclosure, a video decoder may perform inter-view disparity motion vector prediction using merge mode or using motion vector prediction. With respect to merge mode, video decoder 30 may initially select a "target view" for current block 136. In general, the target view includes the predictive block for current block 136. In some examples, the target view may be a first reference view (shown as view _ id1 in fig. 7). In other examples, the target view may be a second reference view (shown as view _ id0 in fig. 7). However, as mentioned above, in some examples, the second reference view may only be used as the target view if the predictive block in the second reference view may be used for purposes of achieving inter-view prediction.

In some examples, video decoder 30 may select the first reference view as the target view. In other examples, video decoder 30 may select the second reference view (when available) as the target view. The selection of the target view may be determined, for example, based on the availability of predictive blocks and/or a predetermined selection algorithm. The reference index (ref idx) of current block 136 corresponds to the index of the picture of the target view containing the predictive block, which is added to the reference picture list of current block 136.

After selecting the target view, video decoder 30 may locate candidate block 132. In an example, for purposes of illustration, assume that the top-left luma sample of current block 136 is located at coordinates (x, y) in a picture (or slice). Video decoder 30 may determine the co-located coordinates of candidate block 132 in view component 1. In addition, in some examples, video decoder 30 may adjust the coordinates based on a disparity between the view component of current block 136 (view component 2) and the view component of candidate block 132 (view component 1). Accordingly, video decoder 30 may determine the coordinates of candidate block 132 as (x ', y'), where (x ', y',) (x, y) + disparity. In some examples, the disparity may be included in and/or calculated in the SPS, PPS, slice header, CU syntax, and/or PU syntax.

After locating candidate block 132, video decoder 30 may scale motion vector 130 of candidate block 132 based on the difference in line-of-sight between motion vector 130 and the actual motion vector of current block 136. That is, video decoder 30 may scale motion vector 130 based on the difference between: a camera position of a camera used to capture the predictive block of candidate block 132, and a camera position of a camera used to capture the predictive block of current block 136 (e.g., the predictive block in the target view). That is, video decoder 30 may scale disparity motion vector 130 (e.g., the motion vector used for prediction) according to the difference between: the view component, denoted by the motion vector 130 of the candidate block 132, and the view component of the target view.

In an example, video decoder 30 may generate a scaled motion vector predictor for the current block according to equation (3) shown below:

where mv' represents the scaled motion vector predictor of the current block, mv represents the motion vector of the candidate block, ViewID (target) is the view component of the selected target view, ViewID (current) is the view component of the current block, ViewID (second reference) is the view component of the second reference view (if available), and ViewID (reference) is the view component of the first reference view. In some examples, the ViewID (target) minus the ViewID (current) may be referred to as the line of sight of the motion vector predictor 134, while the ViewID (second reference) minus the ViewID (reference) may be referred to as the line of sight of the motion vector 130. That is, the view distance of the motion vector predictor 134 is a difference between a target picture (view _ id1) and a reference picture (view _ id2) of the motion vector predictor 134, and the view distance of the motion vector 130 is a difference between a target picture (view _ id0) and a reference picture (view _ id1) of the motion vector 130.

With equation (3) applied to the example in fig. 7, mv' represents either scaled motion vector predictor 134 or scaled motion vector predictor 138, depending on which view component is selected for the target view. For example, if the first reference view (view _ id1) is selected as the target view, mv' represents the scaled motion vector predictor 134, mv represents the motion vector 130, the ViewID (target) is the view component referred to by the motion vector predictor 134, the ViewID (current) is the view component of the current block 136, the ViewID (second reference) is the view component of the second reference view (view _ id0), and the ViewID (reference) is the view component of the first reference view (view _ id 1). In the example shown in fig. 7, therefore, motion vector predictor 134 is a motion vector 130 scaled by one (e.g.,). That is, the horizontal and vertical shift components of motion vector 130 may be the same as the horizontal and vertical shift components of motion vector predictor 134.

Alternatively, if the second reference view (view _ id0) is selected for the target view, mv' represents the scaled motion vector predictor 138, mv represents the motion vector 130, ViewID (target) is the view component referred to by the motion vector predictor 138, ViewID (current) is the view component of the current block 136, ViewID (second reference) is the view component of the second reference view (view _ id0), and ViewID (reference to the second reference view _ id0)) Is a view component of the first reference view (view _ id 1). In the example shown in fig. 7, therefore, motion vector predictor 138 is a motion vector 130 scaled to two times (e.g.,). That is, video decoder 30 may scale both the horizontally-shifted component and the vertically-shifted component of motion vector 130 to two times, generating motion vector predictor 138 for current block 136.

According to aspects of this disclosure, video decoder 30 may perform similar steps when performing motion vector prediction (e.g., generating MVPs). For example, video decoder 30 may select a target view, which may be a first reference view (view _ id1) or a second reference view (view _ id 0). However, if a reference picture containing a view component of a predictive block for the current block is not available for purposes of achieving inter-view prediction, the corresponding predictor may not be used. Thus, the selection of the target view may be determined, for example, based on the availability of predictive blocks and/or a predetermined selection algorithm.

If the predictive block for the current block 136 is not available for direct inter-view prediction in the first reference view (view _ id1) or the second reference view (view _ id0), the video decoder 30 may not perform motion vector prediction. If at least one predictive block is available, video decoder 30 may select a reference view that includes the predictive block associated with the actual motion vector for current block 136.

After selecting the target view, video decoder 30 may then repeat the steps described above with respect to merge mode. For example, video decoder 30 may locate candidate block 132. That is, video decoder 30 may determine the co-located coordinates of candidate block 132 in view component 1. In addition, in some examples, video decoder 30 may adjust the coordinates based on a disparity between the view component of current block 136 (view component 2) and the view component of candidate block 132 (view component 1).

In addition, after locating candidate block 132, video decoder 30 may scale motion vector 130 of candidate block 132 based on the difference between: a camera position of a camera used to capture the predictive block of candidate block 132, and a camera position of a camera used to capture the predictive block of current block 136 (e.g., the predictive block in the target view). That is, video decoder 30 may scale disparity motion vector 130 (e.g., the motion vector used for prediction) according to the difference between: the view component referred to by the motion vector 130 of the candidate block 132, and the view component of the target view. In some examples, video decoder 30 may perform motion vector predictor scaling using equation (2) above. In other examples, motion vector predictor scaling may be extended to other views as described below with respect to fig. 8.

Video decoder 30 may add candidate block 132 to the candidate list when performing merge mode and/or motion vector prediction (e.g., as described with respect to fig. 5 above). According to aspects of this disclosure, a candidate block may be added to a motion vector predictor candidate list in a variety of ways (e.g., for merge mode or motion vector prediction by MVP). For example, video decoder 30 may construct the candidate list by locating merge mode candidates according to the following scheme:

1.A₁if availableFlaga₁Is equal to 1

V if availableFlagV equals 1

3.B₁If availableFlagB₁Is equal to 1

4.B₀If availableFlagB₀Is equal to 1

5.A₀If availableFlaga₀Is equal to 1

6.B₂If availableFlagB₂Is equal to 1

Col if availableFlagCol equals 1

Where V represents the candidate block 132. In other examples, candidate block 132 may be located and the candidate block 132 added to the candidate list in any other location of the candidate list.

Fig. 8 is another conceptual diagram illustrating generating and scaling a motion vector predictor according to an aspect of this disclosure. For example, according to aspects of this disclosure, a video coder (e.g., video encoder 20 or video decoder 30) may scale a disparity motion vector 140(mv) from a disparity motion vector predictor candidate block 142 to generate a motion vector predictor 144 (mv') for a current block 146, where the candidate block 142 belongs to a different view component than the current block 146. Although fig. 8 is described with respect to video decoder 30, it should be understood that the techniques of this disclosure may be implemented by a variety of other video encoders, including other processors, processing units, hardware-based coding units such as encoders/decoders (CODECs), and the like.

The example shown in fig. 8 extends the motion vector prediction shown and described with respect to fig. 7 to an environment that includes more than three views. For example, as shown in fig. 8, candidate block 142 is located in view component 2(view _ id 2). The candidate block 142 is inter-predicted and includes a motion vector 140(mv) that refers to the predictive block in view component 1(view id 1). For example, the target picture of motion vector 140 is in view 2(view _ id2) and its reference picture is in view 1(view _ id 1). The current block 146 is co-located with the candidate block 142 and is located in view component 3(view _ id 3).

According to aspects of this disclosure, video decoder 30 may select the target view for current block 146 as view component 0(view _ id 0). For example, the target view generally includes a predictive block for the current block. If the picture containing the predictive block is an inter-view reference picture and the predictive block for current block 146 is located in a third reference view (view id0), video decoder 30 may select the third reference view as the target view.

After selecting the target view, video decoder 30 may locate candidate block 142. For example, assuming that the top-left luma sample of current block 146 is located at coordinate (x, y) in the picture (or slice) in view component 3, video decoder 30 may determine the co-located coordinates of candidate block 142 in view component 2. In addition, as mentioned above, video decoder 30 may adjust the coordinates based on the disparity between the view component of current block 146 (view component 3) and the view component of candidate block 142 (view component 2).

After locating candidate block 142, video decoder 30 may scale motion vector 140 of candidate block 142 based on a difference in line-of-sight between motion vector 140 and the actual motion vector of current block 146. That is, video decoder 30 may scale motion vector 130 based on the difference between: a camera position of a camera used to capture the predictive block of candidate block 142, and a camera position of a camera used to capture the predictive block of current block 146 (e.g., the predictive block in the target view). That is, video decoder 30 may scale disparity motion vector 140 (e.g., the motion vector used for prediction) according to the difference between: a view component referred to by the motion vector 140 of the candidate block 142, and a view component of the target view (view _ id 0).

In an example, video decoder 30 may generate a scaled motion vector predictor for the current block according to equation (4) shown below:

where mv' represents the scaled motion vector predictor of the current block, mv represents the motion vector of the candidate block, ViewID (third) is the view component of the third reference view, ViewID (current) is the view component of the current block, ViewID (second reference) is the view component of the second reference view (if available), and ViewID (reference) is the view component of the first reference view. In some examples, the ViewID (third) minus the ViewID (current) may be referred to as the line of sight of the motion vector predictor 144, and the ViewID (second reference) minus the ViewID (reference) may be referred to as the line of sight of the motion vector 140. That is, the view distance of the motion vector predictor 144 is a difference between a target picture (view _ id0) and a reference picture (view _ id3) of the motion vector predictor 144, and the view distance of the motion vector 140 is a difference between a target picture (view _ id1) and a reference picture (view _ id2) of the motion vector 140.

In case equation (3) is applied to the example in fig. 8, mv' represents the scaled motion vector predictor 144. For example, the view id (third) is a third reference view (view _ id0), mv' denotes the scaled motion vector predictor 144, mv denotes the motion vector 140, the view id (current) is a view component of the current block 146, the view id (second reference) is a view component of the second reference view (view _ id1), and the view id (reference) is a view component of the first reference view (view _ id 2). In the example shown in fig. 8, therefore, motion vector predictor 144 is a motion vector 140 scaled to three times (e.g.,). That is, video decoder 30 may scale the horizontal and vertical shift components of motion vector 140 by a factor of three, forming motion vector predictor 144.

Although fig. 7-8 provide examples of inter-view disparity motion vector prediction, it should be understood that these examples are provided for illustration purposes only. That is, the techniques for disparity motion vector prediction may be applied to more or fewer views than shown. Alternatively or additionally, techniques for disparity motion vector prediction may be applied in situations where each view has a different view identifier.

Fig. 9 is a flow diagram illustrating an example method of coding prediction information for a block of video data. The example shown in fig. 9 is generally described as being performed by a video coder. It should be understood that, in some examples, the method of fig. 9 may be implemented by video encoder 20 (fig. 1 and 2) or video decoder 30 (fig. 1 and 3) described above. In other examples, the method of fig. 9 may be performed by a variety of other processors, processing units, hardware-based coding units such as encoders/decoders (CODECs), and the like.

According to the example method shown in fig. 9, the video coder may identify a first block of video data in a first view, wherein the first block of video data is associated with a first disparity motion vector (160). For example, a motion vector for a first block of video data may be a disparity motion vector that identifies a reference block in another view component. The video coder may then determine whether a second motion vector associated with a second block of video data is a disparity motion vector (162).

If the second motion vector is not a disparity motion vector ("no" branch of step 162), the video coder may identify a different motion vector predictor candidate (164). That is, according to some aspects of this disclosure, the ability to use a disparity motion vector (e.g., a first motion vector) to predict a temporal motion vector (e.g., predict a second motion vector when the second motion vector is a temporal motion vector) may be disabled. In these examples, the video coder may identify the first motion vector as unavailable for purposes of achieving motion vector prediction.

If the second motion vector is a disparity motion vector ("yes" branch of step 162), the video coder may scale the first motion vector to generate a motion vector predictor for the second motion vector (166). For example, according to aspects of this disclosure, a video coder may scale a first motion vector to generate a disparity motion vector predictor based on a difference of: a line-of-sight associated with the first disparity motion vector, and a line-of-sight associated with the second motion vector. That is, in some examples, the video coder may scale the motion vector predictor for the second block based on the camera position. For example, the video coder may scale the second motion vector according to the difference in view identifiers as shown and described with respect to fig. 6-8.

The video coder may then code the prediction data for the second block using the scaled motion vector predictor (168). For example, the video coder may code the prediction data of the second block using merge mode or using motion vector prediction. For merge mode, the video coder may directly code the prediction data of the second block using the scaled second motion vector predictor. For motion vector prediction, the video coder may code prediction data for the second block by generating an MVD. The MVD may comprise a difference between the first motion vector and the scaled second motion vector.

It should also be understood that the steps shown and described with respect to fig. 9 are provided as an example only. That is, the steps of the method of fig. 9 need not necessarily be performed in the order shown in fig. 9, and fewer, additional, or alternative steps may be performed.

Fig. 10 is a conceptual diagram illustrating generation of a motion vector predictor from a block in a different view than a current block. For example, according to aspects of this disclosure, a video coder (e.g., video encoder 20 or video decoder 30) may use temporal motion vector 180(mv) from temporal motion vector predictor candidate block 182 to generate motion vector predictor 184 (mv') for current block 186, where candidate block 182 belongs to a different view component than current block 186. Although fig. 10 is described with respect to video decoder 30, it should be understood that the techniques of this disclosure may be implemented by a variety of other video coders, including other processors, processing units, hardware-based coding units such as encoders/decoders (CODECs), and the like.

As shown in fig. 10, the current block 186 is located in view component 1(view _ id 1). The candidate block 182 is located in view component 0(view _ id 0). Candidate block 182 is temporally predicted and includes motion vector 180(mv) that refers to predictive blocks in different temporal locations within the same view component. That is, in the example shown in fig. 10, motion vector 180 identifies a predictive block in a picture having a reference index equal to variable i (ref _ idx ═ i).

Assume that the top-left luma sample of the current block 186 is located at coordinates (x, y) in the picture (or slice). Video decoder 30 may locate candidate block 182 by determining the co-located coordinates of candidate block 182 in view component 0. In some examples, video decoder 30 may adjust the coordinates of candidate block 182 based on the disparity between the view component (view _ id1) of current block 186 and the view component (view _ id0) of candidate block 182. Accordingly, video decoder 30 may determine the coordinates of candidate block 182 as (x ', y'), where (x ', y',) (x, y) + disparity. In some examples, the disparity may be included in and/or calculated in the SPS, PPS, slice header, CU syntax, and/or PU syntax.

According to aspects of this disclosure, video decoder 30 may then remap the reference indices of motion vectors 180 for purposes of achieving prediction. In general, as mentioned above, the data of a motion vector includes a reference picture list, an index into the reference picture list (referred to as ref _ idx), a horizontal component, and a vertical component. In HEVC, there may be two conventional reference picture lists (e.g., list 0 and list 1) and a combined reference picture list (e.g., list c). Without loss of generality, assume that the current reference picture list is list t (which may correspond to any of list 0, list 1, or list c). According to the example shown in fig. 10, motion vector 180 of candidate block 182 may identify a predictive block located in a particular picture in view component 0(view id0), which has a POC value of 2 and its ref idx equal to i. According to aspects of this disclosure, video decoder 30 may identify a co-located predictive block for current block 186 that is at the same temporal instant as current block 186. That is, the predictive block of candidate block 182 has the same temporal position as the predictive block of current block 186, but is located in a picture of two different views.

In an example, if the identified predictive block of current block 186 corresponds to the jth reference picture in reference picture list t of the current picture, video decoder 30 may predict the reference index (ref idx) of current block 186 as j, and video decoder 30 may set motion vector predictor 184 to the same value as motion vector 180. Thus, video decoder 30 effectively remaps the reference index of current block 186 from ref _ idx i to ref _ idx j. That is, video decoder 30 determines that motion vector predictor 184 for current block 186 has the same reference picture list, horizontal component, and vertical component as candidate block 182, however, motion vector predictor 184 refers to the jth reference picture in the reference picture list, not the ith reference picture in the reference picture list.

According to aspects of this disclosure, in some examples, the video decoder may also scale the motion vector predictor 184. For example, if the picture containing the identified predictive block for current block 186 is not included in reference picture list t, video decoder 30 may identify the nearest second picture in reference picture list t. In some examples, video decoder 30 may select a picture that is closer to the picture containing current block 186 as the second picture if the two pictures have equal distances from the picture containing the identified predictive block for current block 186. For purposes of explanation, it is assumed that the identified picture has a reference index k. In this example, video decoder 30 may then predict the reference index of motion vector predictor 184 to be k, and video decoder 30 may scale motion vector predictor 184 based on the difference in Picture Order Count (POC). That is, video decoder 30 may scale motion vector predictor 184 based on the difference between: the distance between the current block 186 and the picture at reference index j, and the distance between the current block 186 and the picture at reference index k.

According to some examples, video decoder 30 may perform the same process when performing motion vector prediction. However, after determining motion vector predictor 184, video decoder 30 may generate a motion vector for current block 186 using the MVD. Motion vector prediction may use the same process. In another example, with respect to motion vector prediction, if the predictive block of current block 186 cannot be located (identified as being located at reference index j above), video decoder 30 may not perform merge mode or motion vector prediction for current block 186. That is, rather than scaling the motion vector predictor 184, the video decoder 30 may consider the motion vector predictor 184 unavailable.

Video decoder 30 may add candidate block 182 to the candidate list for performing merge mode and/or motion vector prediction (e.g., as described with respect to fig. 5 above). According to aspects of this disclosure, candidate block 182 may be added to the motion vector predictor candidate list in a variety of ways (e.g., for merge mode or motion vector prediction by MVP). For example, video decoder 30 may construct the candidate list by locating candidates according to the following scheme:

1.A₁if availableFlaga₁Is equal to 1

V if availableFlagV equals 1

3.B₁If availableFlagB₁Is equal to 1

4.B₀If availableFlagB₀Is equal to 1

5.A₀If availableFlaga₀Is equal to 1

6.B₂If availableFlagB₂Is equal to 1

Col if availableFlagCol equals 1

Where V represents a candidate block 182. In other examples, candidate block 132 may be located and the candidate block 132 added to the candidate list in any other location of the candidate list.

Fig. 11 is a flow diagram illustrating an example method of generating a motion vector predictor. The example shown in fig. 11 is generally described as being performed by a video coder. It should be understood that, in some examples, the method of fig. 11 may be implemented by video coder 20 (fig. 1 and 2) or video decoder 30 (fig. 1 and 3) described above. In other examples, the method of fig. 11 may be performed by a variety of other processors, processing units, hardware-based coding units such as encoders/decoders (CODECs), and the like.

According to the example shown in fig. 11, a video coder may identify a first block of video data in a first temporal location of a first view, wherein the first block is associated with a first temporal motion vector (202). According to aspects of this disclosure, when a second motion vector associated with a second block of video data is a temporal motion vector and the second block is from a second view that is different from the first block ("yes" branch of step 204), the video coder may determine a motion vector predictor based on the first motion vector (206). That is, for example, the video coder may determine, from the first motion vector, a motion vector predictor for predicting the second motion vector. The video coder may also code prediction data for the second block using the motion vector predictor (208). For example, a video coder may use a motion vector predictor in merge mode or use a motion vector predictor to generate an MVD value.

If the second motion vector is not a temporal motion vector and/or the second block of video data is not from a different view than the first block of video data ("no" branch of step 204), the video coder may determine whether the second motion vector is a disparity motion vector (210). According to aspects of this disclosure, if the second motion vector is not a disparity motion vector ("no" branch of step 210), the video coder may identify a different motion vector predictor candidate (212). That is, in some examples, the video coder may not use the first motion vector to predict the second motion vector.

If the second motion vector is a disparity motion vector ("yes" branch of step 210), the video coder may determine whether disparity motion vector prediction is disabled (214). That is, according to some aspects of this disclosure, the ability to use a temporal motion vector (e.g., a first motion vector) to predict a disparity motion vector (e.g., predict a second motion vector when the second motion vector is a disparity motion vector) may be disabled. In these examples, the video coder may identify different motion vector predictor candidates (212) (the "no" branch of step 214).

If the video coder determines that disparity motion vector prediction is enabled (e.g., or the ability to enable/disable this function is not present), the video coder may determine a motion vector predictor for the second motion vector based on the first motion vector (206) (the "yes" branch of step 214). In addition, the video coder may also code prediction data for the second block using the motion vector predictor (208). For example, a video coder may use a motion vector predictor in merge mode or use a motion vector predictor to generate an MVD value.

It should also be understood that the steps shown and described with respect to fig. 11 are provided as an example only. That is, the steps of the method of fig. 11 need not necessarily be performed in the order shown in fig. 11, and fewer, additional, or alternative steps may be performed.

It will be understood that certain acts or events of any of the methods described herein can be performed in different sequences, which can be added, merged, or left out all together, depending on the example (e.g., not all described acts or events are necessary for the practice of the methods). Further, in some examples, acts or events may be performed concurrently, e.g., via multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially. Additionally, while certain aspects of this disclosure are described for clarity as being performed by a single module or unit, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with a video coder.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or program code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media (e.g., data storage media) or communication media including any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol.

In this manner, the computer-readable medium may correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, program code, and/or data structures for implementation of the techniques described in this disclosure. The computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium.

It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory tangible storage media. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The instructions may be executed by one or more processors, such as: one or more Digital Signal Processors (DSPs), general purpose microprocessors, Application Specific Integrated Circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the term "processor," as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques may be fully implemented in one or more circuits or logic components.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, Integrated Circuits (ICs), or collections of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperating hardware units (including one or more processors as described above) in conjunction with suitable software and/or firmware.

Various aspects of the present invention have been described. These and other aspects are within the scope of the appended claims.

Claims (64)

1. A method of coding video data, the method comprising:

identifying a first block of video data in a first temporal location from a first view, wherein the first block is associated with a first disparity motion vector;

determining a motion vector predictor for a second motion vector associated with a second block of video data, wherein the motion vector predictor is based on the first disparity motion vector;

wherein when the second motion vector comprises a disparity motion vector, determining the motion vector predictor comprises scaling the first disparity motion vector to produce a scaled motion vector predictor, wherein scaling the first disparity motion vector comprises applying a scaling factor to the first disparity motion vector, the scaling factor comprising a line of sight of the second disparity motion vector divided by a line of sight of the first motion vector; and

coding prediction data for the second block using the scaled motion vector predictor.

2. The method of claim 1, wherein the ability to determine the motion vector predictor from the first disparity motion vector is disabled when the second motion vector is not a disparity motion vector.

3. The method of claim 1, wherein the second block of video data is from the first view.

4. The method of claim 3, wherein the second block of video data is temporally adjacent to the first block of video data.

5. The method of claim 1, wherein the second block of video data is located in the first temporal location.

6. The method of claim 5, wherein the second block of video data is from a second, different view, and further comprising identifying the first block using a disparity value between the second block of the second view and the first block of the first view.

7. The method of claim 1, wherein the second block of video data is located in a second temporal location different from the first temporal location.

8. The method of claim 1, wherein the second block of video data is from the first view and from the first temporal location.

9. The method of claim 8, wherein the second block of video data is spatially adjacent to the first block of video data in the first temporal location from the first view.

10. The method of claim 1, further comprising adding the motion vector predictor to a candidate list for predicting the second motion vector.

11. The method of claim 10, further comprising scaling the motion vector predictor prior to adding the motion vector predictor to the candidate list.

12. The method of claim 1, wherein a line of sight of a disparity motion vector comprises a difference between: a view identifier of a reference picture, and a view identifier of a target picture associated with the disparity motion vector.

13. The method of claim 1, wherein a line of sight of a disparity motion vector comprises a geometric distance between: a camera position containing a view of a reference picture, and a camera position containing a view of a target picture associated with the disparity motion vector.

14. The method of claim 1, wherein coding the prediction data for the second block of video data comprises encoding the prediction data.

15. The method of claim 1, wherein coding the prediction data of the second block of video data comprises decoding the prediction data.

16. The method of claim 1, further comprising coding the second block of video data using the prediction data for the second block of video data.

17. An apparatus for coding video data comprising one or more processors configured to:

identifying a first block of video data in a first temporal location from a first view, wherein the first block is associated with a first disparity motion vector;

determining a motion vector predictor for a second motion vector associated with a second block of video data, wherein the motion vector predictor is based on the first disparity motion vector;

wherein when the second motion vector comprises a disparity motion vector, the one or more processors are configured to determine the motion vector predictor by scaling the first disparity motion vector to generate a scaled motion vector predictor, wherein scaling the first disparity motion vector comprises applying a scaling factor to the first disparity motion vector, the scaling factor comprising a line of sight of the second disparity motion vector divided by a line of sight of the first motion vector; and

coding prediction data for the second block based on the scaled motion vector predictor.

18. The apparatus of claim 17, wherein, when the second motion vector is not a disparity motion vector, the one or more processors are configured to disable an ability to determine the motion vector predictor from the first disparity motion vector.

19. The apparatus of claim 17, wherein the second block of video data is from the first view.

20. The apparatus of claim 19, wherein the second block of video data is temporally adjacent to the first block of video data.

21. The apparatus of claim 17, wherein the second block of video data is located in the first temporal location.

22. The apparatus of claim 21, wherein the second block of video data is from a second, different view, and wherein the one or more processors are further configured to identify the first block using a disparity value between the second block of the second view and the first block of the first view.

23. The apparatus of claim 17, wherein the second block of video data is located in a second temporal location different from the first temporal location.

24. The apparatus of claim 17, wherein the second block of video data is from the first view and from the first temporal location.

25. The apparatus of claim 24, wherein the second block of video data is spatially adjacent to the first block of video data in the first temporal location from the first view.

26. The apparatus of claim 17, wherein the one or more processors are further configured to add the motion vector predictor to a candidate list for predicting the second motion vector.

27. The apparatus of claim 26, wherein the one or more processors are further configured to scale the motion vector predictor prior to adding the motion vector predictor to the candidate list.

28. The apparatus of claim 17, wherein a line of sight of a disparity motion vector comprises a difference between: a view identifier of a reference picture, and a view identifier of a target picture associated with the disparity motion vector.

29. The apparatus of claim 17, wherein a line of sight of a disparity motion vector comprises a geometric distance between: a camera position containing a view of a reference picture, and a camera position containing a view of a target picture associated with the disparity motion vector.

30. The apparatus of claim 17, wherein to code the prediction data for the second block of video data, the one or more processors are configured to encode the prediction data.

31. The apparatus of claim 17, wherein to code the prediction data for the second block of video data, the one or more processors are configured to decode the prediction data.

32. The apparatus of claim 17, wherein the one or more processors are further configured to code the second block of video data using the prediction data for the second block of video data.

33. An apparatus for coding video data, the apparatus comprising:

means for identifying a first block of video data in a first temporal location from a first view, wherein the first block is associated with a first disparity motion vector;

means for determining a motion vector predictor for a second motion vector associated with a second block of video data, wherein the motion vector predictor is based on the first disparity motion vector;

wherein when the second motion vector comprises a disparity motion vector, the means for determining the motion vector predictor is configured to determine the motion vector predictor by scaling the first disparity motion vector to produce a scaled motion vector predictor, wherein scaling the first disparity motion vector comprises applying a scaling factor to the first disparity motion vector, the scaling factor comprising a line of sight of the second disparity motion vector divided by a line of sight of the first motion vector; and

means for coding prediction data for the second block based on the scaled motion vector predictor.

34. The apparatus of claim 33, further comprising means for disabling an ability to determine the motion vector predictor from the first disparity motion vector when the second motion vector is not a disparity motion vector.

35. The apparatus of claim 33, wherein the second block of video data is from the first view.

36. The apparatus of claim 35, wherein the second block of video data is temporally adjacent to the first block of video data.

37. The apparatus of claim 33, wherein the second block of video data is located in the first temporal location.

38. The apparatus of claim 37, wherein the second block of video data is from a second, different view, and further comprising means for identifying the first block using a disparity value between the second block of the second view and the first block of the first view.

39. The apparatus of claim 33, wherein the second block of video data is located in a second temporal location different from the first temporal location.

40. The apparatus of claim 33, wherein the second block of video data is from the first view and from the first temporal location.

41. The apparatus of claim 33, wherein the second block of video data is spatially adjacent to the first block of video data in the first temporal location from the first view.

42. The apparatus of claim 33, further comprising means for adding the motion vector predictor to a candidate list for predicting the second motion vector.

43. The apparatus of claim 33, further comprising means for scaling the motion vector predictor prior to adding the motion vector predictor to the candidate list.

44. The apparatus of claim 33, wherein a line of sight of a disparity motion vector comprises a difference between: a view identifier of a reference picture, and a view identifier of a target picture associated with the disparity motion vector.

45. The apparatus of claim 33, wherein a line of sight of a disparity motion vector comprises a geometric distance between: a camera position containing a view of a reference picture, and a camera position containing a view of a target picture associated with the disparity motion vector.

46. The apparatus of claim 33, wherein means for coding prediction data for the second block of video data comprises means for encoding the prediction data.

47. The apparatus of claim 33, wherein means for coding prediction data for the second block of video data comprises means for decoding the prediction data.

48. The apparatus of claim 33, further comprising means for coding the second block of video data using the prediction data for the second block of video data.

49. A computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to perform the steps of:

identifying a first block of video data in a first temporal location from a first view, wherein the first block is associated with a first disparity motion vector;

determining a motion vector predictor for a second motion vector associated with a second block of video data, wherein the motion vector predictor is based on the first disparity motion vector;

wherein when the second motion vector comprises a disparity motion vector, the instructions cause the one or more processors to determine the motion vector predictor by scaling the first disparity motion vector to generate a scaled motion vector predictor, wherein scaling the first disparity motion vector comprises applying a scaling factor to the first disparity motion vector, the scaling factor comprising a line of sight of the second disparity motion vector divided by a line of sight of the first motion vector; and

coding prediction data for the second block based on the scaled motion vector predictor.

50. The computer-readable storage medium of claim 49, further comprising instructions that cause the one or more processors to disable an ability to determine the motion vector predictor from the first disparity motion vector when the second motion vector is not a disparity motion vector.

51. The computer-readable storage medium of claim 49, wherein the second block of video data is from the first view.

52. The computer-readable storage medium of claim 51, wherein the second block of video data is temporally adjacent to the first block of video data.

53. The computer-readable storage medium of claim 49, wherein the second block of video data is located in the first temporal location.

54. The computer-readable storage medium of claim 53, wherein the second block of video data is from a second, different view, and further comprising instructions that cause the one or more processors to identify the first block using a disparity value between the second block of the second view and the first block of the first view.

55. The computer-readable storage medium of claim 49, wherein the second block of video data is located in a second temporal location different from the first temporal location.

56. The computer-readable storage medium of claim 49, wherein the second block of video data is from the first view and from the first temporal location.

57. The computer-readable storage medium of claim 56, wherein the second block of video data is spatially adjacent to the first block of video data in the first temporal location from the first view.

58. The computer-readable storage medium of claim 49, further comprising instructions that cause the one or more processors to add the motion vector predictor to a candidate list for predicting the second motion vector.

59. The computer-readable storage medium of claim 58, further comprising instructions that cause the one or more processors to scale the motion vector predictor prior to adding the motion vector predictor to the candidate list.

60. The computer-readable storage medium of claim 49, wherein a line of sight of a disparity motion vector includes a difference between: a view identifier of a reference picture, and a view identifier of a target picture associated with the disparity motion vector.

61. The computer-readable storage medium of claim 49, wherein a line of sight of a disparity motion vector includes a geometric distance between: a camera position containing a view of a reference picture, and a camera position containing a view of a target picture associated with the disparity motion vector.

62. The computer-readable storage medium of claim 49, wherein the instructions that cause the one or more processors to code prediction data for the second block of video data comprise instructions that cause the one or more processors to encode the prediction data.

63. The computer-readable storage medium of claim 49, wherein the instructions that cause the one or more processors to code prediction data for the second block of video data comprise instructions that cause the one or more processors to decode the prediction data.

64. The computer-readable storage medium of claim 49, further comprising instructions that cause the one or more processors to code the second block of video data using the prediction data for the second block of video data.