BR112016030188B1 - COMPLIANCE AND INOPERABILITY ADVANCEMENTS IN MULTI-LAYER VIDEO ENCODING - Google Patents

Abstract

COMPLIANCE AND INOPERABILITY ADVANCEMENTS IN MULTI-LAYER VIDEO CODING. The present invention relates to devices and methods for processing video data. The method may include extracting an output operation point from received video data. If the output operation point matches one of an entire bitstream and a temporal subset of the entire bitstream with only the base layer as output, the method may include performing a first bitstream conformance test. If the output operating point matches one of a layer set defined by a base video parameter set (VPS) of an active VPS and a temporal subset of the layer set with only the base layer to be output, the method may include performing a second bitstream conformance test. If the output operating point matches one of an output layer set (OLS) defined by a VPS extension of the active VPS and a temporal subset of the OLS, the method may include performing a third bitstream conformance test.

Description

FUNDAMENTALS OF THE INVENTION TECHNOLOGICAL FIELD

[0001] This application refers to multilayer video encoding. More particularly, this application relates to methods for conformance and interoperability in multilayer video encoding, including signaling profile, category, and level information, signaling output layer sets, the use of hypothetical reference decoder (HRD) parameters, and bitstream conformance testing.

RELATED TECHNIQUE

[0002] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), desktop or portable computers, tablets, electronic book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cell phones or satellite radio, so-called "smartphones", video teleconferencing devices, video streaming devices and the like s. Digital video devices implement one or more video encoding techniques. Video devices can more efficiently transmit, receive, encode, decode and/or store digital video information by implementing such video encoding techniques.

[0003] Video encoding techniques include, without limitation, those described in the standards defined BY ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual and ITU-T H.264 or Advanced Video Coding (AVC) MPEG-4 ISO / IEC (including its Scalable Video Coding extensions (SVC) and Multiview Video Coding (MVC)), and the relatively new High Efficiency Video Coding (HEVC) standard. The HEVC standard was recently finalized by the Video Coding Joint Collaboration Team (JCT-VC) of the Video Coding Expert Group (VCEG) of the Telecommunications Standardization Sector of the International Telecommunication Union (ITU-T) and Moving Picture Expert Group (MPEG), formed by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). The most recent working draft (WD) of the HEVC specification, called HEVC WD10, is available at phenix.int- evry.fr/jct/doc_end_user/documents/12_Geneva/wgl/JCTVC-L1003-v34.zip. The multiview extension for HEVC, i.e. MV-HEVC, is also being developed by JCT-3V. A recent working draft of MV-HEVC WD3 below is available from phenix.it-sudparis.eu/jct2/doc_end_user/documents/3_Geneva/wg11/JCT3V-C1004-v4.zip. A scalable extension to HEVC, called SHVC, is also being developed by JCT-VC. A recent working draft of SHVC and referred to as SHVC WD2 below is available at phenix.int- evry.fr/jct/doc_end_user/documents/13_Incheon/wgl/JCTVC-M1008-vl.zip.

[0004] Video encoding techniques include spatial prediction (intra-picture) and/or temporal prediction (inter-picture) to reduce or remove inherent redundancy in video sequences. For block-based video coding, a slice of video (e.g. a video frame or a portion of a video frame) can be divided into video blocks, which may also be referred to as treeblocks, coding units (CUs) and/or coding nodes. Video blocks in an intracoded slice (I) of a picture are coded using spatial prediction relative to reference samples in neighboring blocks in the same picture. Video blocks in an intercoded slice (P or B) of a picture can use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. Images can be referred to as frames, and reference images can be referred to as reference frames.

[0005] The spatial or temporal prediction results in a predictive block for a block to be encoded. The residual data represents pixel differences between the original block to be encoded and the predictive block. An intercoded block is coded according to a motion vector that points to a block of reference samples that form the predictive block, and residual data that indicates the difference between the coded block and the predictive block. An intracoded block is encoded according to an intra encoding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain into a transform domain, resulting in residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be sorted to produce a one-dimensional vector of transform coefficients, and entropy coding can be applied to achieve even more compression.

[0006] A multiview encoding bitstream can be generated by encoding views, for example, from multiple perspectives. Some three-dimensional (3D) video standards have been developed that utilize multiview coding aspects. For example, different views can transmit left and right eye views to support 3D video. Alternatively, some 3D video encoding processes can apply multiview calling plus depth encoding. In multiview plus depth encoding, 3D video bitstream can contain not only texture view components but also depth view components. For example, each view can comprise a texture view component and a depth view component.

SUMMARY OF THE INVENTION

[0007] In general, this invention describes systems and methods related to advances in compliance and interoperability in multilayer video encoding. The systems, methods and devices of this invention can each have several innovative aspects, none of which are solely responsible for the desirable attributes described herein.

[0008] One aspect of the invention provides a method of processing video data. The method may include receiving encoded video data having a plurality of output operation points. The method may also include extracting a selected output operation point from the plurality of output operation points. In response to the selected output operation point corresponding to one of an entire bitstream and a temporal subset of the entire bitstream with only the base layer being output, the method may also include performing a first bitstream compliance test against the selected output operation point, the first bitstream compliance test being based on a set of hypothetical reference decoder (HRD) parameters at sequence level on a set of active sequence parameters (SPS) for a base layer, and one or more messages non-nested supplemental enhancement information (SEI). In response to the selected output operation point corresponding to one of a set of layers defined by a base video parameter set (VPS) of an active VPS and a temporal subset of the layer set with only the base layer to be output, the method may also include performing a second bitstream compliance test against the selected output operation point, the second bitstream compliance test being based on a set of sequence-level HRD parameters on the base VPS and directly nested SEI messages. In response to the selected output operating point corresponding to one of an output layer set (OLS) defined by a VPS extension of the active VPS and a temporal subset of the OLS, the method may also include performing a third bitstream conformance test against the selected output operating point, the third bitstream conformance test being based on a set of sequence-level HRD parameters on the active VPS and indirectly nested SEI messages.

[0009] Another aspect of the invention provides an apparatus for processing video data. The device may have memory configured to store encoded video data, the encoded video data having a plurality of output operating points. The device may also have a processor operatively coupled to the memory and configured to extract a selected output operation point from the plurality of output operation points. In response to the selected output operating point corresponding to one of an entire bitstream and a temporal subset of the entire bitstream with only the base layer to be output, the processor may be further configured to perform a first bitstream conformance test against the selected output operating point, the first bitstream conformance test being based on a set of hypothetical reference decoder (HRD) parameters at sequence level on a set of active sequence parameters (SPS) for a base layer, and one or more messages of the supplemental enhancement information (SEI) not nested. In response to the selected output operation point corresponding to one of a set of layers defined by a base video parameter set (VPS) of an active VPS and a temporal subset of the layer set with only the base layer to be output, the processor may be further configured to perform a second bitstream compliance test against the selected output operation point, the second bitstream compliance test being based on a set of sequence level HRD parameters in the base VPS and directly nested SEI messages. In response to the selected output operating point corresponding to one of an output layer set (OLS) defined by a VPS extension of the active VPS and a temporal subset of the OLS, the processor may be further configured to perform a third bitstream compliance test against the selected output operating point, the third bitstream compliance test being based on a set of sequence-level HRD parameters on the active VPS and indirectly nested SEI messages.

[0010] Another aspect of the invention provides an apparatus for processing video data. The apparatus may have memory means for storing encoded video data, the video data having a plurality of output operating points. The apparatus may also have decoding means operatively coupled to the memory device for extracting a selected output operation point from the plurality of output operation points. In response to the selected output operation point corresponding to one of an entire bitstream and a temporal subset of the entire bitstream with only the base layer to be output, the decoder means may also perform a first bitstream compliance test against the selected output operation point, the first bitstream compliance test being based on a set of hypothetical reference decoder (HRD) parameters at sequence level on a set of active sequence parameters (SPS) for a base layer, and one or more messages of the non-nested supplemental enhancement information (SEI). In response to the selected output operation point corresponding to one of a set of layers defined by a base video parameter set (VPS) of an active VPS and a temporal subset of the layer set with only the base layer to be output, the decoder means may also perform a second bitstream compliance test against the selected output operation point, the second bitstream compliance test being based on a set of sequence-level HRD parameters in the base VPS and directly nested SEI messages. In response to the selected output operating point corresponding to one of an output layer set (OLS) defined by a VPS extension of the active VPS and a temporal subset of the OLS, the decoder means may also perform a third bitstream compliance test, the third bitstream compliance test being based on a set of sequence-level HRD parameters in the active VPS and indirectly nested SEI messages.

[0011] Another aspect of the invention provides a non-transient computer readable means for processing video data. The non-transient computer-readable medium may contain instructions that, when executed by a processor, cause a computer to receive encoded video data having a plurality of output operating points. The instructions may also direct the computer to extract a selected output operation point from the plurality of output operation points. In response to the selected output operation point corresponding to one of an entire bitstream and a temporal subset of the entire bitstream with only the base layer being output, the instructions may also cause the computer to perform a first bitstream compliance test against the selected output operation point, the first bitstream compliance test being based on a set of hypothetical reference decoder (HRD) parameters at sequence level on a set of active sequence parameters (SPS) for a base layer, and one or more messages of the supplemental enhancement information (SEI) not nested. In response to the selected output operation point corresponding to one of a set of layers defined by a base video parameter set (VPS) of an active VPS and a temporal subset of the layer set with only the base layer to be output, the instructions may also cause the computer to perform a second bitstream compliance test against the selected output operation point, the second bitstream compliance test being based on a set of sequence-level HRD parameters on the base VPS and directly nested SEI messages. In response to the selected output operating point corresponding to one of an output layer set (OLS) defined by a VPS extension of the active VPS and a temporal subset of the OLS, the instructions may also cause the computer to perform a third bitstream compliance test against the selected output operating point, the third bitstream compliance test being based on a set of sequence-level HRD parameters on the active VPS and indirectly nested SEI messages.

[0012] Other features and advantages of the present invention should be apparent from the following description, which illustrates, by way of example, aspects of the invention.

BRIEF DESCRIPTION OF THE FIGURES

[0013] The details of embodiments of the present invention, both as to their structure and operation, can be gathered in part by studying the accompanying drawings, in which like reference numerals refer to like parts, and in which:

[0014] FIG. 1 is a functional block diagram of a video encoding system;

[0015] FIG. 2 is a functional block diagram of the video encoder of FIG. 1;

[0016] FIG. 3 is a block diagram illustrating the video decoder of FIG. 1; It is

[0017] FIG. 4 is a flowchart describing a method of implementing conformance testing.

DETAILED DESCRIPTION OF THE INVENTION

[0018] Generally speaking, this invention describes systems and methods relating to advances in bitstream conformance and interoperability in multilayer video coding. Some of the methods including flagging profile, category, and level information, flagging output layer sets, using HRD parameters, and bitstream conformance testing.

[0019] This invention identifies some shortcomings of the current HEVC specification, regarding the signaling of profile information, category and level, signaling of output layer sets, use of HRD parameters and bitstream compliance tests. Some of these deficiencies include the following:

[0020] With the exception of the 0th output layer set (OLS), the optimal category and level for other base-layer-output OLSs can only be signaled in the video parameter set (VPS) extension part. A base-layer-only-output OLS is an OLS for which there is only one output layer and the only output layer is the base layer. Such a signaling is not necessary because HEVC does not require that a base-layer-only output OLS be signaled in the VPS extension for each layer set signaled in the base VPS. The profile, category, and level information for the entire bitstream is instead signaled to the VPS base. Therefore, from a HEVC version 1 point of view, it is not optimal to perform bitstream conformance tests for all operating points with their own HRD parameters, while just checking the constraints against just the category and level of the entire bitstream. This can allow the HRD parameters for all operating points other than that corresponding to the entire bitstream to be much "higher" than ideal, while still satisfying conformance tests.

[0021] HEVC allows signaling of base-layer-egress OLSs only in the VPS extension. However, the base VPS already fulfills such a role. Therefore, this signaling in the VPS extension is unnecessary, which results in increased overhead.

[0022] The HEVC standard is unclear as to the applicability of unnested and directly nested buffering period (BP) enhancement supplemental information (SEI), picture time (PT) and decoding unit information (DUI) messages with respect to which of these SEI messages should be selected for use in some bitstream conformance tests.

[0023] The use of the profile, category, level (PTL) syntax structure is not clearly defined during bitstream conformance testing. For example, some bitstream conformance tests for base-only-output-layer OLSs are missing. The use of the hrd_parameters() syntax structure and SEI BT, PT and DUI messages is unclear for some compliance tests.

[0024] The following detailed description, in connection with the accompanying drawings, is intended as a description of various embodiments and is not intended to represent the only embodiments in which the invention can be practiced. The detailed description includes specific details for the purpose of providing a complete understanding of the embodiments. However, the invention will be apparent to those skilled in the art without these specific details. In some cases, well-known structures and components are shown in simplified form for brevity of description. VIDEO CODING SYSTEM

[0025] FIG. 1 is a functional block diagram of a video encoding system. A video encoding system ("system") 10 that may utilize methods in accordance with aspects described in the present invention. As described herein, the term "video encoder" refers generically to both video encoders and video decoders. In the present invention, the terms "video encoding" and "encoding" can refer generically to video encoding and video decoding.

[0026] As shown in FIG. 1, the system 10 includes a source device 12 that can generate encoded video data to be decoded at a later time by a destination device 14. The source device 12 and the destination device 14 can comprise any of a wide range of devices, including desktop computers, handheld computers (i.e., laptop), tablets, set-top boxes, telephone devices such as so-called "smartphones", so-called "smart pads", televisions, cameras, display devices, media players digital devices, video game consoles, video streaming device, or the like. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

[0027] The destination device 14 may receive the encoded video data to be decoded over a link 16. The link 16 may comprise any type of medium or device capable of moving the encoded video data from the source device 12 to the destination device 14. In one example, the link 16 may comprise a communication medium to allow the source device 12 to transmit encoded video data directly to the destination device 14 in real time. Source device 12 can modulate video data encoded according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium can comprise any wired or wireless communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from source device 12 to destination device 14.

[0028] Encoded data can be output from the output interface 22 to a storage device 34. Likewise, encrypted data can be accessed from the storage device 34 through the input interface 28 of the target device 14. The storage device 34 can include any of a variety of distributed or locally accessed data storage media, such as a hard disk, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory , or any other suitable digital storage media for storing encoded video data. In another example, storage device 34 may correspond to a file server or other intermediate storage device that may contain the encoded video generated by source device 12. Destination device 14 may access the stored video data from storage device 34 via streaming or download. The file server can be any type of server capable of storing encoded video data and transmitting that encoded video data to the target device 14. Exemplary file servers include a network server (e.g., for a website), an FTP server, attached network storage (NAS) devices, or a local disk drive. The target device 14 can access the encoded video data over any standard data connection, including Internet access. This may include a wireless channel (e.g. Wi-Fi connection), a wired connection (e.g. DSL, cable modem, etc.), or a combination of both, which is suitable for accessing encoded video data stored on a file server. The transmission of encoded video data from storage device 34 may be a live stream, a download stream, or a combination of both.

[0029] The methods of this invention are not necessarily limited to wireless applications or configurations. The methods can be applied to video encoding in support of any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television broadcasts, satellite television broadcasts, live video broadcasts, for example over the Internet, encoding digital video for storage on a data storage medium, decoding digital video stored on a data storage medium, or other applications. In some examples, system 10 can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

[0030] In the example of FIG. 1, the source device 12 includes a video source 18, video encoder 20 and an output interface 22. In some cases, the output interface 22 may include a modulator/demodulator (modem) and/or a transmitter. At source device 12, video source 18 may include a source such as a video capture device, e.g., a video camera, a video file containing previously captured video, a video feed interface for receiving video from a video content provider, and/or a computer graphics system for generating computer graphics data as the source video, or a combination of such sources. As an example, if video source 18 is a video camera, source device 12 and destination device 14 may form so-called camera phones or video phones. However, the methods described in the present invention can be applicable to video encoding in general, and can be applied to wireless and/or wired applications.

[0031] The video encoder 20 can encode captured, pre-captured or computer generated video. Furthermore, in some embodiments, the video decoder 30 can differentiate between a first bitstream and a second bitstream based on the parameter set ID of the first bitstream and the unique parameter set ID. In other examples, the output interface 22 of source device 12 may transmit the encoded video data directly to destination device 14 via link 16. The encoded video data may also (or alternatively) be stored on storage device 34 for later access by destination device 14 or other devices, for decoding and/or playback.

[0032] The target device 14 includes an input interface 28, a video decoder 30 and a display device 32. In some cases, the input interface 28 may include a receiver and/or a modem. The input interface 28 of destination device 14 receives the encoded video data over connection 16. The encoded video data communicated over connection 16, or provided on storage device 34, may include a variety of syntax elements generated by video encoder 20 for use by a video decoder, such as video decoder 30, in decoding video data. Such syntax elements may be included with encoded video data transmitted on a communication medium, stored on a storage medium, or stored on a file server.

[0033] Display device 32 may be integrated with, or external to target device 14. In some examples, target device 14 may include an integrated display device and also be configured to interface with an external display device. In other examples, target device 14 can be a display device. In general, the display device 32 shows the decoded video data to a user, and may comprise any of a variety of display devices, such as a liquid crystal display (LCD), plasma display, organic light emitting diode (OLED) display, or other type of display device.

[0034] The video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the methods are implemented partially in software, a device can store instructions for the software on a suitable non-transient computer-readable medium, and execute the instructions in hardware using one or more processors to execute the methods of this invention. Video encoder 20 and video decoder 30 can each be included in one or more encoders or decoders, each of which can be integrated as part of a combined encoder/decoder (codec) in a respective device.

[0035] HEVC standardization efforts are based on an evolution model of a video encoding device referred to as the HEVC Test Model (HM). The HM assumes several additional capabilities of video encoding devices over existing devices in accordance with, for example, ITU-T H.264/AVC. For example, while H.264 provides nine intra prediction encoding modes, HM can provide up to thirty-three intra prediction encoding modes.

[0036] In general, the HM working model describes that a video frame or image can be divided into a sequence of treeblocks or larger coding units (LCU) that include both luminance and chrominance samples. A treeblock has a similar effect to an H.264 macroblock. A slice includes a number of consecutive treeblocks in coding order. A video frame or image can be divided into one or more slices. Each treeblock can be divided into coding units (CUs) according to a quadtree. For example, a treeblock, such as a root node of the quadtree, can be split into four child nodes, and each child node can, in turn, be a parent node and be split into four more child nodes. An undivided, final child node, like a leaf node of the quadtree, comprises an encoding node, i.e., an encoded video block. The data syntax associated with an encoded bitstream can define a maximum number of times a treeblock can be split, and it can also define a minimum size of encoding nodes.

[0037] A CU includes a coding node and prediction units (PUs) and transform units (TUs) associated with the coding node. A CU size corresponds to a coding node size and must be square in shape. The CU size can range from 8x8 pixels up to the treeblock size with a maximum of 64x64 pixels or more. Each CU can contain one or more PUs and one or more TUs. Syntax data associated with a CU can describe, for example, the partitioning of the CU into one or more PUs. Partitioning modes can differ between whether the CU is ignored or coded in direct mode, coded in intraprediction mode, or coded in interprediction mode. PUs can be split to not be square in shape. Syntax data associated with a CU can also describe, for example, the partitioning of the CU into one or more TUs according to a quadtree. A TU can be square or not.

[0038] The HEVC standard allows transformations according to TUs, which may be different for different CUs. TUs can be sized based on the size of the PUs within a given CU defined for a partitioned LCU, although this may not always be the case. TUs can be the same size or smaller than PUs. In some examples, the residual samples corresponding to a CU can be subdivided into smaller units using a quadtree structure known as a “residual quadtree” (rqt). RQT leaf nodes can be referred to as transform units (TUs). The pixel difference values associated with the TUs can be transformed to produce transform coefficients, which can be quantized.

[0039] In general, a PU includes data related to the prediction process. For example, when the PU is encoded in intra mode, the PU may include data that describes an intra mode prediction for the PU. As another example, when the PU is coded in inter mode, the PU may include data that defines a motion vector for the PU. The data defining the motion vector for a PU may describe, for example, a horizontal component of the motion vector, a vertical component of the motion vector, a resolution for the motion vector (e.g., quarter-pixel precision or eighth-pixel precision), a reference image to which the motion vector points, and/or a reference image list (e.g., list 0, list 1, or list C) for the motion vector.

[0040] Generally speaking, a TU is used for the transform and quantization processes. A given CU with one or more PUs may also include one or more Transform Units (TUs). After prediction, the video encoder 20 can calculate residual values corresponding to the PU. The residual values comprise pixel difference values which can be transformed into transform coefficients, quantized and sorted using the TUs to produce serial transform coefficients for entropy encoding. As used herein, the term "video block" can generally refer to an encoding node of a CU. In some specific cases, this invention may also use the term "video block" to refer to a treeblock, i.e. LCU, or a CU, which includes a coding node and PUs and TUs.

[0041] The video encoder 20 can generate a bit stream that includes encoded video data. The bitstream can include a number of Network Abstraction Layer (NAL) units. Bitstream NAL units can include Video Coding Layer (VCL) NAL units and non-VCL NAL units. NAL VCL units can include coded slices of images. The non-VCL NAL unit may include a VPS, a sequence parameter set (SPS), a picture parameter set (PPS), SEI, or other types of data. A VPS is a syntax structure that can contain syntax elements that apply to zero or more entire encoded video sequences. An SPS is a syntax structure that can contain syntax elements that apply to zero or more entire encoded video sequences. A single VPS can be applicable to multiple SPSs. A PPS is a syntax structure that can contain syntax elements that apply to zero or more entire encoded images. A single SPS can be applicable to several PPS's. Various aspects of VPS, SPS and PPS can be formed in general as defined by the HEVC standard. The video encoder 20 can use SEI messages to include, in the bit stream, metadata that is not necessary for correct decoding of the image sample values. However, video decoder 30 or other devices may use metadata included in SEI messages for a variety of other purposes. For example, video decoder 30 can utilize the metadata in SEI messages for picture output timing, picture display, loss detection, and error concealment.

HIGH LEVEL SYNTAX

[0042] The NAL units may contain syntax elements encoded by the video encoder 20 and transmitted to the video decoder 30. Syntax elements that describe the structure of the bitstream or provide information that applies to multiple pictures or multiple encoded block regions within a picture, such as parameter sets (e.g., SPS, VPS, PPS, etc.), reference picture management syntax, and SEI messages, are known as part of the “high-level syntax” ( HLS) from HEVC.

[0043] In particular, HEVC employs the following aspects, relevant to this invention:

[0044] Parameter set structure: Parameter sets contain information that can be shared for decoding multiple regions of the decoded video. The parameter set structure provides a robust mechanism for transporting data that is essential for the decoding process.

[0045] Enhancement Supplemental Information (SEI): The syntax includes support for several types of metadata known as SEI. This data provides information about the timing of the video images, proper interpretation of the color space used in the video signal, stereoscopic 3-D frame packing information, other display cue information, and so on.

[0046] FIG. 2 is a functional block diagram of the video encoder of FIG. 1. Video encoder 20 may also include a partitioning module 35 configured to receive captured video data and partition the video data into video blocks. The partitioning module 35 can partition the video data into slices, tiles, or other larger units. The partitioning module 35 can also partition video blocks, for example according to a quadtree structure of LCUs and CUs. Video encoder 20 may divide the slice into multiple video blocks (and possibly into sets of video blocks referred to as tiles).

[0047] The video encoder 20 may also have a prediction module 41 operatively coupled to the partitioning module 35 and configured to receive the partitioned video data. The prediction module 41 can select one of a plurality of possible encoding modes, such as one of a plurality of intra encoding modes or one of a plurality of inter encoding modes, for the current video block based on error results (e.g., encoding rate and distortion level). The prediction module 41 can allow the video encoder 20 to perform intra and inter encoding of video blocks within video slices. Intra coding relies on spatial prediction to reduce or eliminate spatial redundancy in video within a given video frame or image. Inter coding relies on temporal prediction to reduce or eliminate temporal redundancy in video within adjacent frames or images of a video sequence. Intra mode (I mode) can refer to any of several spatially based compression modes. Inter modes, such as unidirectional prediction (P mode) or bi-prediction (B mode), can refer to any of several time-based compression modes.

[0048] The prediction module 41 may include a motion estimation module 42, a motion compensation module 44, and an intra prediction module 46. Each of these components may be operatively linked or integrated into one or more processors within the video encoder 20. Although discussed separately for conceptual purposes, the motion estimation module 42, motion compensation module 44, and the intra prediction module 46 may be highly integrated.

[0049] The motion estimation module 42 can be configured to determine the inter prediction mode for a video slice according to a predetermined pattern for a video sequence. Such interprediction or interpredictive encoding of the current video block against one or more predictive blocks in one or more reference pictures can provide temporal compression. The default pattern can designate video slices in the sequence as P-slices, B-slices or GPB slices. Motion estimation, as used herein, can generally refer to the process of generating motion vectors, which calculate motion for blocks of video. A motion vector, for example, can indicate the displacement of a PU of a video block within a current video frame or image relative to a predictive block within a reference frame or reference image. The reference frames may be stored in a reference picture memory 64 operatively linked to the prediction module 41. The reference frame memory 64 may further include or be referred to herein as a decoded picture buffer (DPB). Reference frame memory 64, or DPB, can store reconstructed code blocks after a filter module 63 performs unlock operations on the reconstructed code blocks.

[0050] A predictive block is a block deemed to closely match the PU of the video block to be encoded in terms of pixel difference, which can be determined by sum of absolute difference (SAD), sum of squared difference (SSD), or other difference metrics. In some examples, the motion prediction module ("prediction module") 41 may calculate values for subinteger pixel positions of reference images stored in reference image memory 64. For example, the prediction module 41 may interpolate the values of one-quarter pixel positions, one-eighth pixel positions, or other fractional pixel positions of the reference image. Therefore, the motion estimation module 42 can perform a motion search relative to integer pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

[0051] The motion estimation module 42 can calculate a motion vector for a PU of a video block in an intercoded slice by comparing the position of the PU to the position of a predictive block of a reference picture. The reference image may be selected from a first reference image list (List 0) or a second reference image list (List 1), each of which identifies one or more reference images stored in reference image memory 64. Motion estimation module 42 may send the calculated motion vector to motion compensation module 44 and an entropy encoding module 56.

[0052] Motion compensation may involve fetching or generating the predictive block based on motion vector determined by motion estimation. This can include interpolations for subpixel precision. Upon receiving the motion vector for the PU of the current video block, the motion compensation module 44 can locate the predictive block to which the motion vector points in one of the reference picture lists. Motion compensation module 44 may also generate syntax elements associated with the video blocks and the video slice for use by the video decoder 30 in decoding the video blocks of the video slice.

[0053] The intra prediction module 46 within the prediction module 41 can perform intrapredictive coding of the current video block relative to one or more neighboring blocks in the same frame or slice to provide spatial compression. The intra prediction module 46 can intrapredict a current block, as an alternative to the inter prediction performed by the motion estimation module 42 and the motion compensation module 44, as described above. In particular, the intra prediction module 46 can determine an intra prediction mode to be used for encoding a current block. In some examples, the intra prediction module 46 may code a current block using several intra prediction modes, for example, during separate coding passes. For example, the intra prediction module 46 can calculate the distortion rate values using a distortion rate analysis for the various intra prediction modes tested, and select the intra prediction mode with the best distortion rate characteristics among the tested modes. Distortion rate analysis can determine an amount of distortion (or error) between an encoded block and an original, unencoded block that was encoded to produce the encoded block, as well as a bit rate (that is, a number of bits) used to produce the encoded block. The intra prediction module 46 can calculate the distortion ratios and rates for the various coded blocks to determine which intra prediction mode has the best distortion rate value for the block.

[0054] The intra prediction module 46 can provide information indicative of the selected intra prediction mode for the block to the entropy encoding module 56. The entropy encoding module 56 can encode information indicating the selected intra prediction mode. The video encoder 20 may also include configuration data in the transmitted bitstream, which may include a plurality of intra prediction mode index tables and a plurality of modified intra prediction mode index tables (also referred to as codeword mapping tables), definitions of coding contexts for various blocks, and indications of a most likely intra prediction mode, an intra prediction mode index table and a modified intra prediction mode index table for use in each of the contexts.

[0055] After the prediction module 41 generates the predictive block for the current video block, either through inter prediction or intra prediction, the video encoder 20 forms a residual video block by subtracting the predictive block from the current video block. This process can result in pixel difference values. The pixel difference values form the residual data for the block, and can include both luminance and chrominance difference components. An adder 50, operatively coupled to the prediction module 41 and the partitioning module 35, represents the component or components that can be configured to perform this subtraction operation.

[0056] The residual video data in the residual block may be included in one or more TUs and applied to a transform module 52 operatively coupled to the adder 50. The transform module 52 may transform the residual video data into residual transform coefficients using a transform such as a discrete cosine transform (DCT) or conceptually similar transform. Transform module 52 can convert residual video data from a pixel domain to a transform domain, such as a frequency domain. The prediction module 41 can provide the resulting intra or intercoded block to the adder 50 to generate residual block data. The resulting intra or intercoded block may be further provided to an adder 62 to reconstruct the coded block for use as a reference picture.

[0057] Video encoder 20 may also include filter module 63 operatively coupled to adder 62. Filter module 63 may represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF) and an adaptive sample shift (SAO) filter. Although filter module 63 is shown in FIG. 2 as being a loop filter, in other configurations, filter module 63 can be implemented as a post-loop filter. Filter module 63 can provide reference images to memory image reference 64.

[0058] The transform module 52 can send the resulting transform coefficients to a quantization module 54. The quantization module 54 can quantize the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting a quantization parameter. In some examples, the quantization module 54 may then perform a matrix sort including the quantized transform coefficients. In some embodiments, entropy encoding module 56 can perform sorting.

[0059] After quantization, the entropy coding module 56 can encode quantized transform coefficients by entropy. For example, entropy coding module 56 can perform context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), Probability Interval Partitioning Entropy coding (PIPE), or other entropy coding methodology. Video encoder 20 may also entropy encode syntax elements associated with the encoded video data for use by video decoder 30 in decoding the video data.

[0060] After entropy encoding by the entropy encoding module 56, the encoded bit stream can be transmitted to the video decoder 30. The bit stream can also be archived for further transmission or retrieval by the video decoder 30. The entropy encoding module 56 can also entropy encode the motion vectors and other syntax elements for the current video slice being encoded.

[0061] The video encoder 20 may also include an inverse quantization module 58 operatively coupled to the entropy encoding module 56. An inverse transform module 60 may also be operatively coupled to the inverse quantization module 58 and the adder 62. later as a reference block of the reference image. The motion compensation module 44 can compute a reference block by adding the residual block to a predictive block of one of the reference pictures within one of the reference picture lists. The adder 62 can add the reconstructed residual block to the motion compensated prediction block produced by the motion compensation module 44 to produce a reference block for storage in the reference image memory 64. The reference block can be used by the motion estimation module 42 and the motion compensation module 44 in the form of a reference block to interpret a block in a subsequent video frame or image.

[0062] In some examples, the video encoder 20 generates a bit stream. The bitstream generated by the video encoder 20 can include various operating points that can be extracted as sub-bitstreams of the bitstream. Operation points can include, for example, multiple layers and/or views, as well as multiple frame rates. The video encoder 20 can encode information indicative of the output operating points in the VPS. In some examples, for each of the operation points that the video encoder 20 signals on the VPS associated with the bitstream, an operation point syntax structure specifies a set of layer identifiers (IDs) used to identify the NAL units in the bitstream that belong to a substream of bits of a given operation point. In this way, the NAL units constituting the sub-bitstream of the given operating point can be extracted from the original bitstream on the basis of the layer identifiers of the NAL units.

[0063] In some cases, the video encoder 20 may further encode HRD parameters that are related to one or more of the operating points, as discussed in greater detail below. In this case, the video encoder 20 signals HRD np VPS parameter information. Video encoder 20 may contain an implementation of an HRD to ensure that each generated bitstream is a conforming bitstream. The HRD implementation may also be part of other entities, such as decoders (e.g. video decoder 30) and bitstream analyzers to verify that a bitstream is a conforming bitstream. As discussed in more detail below, for each of the one or more operating points that have HRD parameters, an HRD parameter syntax structure can specify a set of HRD parameters that define an HRD used to check the compliance of a substream of bits of a given operating point.

[0064] As used here, an operation point is a bitstream created from another bitstream by operating the substream bitstream extraction process with another bitstream, a higher target temporal_ID, and a list of target layer identifiers as inputs. More specifically, “operation points” are substreams of bits that can be extracted from an original bitstream that is temporally scalable and/or multilayered and/or viewed. That is, an operation point is a bitstream that is created from another bitstream by operating the substream bitstream extraction process with the other bitstream, a target highest temporal_ID, and a list of target layer identifiers as inputs. The bitstream includes one or more views, including the point-of-operation view(s). The highest target temporal_ID defines the temporal subset. The target layer list includes a set of layers: the set of layers to be decoded for an operation point, and the set of layers to be emitted for an output operation point. An output operation point is associated with, inter alia, a set of output layers, i.e. the layers identified in the set of output layers, which are the layers that are to be output. In general, each bitstream can contain one or more sets of output layers. Each set of output layers can contain one or more output operation points. A bitstream conformance test is performed for each output operation point to ensure that the corresponding bitstream is a conforming bitstream.

[0065] The term “layer set” is used to refer to a set of layers to be decoded. A “layer set” is similar to an operation point, except that a layer set is not associated with a particular temporal_ID value. Instead, all temporal sublayers present in the layers belong to the layer set. Likewise, a set of layers can be said to be associated with the highest possible temporal_ID. Therefore, all temporal sublayers present in the layers belong to the layer set. A layer set can be derived from a bit stream, just as an operating point can be derived from a bit stream, with the highest temporal_ID being the highest possible value.

[0066] The term “output layer set” refers to a set of layers for which the output layers are defined. Thus, although the set of layers to be decoded is defined as a set of layers, the set of layers to be output is defined for a set of output layers. In some examples, the output layer set identifies the corresponding layer set including an index to the corresponding layer set, where the corresponding layer set identifies the layers to be decoded. In other examples, the output layer set identifies the otherwise defined corresponding layer set.

[0067] Layers in the layer set not targeted for emission may still be decoded, for example, when used for interlayer (or interview) prediction. The layers that are targeted for emission are a subset of the layers that are to be decoded: one, some, or all of the layers to be decoded can be targeted for emission. It should be understood that the term "subset" does not necessarily mean that the layer to be produced is a strict subset of the layers to be decoded. That is, in some examples, the layers to be decoded and the layer to be emitted are the same. Thus, the distinction between an output layer set and an output operation point is similar to the distinction between a layer set and an operation point, as described above.

[0068] Bit substreams can be extracted from the bitstream based on values of layer identifiers (identifying the layers for the output operation point) and temporal sublayer identifiers (which identify the temporal subset for the output operation point), which identify the output operation point of the bitstream. Output operating points can be signaled on a VPS within the bitstream. For each of the operating points, an operating point syntax structure specifies a set of layer identifiers used to identify the Network Abstraction Layer (NAL) units in the bitstream that belong to a substream of bits of a given operating point. In this way, the NAL units constituting the sub-bitstream of a given output operation point can be extracted from the original bitstream based on the layer identifiers of the NAL units and the highest tempora_ID associated with the output operation point. NAL units are the components of encoded video data; encoded video data is organized into NAL units.

[0069] FIG. 3 is a block diagram illustrating the video decoder of FIG. 1. For purposes of explanation, this invention describes video decoder 30 in the context of HEVC encoding. However, this description may be applicable to other coding standards or methods.

[0070] The video decoder 30 may include an entropy decoding module 70, a prediction module 71, an inverse quantization module 76, an inverse transform module 78, an adder 80, a filter module 84 and a decoded picture buffer (DPB) 82. The prediction module 71 may include a motion compensation module 72 and an intraprediction module 74. In other examples, video decoder 30 may include more, less or different functional components.

[0071] A coded picture buffer (CPB) 90 can receive and store coded video data (eg, NAL units) from a bit stream. The entropy decoding module 70 can receive NAL units from CPB 90 and parse the NAL units to decode syntax elements. The entropy decoding module 70 can entropy decode entropy encoded syntax elements into NAL units. Prediction module 71, inverse quantization module 76, inverse transform module 78, adder 80 and filter module 84 can generate decoded video data based on syntax elements extracted from the bit stream.

[0072] Bitstream NAL units may include slice coded NAL units. As part of decoding the bitstream, the entropy decoding module 70 can extract and entropy decode syntax elements from the encoded slice NAL units. Each of the encoded slices may include a slice header and slice data. The slice header can contain syntax elements that belong to a slice. Syntax elements in the slice header can include a syntax element that identifies a PPS associated with an image containing the slice.

[0073] In addition to decoding syntax elements from the bitstream, the video decoder 30 can perform a rebuild operation on a non-partitioned CU. To perform the rebuild operation on a non-partitioned CU, the video decoder 30 can perform a rebuild operation on each TU of the CU. By performing the rebuild operation for each TU of the CU, the video decoder 30 can reconstruct residual blocks of the CU.

[0074] As part of performing a rebuild operation on a TU of a CU, the inverse quantization module 76 may inverse quantize, that is, dequantize, blocks of coefficients associated with the TU. The inverse quantization module 76 can use a quantization parameter (QP) value associated with the TU CU to determine a degree of quantization and likewise a degree of inverse quantization for the inverse quantization module 76 to apply. That is, the compression ratio, that is, the ratio between the number of bits used to represent the original and the compressed sequence, can be controlled by adjusting the QP value used when quantizing the transform coefficients. The compression ratio may also depend on the entropy encoding method used.

[0075] After the inverse quantization module 76 inversely quantizes a block of coefficients, the inverse transform module 78 can apply one or more inverse transforms to the coefficient block in order to generate a residual block associated with the TU. For example, the inverse transform module 78 can apply an inverse DCT, an inverse integer transform, an inverse directional transform, or another inverse transform to the coefficient block.

[0076] If a PU is coded using intra prediction, the intra prediction module 74 can perform intra prediction to generate predictive blocks for the PU. The intra prediction module 74 can use an intra prediction mode to generate the predictive Luminance, Cb and Cr blocks for the PU based on the prediction blocks of spatially neighboring PUs. The intra prediction module 74 can determine the intra prediction mode for the PU based on one or more syntax elements decoded from the bit stream.

[0077] The prediction module 71 can build a first reference picture list (RefPicListO) and a second reference picture list (RefPicList1) based on syntax elements extracted from the bitstream. Furthermore, if a PU is encoded using inter prediction, the entropy decoding module 70 can extract motion information for the PU. Motion compensation module 72 can determine, based on motion information from the PU, one or more reference regions for the PU. Motion compensation module 72 can generate predictive luminance, Cb and Cr blocks for the PU based on blocks of samples in the one or more reference blocks for the PU.

[0078] The adder 80 can use the luminance, Cb and Cr transform blocks associated with TUs of a CU and the luminance, Cb and Cr predictive blocks of the CU PUs, i.e. either intra prediction data or inter prediction data, as applicable, to reconstruct the luminance, Cb and Cr coding blocks of the CU. For example, adder 80 can add samples from the luminance transform, Cb and Cr blocks to corresponding samples from the predictive luminance, Cb and Cr blocks to reconstruct the luminance, Cb and Cr coding blocks of the CU. Filter module 84 can be an unblocking filter, for example, and perform an unblocking operation to reduce blocking artifacts associated with the luminance, Cb, and Cr encoding blocks of the CU. Video decoder 30 can then store the luminance, Cb, and Cr coding blocks from the CU in decoded picture buffer 82. Decoded picture buffer 82 can provide reference pictures for later motion compensation, intra prediction, and presentation on a display device, such as display device 32 of FIG. 1. For example, the video decoder 30 can perform intra prediction or inter prediction operations on PUs of other CUs based on the luminance, Cb and Cr blocks in the decoded picture buffer 82. Thus, the video decoder 30 can decode transform coefficient levels of the significant luminance coefficient block from the bit stream, inverse quantize the transform coefficient levels, apply a transform to the transform coefficient levels for generating a transform block, generating, based at least in part on the transform block, a coding block, and outputting the coding block for display.

BIT SUBFLOW EXTRACTION

[0079] Video coding standards may include various features to support the bitstream substream extraction process. For example, bitstream video data can be divided into a set of layers. For each of the layers, data in a lower layer can be decoded without reference to data in any higher layer. An individual NAL unit only encapsulates the data of a single layer. Thus, NAL units that encapsulate data from the upper remaining layer of the bitstream can be removed from the bitstream without affecting the decoding capability of data in the lower remaining layers of the bitstream. In SVC, SHVC, and other scalable video encoding methods, higher layers can include enhancement data that improves the quality of images in lower layers (quality scaling), enlarges the spatial format of images in lower layers (spatial scaling), or increases the temporal frame rate in lower layers (temporal scaling). In MVC, MV-HEVC, and other three-dimensional video encoding, higher layers can include additional views.

[0080] NAL units may specify a temporal identifier of the NAL unit. If the temporal identifier of a first NAL unit is smaller than the temporal identifier of a second NAL unit, the data encapsulated by the first NAL unit can be decoded without reference to the data encapsulated by the second NAL unit.

[0081] Each point of operation of a data stream is associated with a set of layer identifiers and a temporal identifier. The set of layer identifiers can be denoted as OpLayerIdSet and the temporal identifier can be represented as TemporalId. If the layer identifier of a NAL unit is in an operation point set of layer identifiers and the temporal identifier of the NAL unit is less than or equal to the temporal identifier of the operating point, the NAL unit is associated with the operating point. A bitstream substream is a bitstream subset that is associated with an operating point. The bitstream can include each NAL unit that is associated with the operating point. The bitstream does not include NAL VCL units that are not associated with the operating point.

HYPOTHETICAL REFERENCE DECODER

[0082] Video encoding standards specify video buffering models also referred to as HRD. The HRD describes how data must be buffered for decoding and how decoded data is buffered for emission. For example, DRH describes the operation of a CPB and a DPB of the video coding system 10. The CPB and DPB may comprise the CPB 90 (FIG. 3) and the reference picture memory 64 (FIG. 2), respectively. In some embodiments, the DPB can also be similar to the DPB 82 (FIG. 3). CPB can be first-in-first-out buffer containing access units in decoding order defined by HRD. The DPB is a buffer that holds decoded pictures for reference, output reordering, or output delay defined by the HRD.

[0083] The video encoder 20 can signal a set of HRD parameters at the sequence level. The sequence-level HRD parameters control various aspects of the HRD. Stream-level HRD parameters include an initial CPB pruning delay, a CPB size, a bitrate, an initial DPB output delay, and a DPB size. These string-level HRD parameters are encoded in a hrd_parameters() syntax structure defined in a VPS and/or an SPS. Picture-level or sub-picture level HRD parameters can be set in a SEI BP, PT, or DUI message.

[0084] A sub-bitstream (from eg an operation point) may have a different frame rate and/or bitrate than its original bitrate. This could be because the sub-bitstream might not include some images and/or some of the data in the original bitstream. Thus, if the video decoder 20 had to remove data from the CPB and/or DPB at a particular rate when processing the original bitstream, and if the video decoder 30 had to remove data from the CPB and/or DPB at the same rate when processing a sub-bitstream, the video decoder 30 can remove too much or too little data from the CPB and/or DPB. In this way, video encoder 20 can signal different sets of HRD parameters for different operating points. In HEVC, video encoder 20 can signal HRD parameter sets on a VPS or video encoder 20 can signal HRD parameter sets on an SPS.

[0085] In some versions of HEVC, only the HRD parameter sets in the VPS are selected for HRD operations. That is, although HRD parameters can be provided in SPSs, sets of HRD parameters in SPSs are not selected by HEVC video decoders for HRD operations. The video decoder 30 can parse and decode the VPS of a bit stream. Therefore, the video decoder 30 can parse and decode the VPS HRD parameter sets. This can occur regardless of whether the bit stream includes non-base layer NAL units.

[0086] The video encoder 20 can generate a bit stream that includes an SPS, which is applicable to a sequence of images. In some embodiments, the SPS includes a set of HRD parameters. The HRD parameter set is applicable to each data stream operation point that has a set of layer identifiers that correspond to a set of target layer identifiers. Thus, HRD parameter sets in SPSs are not wasted but can be used for HRD operations. For example, a device can select, from a set of HRD parameters in a VPS and a set of HRD parameters in an SPS, a set of HRD parameters applicable to a particular operating point. The video encoder 20 can perform, based at least in part on the set of HRD parameters applicable to the particular operation point, a bitstream conformance test that tests whether a subset of the bitstream associated with the particular operation point conforms to a video encoding standard.

[0087] The video encoder 20, the video decoder 30 or another type of device, such as a content delivery network (CDN) device, a Media-Aware Network Element (MANE), for example, can perform bitstream compliance tests on a substream of bits for an operating point. Bitstream conformance tests can verify that the bitstream conforms to a video encoding standard such as HEVC.

[0088] In H.264/AVC and HEVC, bitstream compliance and decoder compliance are defined as parts of the HRD specification. The HRD model specifies tests to determine whether a bit stream conforms to a standard and tests to determine whether a decoder conforms to the standard. Although HRD is referred to as a type of decoder, video encoder 20 may use HRD to ensure bitstream compliance, while video decoder 30 may in fact not need HRD.

[0089] Two types of bitstream subsets are subject to HRD compliance, namely Type I and Type II. A Type I bitstream is a NAL unit bitstream containing only the NAL units VCL and NAL units of padding data for all access units in the bitstream. A Type II bitstream is a bitstream of NAL units that contains, in addition to the NAL units of VCL and NAL padding data units for all access units in the bitstream, at least one of the following: additional non-VCL NAL units other than NAL padding data units; and all leading_zero_8bits, zero_byte, start_coded_prefix_one_3, and trailing_zero_8bits syntax elements that form a byte stream of the NAL unit stream.

[0090] When a device (e.g., video encoder 20) performs a bitstream conformance test that determines whether a bitstream conforms to a video encoding standard, the device may select an operating point from the bitstream. The device can then determine a set of HRD parameters applicable to the selected operating point. The device can use the set of HRD parameters applicable to the selected duty point to configure the behavior of the HRD. More particularly, the device can use the applicable set of HRD parameters to configure the behaviors of particular components of the HRD, such as a hypothetical stream scheduler (HSS), the CPB, a decoding process, the DPB, and so on.

[0091] Additionally, the device can invoke a decoding process that decodes the video data encoded in the CPB 90. The decoding process can output decoded images to the DPB 82. As the device moves data through the HRD, the device can determine whether a given set of constraints is still being met. For example, the device can determine whether a negative flow or positive flow condition occurs in the CPB 90 or DPB 82 while the HRD is decoding the negative bit stream of the selected set point. The device can select and process each operating point of the bitstream in this way. If no operating point of the bitstream causes the limitations to be violated, the device can determine that the bitstream complies with the video encoding standard.

[0092] Both H.264/AVC and HEVC specify two types of decoder conformance, namely output-time decoder conformance and output-order decoder conformance. A decoder (e.g., video decoder 30) conforming to a specific Profile, Category, and Level (PTL) is capable of successfully decoding all bitstreams that conform to the bitstream compliance requirements of a video encoding standard, such as HEVC. As used herein, a "profile" can generally refer to a subset of the bitstream syntax. “Categories” and “Levels” can be defined within each profile. A level of a category can be a specific set of constraints imposed on the values of syntax elements in the bitstream. These constraints can be simple limits for various values. Alternatively, they can take the form of limitations on arithmetic combinations of values (eg, image width multiplied by image height multiplied by the number of decoded images per second). A specific level for a lower category is more limited than a specific level for a higher category.

[0093] The HEVC standard stipulates two sets of bitstream conformance tests based on the content of the video data. The first set of bitstream conformance tests is to test the conformance of the entire bitstream and its temporal subsets, regardless of the existence of a layer set defined by the active VPS that contains all nuh_layer_id values of NAL VCL units present in the entire bitstream. The second set of bitstream conformance tests is for conformance testing of the sets of tiers defined by the active VPS and its temporal subsets. For all these tests, only the base layer images (that is, images with nuh_layer_id equal to 0) are decoded and the other images are ignored by the decoder when the decoding process is invoked.

TARGETED LAYER SETS IN THE BASE VPS

[0094] In some examples, the video encoder 20 may signal one or more sets of VPS layers. A VPS, as defined by the HEVC standard, is a syntax structure containing syntax elements that apply to zero or more entire encoded video sequences as determined by the contents of a video_parameter_set_id syntax element found in the SPS referenced by the seq_parameter_set_id syntax element, which is found in the PPS referenced by the pic_parameter_set_id syntax element found in each slice segment header.

[0095] Since VPSs apply to entire encoded video sequences, VPS includes syntax elements that change infrequently. The mechanism of VPS, SPS and PPS in some versions of HEVC decouples the transmission of rare change information from the transmission of encoded video block data. In HEVC, an identifier (ID) of a VPS, an SPS or a PPS can be encoded for each set of parameters. Each SPS includes an SPS ID and a VPS ID, each PPS includes a PPS ID and an SPS ID, and each slice header includes a PPS ID and possibly an APS ID. Therefore, ID's can be used to identify the set of suitable parameters to be used in different cases.

[0096] A VPS contains two parts, the base VPS and the VPS extension. The base VPS, as defined in the first edition, contains information related to the HEVC version 1 compliant layer, as well as operating points corresponding to sets of layers. The base VPS also contains temporal scaling information, including the maximum number of temporal tiers. The VPS extension contains information related to additional layers beyond the base layer.

[0097] In MV-HEVC WD8 and SHVC WD6, all layer sets flagged in the base VPS are equivalent to output layer sets (OLSs) for which the only output layer is the base layer. These are often referred to here as “base-only-layer-output OLSs”. In general, the 0th OLS is one of the output-layer-only OLSs, however, there may be other OLSs that are also the output-base-layer-only OLSs. This is because a video decoder 30 conforming to profiles defined in Annex A of the HEVC specification and its corresponding HRDs only decode the base layer, ignoring the enhancement layers, even though all enhancement layer bits are counted in CPB operations when the OLS is decoded.

[0098] In some examples, signaling of egress-base-layer-only OLSs in the VPS extension is also allowed. However, since this is already done in the base VPS, this signaling in the VPS extension can represent a waste of bits. Therefore, in some embodiments, the HEVC standard may not allow signaling of the base-layer-only-egress OLSs in the VPS span.

[0099] Additionally, regardless of the fact that the entire bitstream corresponds to a signaled layer set in the base VPS, it is also equivalent to a base-layer-only-output OLS. For convenience, an OLS for which there is at least one output layer that is not the base layer is generally referred to here as a “non-base-layer-only-output OLS”.

PROFILE, CATEGORY, LEVEL

[0100] PTL information specifies conformance points for implementing the standard (eg HEVC) interoperably across multiple applications that have similar functional requirements. A profile defines a set of encoding tools or algorithms that can be used in generating a conforming bitstream, whereas a level imposes restrictions on some key parameters of the bitstream, corresponding to the decoder's processing load and memory capacities. Level restrictions are established in terms of the maximum sample rate, the maximum image size, maximum bitrate, minimum compression ratio and capabilities of the DPB, and the encoded picture buffer (CPB) that contains compressed data before its decoding for data stream management purposes. In designing HEVC, it was determined that there were some applications that had requirements that differed only in terms of maximum bit rate and CPB capabilities.

[0101] The profile is the same for all base-layer-only output OLSs, including the one corresponding to the entire bitstream. This is true even when the base layer is provided externally, in which case the profile is not defined. For example, in such a case, the PTL syntax structure in the base VPS only contains all zero bits.

[0102] When the base layer is in the bit stream, the profile for all output-base-layer-only OLSs is signaled in the PTL syntax structure of the active SPS for the base layer and in the first PTL syntax structure of the active VPS, i.e. the PTL syntax structure in the base part of the active VPS.

[0103] The category and level flagged in the active SPS PTL syntax structure for the base layer and in the first active VPS PTL syntax structure are for the entire bit stream, which may or may not correspond to a layer set or a signaled OLS in the VPS. However, as noted above, the entire bitstream can correspond to a base-layer-output-only OLS.

[0104] In a similar example, the base VPS does not contain the category and level information for other flagged-only-base-layer-output OLSs, including the 0° OLS which has only its own base layer in the layer set. Consequently, the category and level of the entire bitstream would also apply to any of the base-only-only-output OLSs other than the one corresponding to the entire bitstream, albeit potentially at a suboptimal rate. This could be because category and level information is identified for the entire bitstream. Thus, for a subset of the entire bitstream, the category and profile information may be suboptimal. For example, the optimal level for a subset of the entire bitstream might be less than that of the entire bitstream.

[0105] Except for the 0° OLS, the optimal category and level information for other base-layer-output OLSs can only be signaled in the VPS extension, but this is not required. This may be because there is no requirement to have a signaled-only base-tier-egress OLS in the VPS extension for each set of layers signaled in the base VPS.

[0106] From an HEVC version 1 point of view, testing a bitstream for conformance using its own HRD parameter and just the category and level information for the entire bitstream is not ideal. This can allow the HRD parameters for all operating points other than the one corresponding to the entire bitstream to be sub-optimal, while still passing the conformance tests. Thus, certain changes to compliance testing should be considered. This is described below in connection with FIG. 4.

HRD PARAMETERS IN SYNTAX STRUCTURES hrd_parameters()

[0107] HEVC specifies video buffering models to deal with fluctuations in bit rate, and variation of compressed video during transmission at constant or near constant bit rate. In this way, the video encoder 20 and the video decoder 30 can buffer the data in the CPB or the DPB, respectively, using HRD. The HRD describes how data must be buffered for decoding and how decoded data is buffered for emission.

[0108] The HRD can be used to verify bitstream and decoder compliance at various operating points. As noted above, an operating point defines the parameters used for bitstream extraction, which include a list of target layers and a target highest temporal layer. Various operating points can be specific to a particular bit stream. Operation points for which output layers are specific are generally referred to as output operation points herein.

[0109] In video coding standards, a compliant bitstream must be decodable by an HRD connected to the output of the video encoder 20 and consists of at least a pre-decoder buffer, a decoder and an output/display unit (e.g. display device 32).

[0110] A bit stream is compatible if it is decodable by HRD with no positive stream or negative stream buffered. Positive buffer flow occurs when more bits must be placed in a buffer when it is full. Buffer underflow occurs when some bits are not in the buffer when bits must be extracted from the buffer for decoding/playback.

[0111] The video encoder 20 may signal a set of HRD parameters to impose restrictions on the encoded sizes of pictures and to help decide the required buffer sizes and start-up delay.

[0112] For an HRD, two types of HRD parameter sets (HRD NAL parameters and HRD VCL parameters) can be used. In some examples, sequence-level HRD parameter sets are flagged through the hrd_parameters() syntax structure, which can be part of the SPS syntax structure or the VPS syntax structure.

[0113] Within the hrd_parameters() syntax structure, only three syntax elements are related to DPB operations. The other elements are for CPB operations only. These three DPB-related syntax elements are: fixed_pic_rate_general_flag[i], fixed_pic_rate_within_cvs_flag[i], and elemental_duration_in_tc_minus1[i], one set to each TemporalId value.

[0114] When images are aligned with layer crossing, that is, each access unit (AU) has an image for each layer of the layer set, and fixed image rate is used, a hrd_parameters() syntax structure that applies to a layer set would apply to all OLSs that are associated with the layer set. However, in other situations, a hrd_parameters() syntax structure that applies to a layer set may not be applicable to all other OLSs associated with the layer set than the base-layer-output-only OLS.

[0115] In other words, it is necessary to allow the signaling of hrd_parameters() syntax structures in the VPS extension to specific OLSs in the VPS extension.

APPLICABILITY OF NON-NESTED AND DIRECTLY NESTED SEI BP / PT / DUI MESSAGES

[0116] SEI messages may contain information about the timing of video images, the proper interpretation of the color space used in the video signal, stereoscopic 3-D frame packing information, other display hint information, and so on. SEI BP, PT, and DUI messages contain HRD parameters at image level and sub-image level. Non-nested SEI BP, PT, and DUI messages can apply to the entire bitstream, regardless of whether it corresponds to a layer set or signaled OLS on the VPS.

[0117] Directly nested SEI BP, PT, and DUI messages refer to SEI BP, PT, and DUI messages that are directly contained within scalable nesting SEI messages, rather than being indirectly nested. Indirectly nested refers to SEI messages contained in bitstream partition nesting SEI messages which are in turn contained in scalable nesting SEI messages. A non-nested SEI message is directly included in a NAL SEI unit, rather than being directly or indirectly nested.

[0118] As per HEVC version 1, directly nested SEI BP, PT, and DUI messages apply to operating points that correspond to signaled layer sets in the base VPS, which are equivalent to base-layer-only-output OLSs. In some examples, directly nested SEI BP, PT, and DUI messages do not apply to non-base-only-layer-exit OLSs, as there is no way to associate these SEI messages with a non-base-layer-exit OLS just by directly nesting them.

[0119] In some examples, however, the applicability of non-nested and directly nested SEI BP, PT and DUI messages is unclear based on the current standard. Therefore, it is not always clear which of the SEI messages should be selected for use in some bitstream conformance tests.

[0120] In some embodiments, it may be beneficial to specify which SEI messages are pertinent to which conformance test. Thus, directly nested SEI messages can be selected for use with operation points corresponding to base-layer-output-only OLSs. In some embodiments, only directly nested SEI messages can be selected for use with operation points corresponding to base-layer-output-only OLSs. This can eliminate some ambiguity. In some other embodiments, the SEI messages may be one of SEI BP, PT and DUI messages.

[0121] In some embodiments, indirectly nested SEI messages may be applied to specific non-base-only-output-layer OLSs in the VPS extension. In some other embodiments, the indirectly nested SEI messages may be one of SEI BP, PT, and DUI messages.

BITS STREAM COMPLIANCE TESTS

[0122] In some examples, except for the 0° OLS, optimal category and level information for output-base-layer-only OLSs can be signaled in the VPS extension. In HEVC, this may not be necessary. This is because the video encoder 20 does not need to signal the base-layer-only-output OLS in the VPS extension for each set of layers signaled in the base VPS. In base VPS, only one set of PTL information is signaled, but multiple hrd_parameters() structures can be signaled, for example, one for each base-layer-output-only-OLS. Thus, conformance of the bitstream to conformance points of a base-layer-output OLS is only tested against its own HRD parameters. However, the constraints are checked against just the category and level information of the entire bitstream. This is the only PTL information signaled in the base VPS, which is shared by all base-only-defined-only-defined-layer-egress OLSs by the base VPS. Existing requirements may then allow the HRD parameters for all other operating points other than that corresponding to the entire bitstream to be “higher” than ideal, while still passing the conformance tests. In some examples, the category and level information provides an upper bound for many of the HRD parameters. Therefore, the category or level of the entire bitstream may be greater than the ideal category or level for a base-layer-only-output OLS. Thus, it can be beneficial to implement bitstream conformance tests for different output-base-layer-only OLSs according to their respective content. This can be achieved by signaling different category and level information to the base-only-layer-output OLSs.

[0123] In some embodiments, there may be three sets of bitstream conformance tests implemented. The first set of bitstream conformance tests is to test the conformance of the entire bitstream and its temporal subsets, regardless of the existence of a layer set defined by the active VPS that contains all nuh_layer_id values of NAL VCL units present in the entire bitstream. The second set of bitstream conformance tests is for conformance testing of the tier sets defined by the base part of the active VPS (eg base VPS) and its temporal subsets. For all these tests, only base layer images (that is, images with nuh_layer_id equal to 0) are decoded and other images are ignored by the decoder when the decoding process is invoked. The third set of bitstream conformance tests can be implemented to test the conformance of the OLSs defined by the VPS extension of the active VPS and the temporal subsets of those OLSs.

[0124] In some embodiments, the three compliance tests are performed at one or more output operating points. Output operating points can be selected and extracted from the encoded video data that make up the bitstream. In some embodiments, the selection of output operation points is based on the contents of the parameter sets (e.g., SPS, VPS, PPS) and the contents of the bit stream.

[0125] In some cases, the selected output operation point can be identified as an output operation point under test (also referred to as “Target Op”) by selecting a list of layer identifiers (OpLayerIdList) and a target highest temporal ID (OpTid). active. In such an example, one of the first or second bitstream conformance tests is applied to the selected output operating point.

[0126] As noted above, the third bitstream compliance test is applied to OLSs defined by the VPS extension part of the active VPS and its temporal subsets. For each test in the third set of bitstream conformance tests, the output operating point or points may be selected in a slightly different way than the first and second bitstream conformance tests. The selected output operation point (for example, the output operation point under test, or TargetOp) is selected by identifying a target OLS in the VPS span and the highest temporal ID (HighestTid) within the OLS.

[0127] FIG. 4 is a flowchart describing a method of implementing conformance testing. At block 410, encoding system 10 may receive encoded video data including a plurality of output operation points. The encoded video data may also include an indication of which layers are to be output for each of the output operating points. The indication can be included in a set of parameters (eg a VPS, SPS). The indication may also be implied when the output operation point corresponds to an entire bitstream or a temporal subset of the entire bitstream; either a layer set defined by the VPS base or a temporal subset of the layer set. In implied cases, only the base layer should be emitted. At block 415, the coding system 10 may extract an output operating point to be tested according to a bitstream conformance test. The output operating point can be selected based on the identification of OLSs, temporal subsets of the OLSs, temporal subsets of the entire bit stream, or as otherwise indicated in the VPS base or VPS extension. The one or more processors (processor) comprising the video encoder 20 and the video decoder 30 can then apply a certain compliance test to the output operating point (e.g. encoded video data) based on its content.

[0128] For example, in decision block 420, if the selected output operation point corresponds to an entire bitstream or a temporal subset of the entire bitstream with only the base layer as the output, method 400 may move to block 430. Active SPS for a base tier and non-nested SEI messages (for example, BP, PT, and DUI). The set of string-level HRD parameters can be defined in a hrd_parameters() syntax structure in the active SPS. In some embodiments, the first bitstream conformance test may further be based on a PTL syntax structure in the active SPS.

[0129] If the selected output operation point does not match the entire bitstream or a temporal subset of the entire bitstream with only the base layer as the output in decision block 420, method 400 moves to decision block 440. If, in decision block 440, the selected output operation point corresponds to one of a layer set defined by an active VPS base or a temporal subset of the layer set with only the base layer to be produced, the processor may perform a second bitstream compliance test at block 450. The second bitstream compliance test may be performed against the selected output operation point based on a sequence level HRD parameter set in the base VPS and directly nested SEI BP, PT or DUI messages. The sequence-level set of HRD parameters can be defined in a hrd_parameters() syntax structure in the active VPS base.

[0130] If, at decision block 440, the encoded video data does not match one of a layer set defined by a base part of an active VPS or a temporal subset of the layer set, with only the base layer being produced, the encoded video data corresponds to one of an OLS defined by a VPS extension of an active VPS or a temporal subset of the OLS, as shown in block 460. D) MAY perform a third bitstream conformance test at the selected output operating point based on a set of sequence-level HRD parameters in the base or extension part of the VPS actively and indirectly nested BP, PT, or DUI SEI messages. The set of string level HRD parameters can be defined in an hrd_parameters() syntax structure in the base or extension part of the active VPS. Thus, with the three different types of conformance tests, bitstream conformance tests for different types of output operation points are clearly and efficiently defined and in a way that is compatible with the HEVC specification version 1.

[0131] In some embodiments, for the output-from-base-layer-only OLSs among all OLSs defined in the VPS extension, if any, the hrd_parameters() syntax structure applicable in the base part of the active VPS, and the applicable directly nested SEI BP, PT, and DUI messages are used. The existing syntax and process allow for this in a way that is HEVC version 1 compliant, however it is not allowed for other OLSs defined in the VPS extension.

[0132] It should be noted that if a non-base-only-layer-output OLS contains all layers in the entire bitstream, still only indirectly nested SEI BP, PT, and DUI messages can be used, not non-nested ones, since non-nested ones are for the base-only-layer-output OLS containing all layers in the entire bitstream.

[0133] The use of the PTL syntax structure is the same for both base-only-layer-exit OLSs and non-base-layer-exit-only OLSs among all OLSs defined in the VPS extension. The most appropriately applicable PTL syntax structure is used, either in the base VPS or in the VPS extension.

[0134] Note that if a non-base-only-layer-output OLS contains all layers in the entire bitstream, a PTL syntax structure in the VPS may still be used, and not the one in the PTL syntax structure in the active SPS for the base layer, as this is for the base-layer-only output OLS containing all layers in the entire bitstream.

[0135] The information and signals disclosed herein may be represented using any of a variety of different technologies and methods. For example, data, instructions, commands, information, signals, bits, symbols and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0136] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments described herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, a number of illustrative components, blocks, modules, and steps have been described above, generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design limitations imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of the present invention.

[0137] The embodiments described herein can be implemented in hardware, software, firmware or any combination thereof. Such embodiments may be implemented in any of a variety of devices, such as general purpose computers, portable wireless communication devices, or integrated circuit devices that have multiple uses including application to portable wireless communication devices and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the methods may be performed, at least in part, by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random-access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. The methods additionally, or alternatively, may be performed, at least in part, by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer, such as signals or propagating waves.

[0138] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASIC), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. A similar processor can be configured to perform any of the methods and processes described in this invention. A general purpose processor can be a microprocessor; but alternatively the processor may be any conventional processor, controller, microcontroller or state machine. A processor can also be implemented as a combination of computing devices, for example a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors together with a DSP core, or any other type of configuration. Accordingly, the term "processor" as used herein can refer to any structure above, any combination of the structure above, or any other structure or apparatus suitable for implementing the methods described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0139] Although embodiments of the invention are described above for particular embodiment, many variations of the present invention are possible. For example, the number of different components can be increased or decreased, and modules and steps that determine a supply voltage can be modified to determine frequency, another system parameter, or a combination of parameters. Furthermore, the features of the various embodiments can be combined in combinations that differ from those described above.

[0140] It will be appreciated by those skilled in the art that the various illustrative blocks and modules described in connection with the embodiment disclosed herein may be implemented in a variety of ways. Some blocks and modules have been described above, generally in terms of their functionality. How this functionality is implemented depends on the design constraints placed on an overall system. Those skilled in the art may implement the described functionality in different ways for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of the invention. Also, grouping functions within a module, block, or step is for ease of description. Specific functions or steps can be moved from a module or block or distributed across modules or blocks without departing from the invention.

[0141] The above description of the embodiment presented is provided to enable any person skilled in the art to make or use the subject matter of the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein can be applied to another embodiment without departing from the spirit or scope of the invention. Thus, it is to be understood that the description and drawings presented herein represent a presently preferred embodiment of the invention and are, therefore, representative of the subject matter which is broadly contemplated by the present invention. It is further understood that the scope of the present invention fully encompasses any other embodiment which may become obvious to those skilled in the art and that the scope of the present invention is limited only by the appended claims.

Claims (15)

1. A method of processing video data, comprising: receiving (410) encoded video data having a plurality of output operation points; extracting (415) a selected output operation point from the plurality of output operation points, the selected output operation point being a sub-bitstream of an entire bitstream; and in response (420) to the selected output operating point corresponding to one of an entire bitstream with only the base layer being output, and a temporal subset of the entire bitstream with only the base layer being output, performing (430) a first bitstream conformance test on the selected output operating point, the first bitstream conformance test being based on a set of hypothetical reference decoder parameters, HRD, at sequence level on a set of sequence parameters, SPS, active for the base layer, and one or more non-nested supplemental enhancement information, SEI, messages, each non-nested SEI message of the one or more non-nested SEI messages being directly included in a network abstraction layer unit, NAL, SEI, in response (440) to the selected output operation point corresponding to one of a set of layers specified by a set of base video parameters, VPS, of an active VPS and a temporal subset of the layer set with only the base layer to be issued, performing (450) a second bitstream conformance test on the selected output operation point, the second bitstream conformance test being based on a sequence-level HRD parameter set in the base VPS and directly nested SEI messages, the directly nested SEI messages being directly included in scalable nesting SEI messages, and in response (460) to the selected output operation point corresponding to one of a set of output layers, OLS, specified by a VPS extension of the V active PS and a temporal subset of the OLS, performing (470) a third bitstream conformance test at the selected output operation point, the third bitstream conformance test being based on a set of sequence-level HRD parameters in the active VPS and indirectly nested SEI messages, the indirectly nested SEI messages being contained in bitstream partition nesting SEI messages, the bitstream partition nesting SEI message being contained in a scalable nesting SEI message able. 2. Method, according to claim 1, characterized in that it additionally comprises applying directly nested SEI messages only when the selected output operation point corresponds to an OLS identified in the base part of the active VPS, the directly nested SEI messages being one among SEI messages of buffer period, BP, of picture time, PT, and of decoding unit information, DUI. 3. Method, according to claim 1, characterized in that it additionally comprises applying the indirectly nested SEI messages only when the selected output operation point corresponds to an OLS specified in the VPS extension, the indirectly nested SEI messages being one of the SEI BP, PT and DUI messages. 4. Method according to claim 1, characterized in that it further comprises including the set of HRD parameters at sequence level in a hrd_parameters() syntax structure in one of the SPS, the base VPS and the VPS extension. 5. Method, according to claim 1, characterized in that the SEI messages comprise one of SEI BP, PT and DUI messages. 6. Method according to claim 1, characterized in that it further comprises performing the third bitstream compliance test at the selected output operation point, the third bitstream compliance test additionally being based on the set of HRD parameters at sequence level in the active VPS, and indirectly nested SEI messages being located in one of the base VPS and the VPS extension. 7. Method according to claim 1, characterized in that it further comprises selecting the output operation point selected from the plurality of output operation points of the encoded video data. 8. Apparatus for processing video data, comprising: memory means for storing encoded video data, the video data having a plurality of output operation points; and a decoder means (30) operatively coupled to the memory means for: extracting (415) a selected output operation point from the plurality of output operation points, the selected output operation point being a sub-bitstream of an entire bitstream; and in response (420) to the selected output operating point corresponding to one of an entire bitstream with only the base layer being output, and a temporal subset of the entire bitstream with only the base layer being output, performing (430) a first bitstream conformance test on the selected output operating point, the first bitstream conformance test being based on a set of hypothetical reference decoder parameters, HRD, at sequence level on a set of sequence parameters, SPS, active for the base layer, and one or more non-nested supplemental enhancement information, SEI, messages, each non-nested SEI message of the one or more non-nested SEI messages being directly included in a network abstraction layer unit, NAL, SEI, in response (440) to the selected output operation point corresponding to one of a set of layers specified by a set of base video parameters, VPS, of an active VPS and a temporal subset of the layer set with only the base layer to be issued, performing (450) a second bitstream conformance test on the selected output operation point, the second bitstream conformance test being based on a sequence-level HRD parameter set in the base VPS and directly nested SEI messages, the directly nested SEI messages being directly included in scalable nesting SEI messages, and in response (460) to the selected output operation point corresponding to one of a set of output layers, OLS, specified by a VPS extension of the V active PS and a temporal subset of the OLS, performing (470) a third bitstream conformance test at the selected output operation point, the third bitstream conformance test being based on a set of sequence-level HRD parameters in the active VPS and indirectly nested SEI messages, the indirectly nested SEI messages being contained in bitstream partition nesting SEI messages, the bitstream partition nesting SEI message being contained in a scalable nesting SEI message able. 9. Apparatus according to claim 8, characterized in that the decoder means (30) is a hypothetical reference decoder implemented by at least one processor. 10. Apparatus according to claim 8, characterized in that the decoding means (30) is configured to apply directly nested SEI messages only when the selected output operating point corresponds to an OLS identified in the VPS base of the active VPS, the directly nested SEI messages being one of the SEI messages of buffer period, BP, picture time, PT, and decoding unit information, DUI. 11. Apparatus according to claim 8, characterized in that the decoding means (30) is configured to apply indirectly nested SEI messages only when the selected output operating point corresponds to an OLS identified in the VPS extension of the active VPS, the indirectly nested SEI messages being one of SEI BP, PT and DUI messages. 12. Apparatus according to claim 11, characterized in that the set of HRD parameters at the sequence level is included in a hrd_parameters() syntax structure in one of the SPS, the base VPS and the VPS extension. 13. Apparatus according to claim 11, characterized in that the SEI messages comprise one of SEI BP, PT and DUI messages. 14. Apparatus according to claim 11, characterized in that the processor is additionally configured to perform the third bitstream compliance test at the selected output operation point, the third bitstream compliance test additionally being based on the sequence-level HRD parameter set in the active VPS, and indirectly nested SEI messages being located in one of the base VPS and the VPS extension. 15. Memory characterized by the fact that it comprises instructions stored therein, the instructions being executed by a computer to carry out the method as defined in any one of claims 1 to 7.