HK40010466A - Systems and methods for adaptive buffering for digital video streaming - Google Patents

Description

System and method for adaptive buffering for digital video streaming

Technical Field

The present invention relates generally to adaptive streaming, and more particularly to a system for adaptively buffering streaming of digital video.

Background

The term "streaming media" describes playback of media on a playback device, where the media is stored on a server and continuously transmitted to the playback device over a network during playback. Typically, the playback device stores a sufficient amount of media in the buffer at any given time during playback to prevent playback interruption due to the playback device completing playback of all buffered media before receiving the next portion of media. Adaptive bit rate streaming or adaptive streaming involves detecting the current streaming conditions (e.g., the user's network bandwidth and CPU capacity) in real-time and adjusting the quality of the streamed media accordingly. Typically, the source media is encoded at multiple bit rates, and the playback device or client switches between streaming different encodings depending on the available resources.

Adaptive streaming solutions typically stream media between a server and a playback device using either the hypertext transfer protocol (HTTP) published as RFC 2616 by the internet engineering task force and the world wide web consortium, or the real-time streaming protocol (RTSP) published as RFC 2326 by the internet engineering task force. HTTP is a stateless protocol that enables playback devices to request byte ranges within a file. HTTP is described as stateless because the server does not need to record information about the state of the playback device requesting the information or the byte range requested by the playback device in order to respond to a request received from the playback device. RTSP is a network control protocol for controlling a streaming media server. The playback device issues control commands, such as "play" and "pause," to the server streaming the media to control playback of the media file. When using RTSP, the media server records the state of each client device, and determines media to be streamed based on the instructions received from the client device and the state of the client.

In adaptive streaming systems, source media is typically stored on a media server as a top-level index that points to several alternate streams containing the actual video and audio data. Each stream is typically stored in one or more container files. Different adaptive streaming solutions typically utilize different indexes and media containers. Synchronized Multimedia Integration Language (SMIL), developed by the world wide web consortium, is used to create an index into several adaptive Streaming solutions, including IIS Smooth Streaming (IIS Smooth Streaming) developed by Microsoft Corporation of redmond, washington and Flash Dynamic Streaming (Flash Dynamic Streaming) developed by Adobe systems incorporated of san jose, california. HTTP adaptive bit rate Streaming (HTTP adaptive bit rate Streaming) developed by Apple Computer Incorporated of cupertino, california, implements an index file using an extended M3U playlist file (.m3U 8). M3U8 is a text file containing a list of URIs that typically identify media container files. The most commonly used media container formats are the MP4 container format specified in MPEG-4 part 14 (i.e., ISO/IEC 14496-14) and the MPEG Transport Stream (TS) container specified in MEPG-2 part 1 (i.e., ISO/IEC standard 13818-1). The MP4 container format is used in IIS smooth streaming and flash dynamic streaming. TS containers are used in HTTP adaptive bit rate streaming.

The Matroska container is a media container developed as an open standard project by the Matroska nonprofit organization of aussone, france. The Matroska container is based on the Extended Binary Meta Language (EBML), which is a binary derivative of the extensible markup language (XML). Decoding of Matroska containers is supported by many Consumer Electronic (CE) devices. The DivX Plus file format developed by DivX llc of san diego, california takes advantage of extensions to the Matroska container format (i.e., is based on the Matroska container format, but includes elements not specified within the Matroska format).

In order to provide a consistent means for delivering media content over the internet, the international organization for standardization (ISO) and the International Electrotechnical Commission (IEC) have proposed the dynamic adaptive streaming over HTTP (DASH) standard. The DASH standard specifies a format for media content and a description for content that delivers MPEG using HTTP. According to DASH, each component of media content for presentation is stored in one or more streams. Each stream is divided into segments. A Media Presentation Description (MPD) is a data structure that includes information about segments of each stream and other information needed to present media content during playback. The playback device uses the MPD to obtain components of the media content for playback using adaptive bitrate streaming.

Disclosure of Invention

Systems and methods for adaptive buffering for digital video streaming according to some embodiments of the present invention are illustrated.

One embodiment includes a playback apparatus comprising: a memory; a network interface; and a processor that reads instructions stored in the memory, the instructions directing the processor to perform the steps of: downloading digital video content in a buffer of a playback device; receiving a minimum buffer time from the digital video content; playing digital video content at slow motion speed using a playback device; decreasing the minimum buffer time by a slow motion playback speed factor; continuing to play the digital video content at the slow motion speed using the playback device until the reduced minimum buffer time is reached; and playing the digital video content using the playback device at a speed faster than the slow motion speed once the minimum buffer time is reached.

In a further embodiment, downloading the digital video content further comprises: requesting a list; extracting from the manifest a location of a container file containing the encoded media; and request a portion of the container file containing the encoded media.

In another embodiment, the encoded media downloaded in the buffer is compared to a reduced minimum buffering time; and playing the encoded media using the playback device when the encoded media exceeds the reduced minimum buffer time.

In still further embodiments, the minimum buffer time is the amount of time to download a minimum amount of media stored in the buffer to avoid interruption of playback.

In yet another embodiment, the minimum buffer time is stored in the digital video content at a location selected from the group consisting of a manifest, an index file, and a file to which the digital video content points.

In yet another embodiment, the digital video content further comprises a manifest and a plurality of container files, each container file containing an alternate digital video stream.

In yet another embodiment, the alternative digital video stream is a digital video stream encoded at a different bit rate.

In yet a further embodiment, the digital video content is a DASH standard file, with the minimum buffering time being stored in the media presentation description data structure.

In yet another embodiment, reducing the minimum buffering time further comprises reducing the digital video content filling the buffer by playing the available digital video content at a reduced speed.

In further additional embodiments, the slow-motion playback speed factor varies an amount of digital video content consumed for playback.

In another additional embodiment, the slow motion speed corresponds to a slow motion playback speed factor.

In a still further embodiment, digital video content is downloaded in a buffer of a playback device using a processor that reads instructions stored in a memory; receiving, using a processor that reads instructions stored in a memory, a minimum buffer time from the digital video content; playing digital video content at slow motion speed using a playback device and a processor that reads instructions stored in a memory; reducing the minimum buffer time by a slow-motion playback speed factor using the playback device and using a processor that reads instructions stored in a memory; and playing the digital video content at a speed faster than the slow motion speed using the playback device and a processor that reads instructions stored in memory once the minimum buffer time is reached.

In yet another embodiment, downloading the digital video content further comprises: using a processor request manifest that reads instructions stored in a memory; extracting, using a processor that reads instructions stored in a memory, a location of a container file containing the encoded media from a manifest; and requesting, using a processor that reads instructions stored in a memory, a portion of a container file containing the encoded media.

In yet a further embodiment, the encoded media downloaded in the buffer is compared to the reduced minimum buffering time using a processor that reads instructions stored in a memory; and playing the encoded media using a playback device using a processor that reads the instructions stored in the memory when the encoded media exceeds the reduced minimum buffer time.

In yet another embodiment, the minimum buffer time is the amount of time to download a minimum amount of media stored in the buffer to avoid interruption of playback.

In a further embodiment, the minimum buffer time is stored in the digital video content at a location selected from the group consisting of a manifest, an index file, and a file to which the digital video content points.

In another embodiment, the digital video content further comprises a manifest and a plurality of container files, each container file containing an alternate digital video stream.

In yet another embodiment, the alternative digital video stream is a digital video stream encoded at a different bit rate.

In yet another embodiment, the digital video content is a DASH standard file, with the minimum buffering time being stored in the media presentation description data structure.

In yet another embodiment, reducing the minimum buffering time further comprises reducing the digital video content filling the buffer by playing the available digital video content at a reduced speed.

In yet another embodiment, the slow-motion playback speed factor varies the amount of digital video content consumed for playback.

Again in a further embodiment, the slow motion speed corresponds to a slow motion playback speed factor.

Drawings

Fig. 1 is a diagram illustrating a network diagram of an adaptive bitrate streaming system according to an embodiment of the present invention.

Fig. 2 is a diagram illustrating a playback apparatus of an adaptive bitrate streaming system according to an embodiment of the present invention.

Fig. 3 is a diagram illustrating a server of an adaptive bitrate streaming system according to an embodiment of the present invention.

Fig. 4 is a flowchart illustrating an adaptive buffering process according to an embodiment of the present invention.

Detailed Description

Turning now to the drawings, systems and methods for providing adaptive buffering of digital video content during playback of the video content are illustrated in accordance with many embodiments of the present invention. In several embodiments, it is desirable to store a minimum amount of media in the buffer before video playback begins in order to reduce the likelihood of playback interruption to an acceptable level. In various embodiments, the amount of time to download this minimum amount of media may be referred to as the Minimum Buffer Time (MBT). The MBT may be stored in a Media Presentation Description (MPD) data structure in the DASH standard. In several embodiments, the MBT may be included in the manifest and/or in an index file of the MPD data structure. In many other embodiments, the MBT may be contained in a file pointed to by the MPD data structure. It should be readily apparent that the use of the DASH standard is merely illustrative, and that the minimum buffering time (or its equivalent) may be stored in various locations in the media manifest and/or in other media file types.

Changes in playback speed (i.e., slow motion and/or fast forward) generally change the amount of media consumed for playback. Slow motion playback reduces the amount of media consumed in a given amount of time compared to normal speed playback. In several embodiments of the present invention, reducing playback speed may enable the player to begin playback with reduced buffered media without significantly increasing the likelihood of playback interruption.

As an illustrative example, media according to several embodiments of the invention has an MBT of 8 seconds. The media will typically be buffered for 8 seconds before playback begins. If the same media is played back in slow motion at a rate of 1/2, the same media can begin playback based on a reduced MBT of 4 seconds. In the alternative, if the same media is played back in slow motion at a rate of 1/4, the same media may begin playback based on a reduced MBT of 2 seconds. Additionally, if the media is played back at a slow motion rate of 1/8, the same media may begin playback based on a reduced MBT of 1 second. Those of ordinary skill in the art will readily appreciate that these are merely examples and that different media may be played at different rates.

The encoding of source video for an adaptive bit rate streaming system that uses adaptive bit rate streaming to provide audio playback during trick play modes and media content playback in trick modes according to some embodiments of the present invention is discussed further below.

Adaptive streaming system architecture

Turning now to fig. 1, an adaptive streaming system including a playback device providing adaptive buffering according to an embodiment of the present invention is illustrated. Adaptive streaming system 10 includes a source encoder 12, source encoder 12 being configured to encode source media into a number of alternative streams. In the illustrated embodiment, the source encoder is a server. In other embodiments, the source encoder may be any processing device that includes a processor and resources sufficient to perform transcoding of source media (including, but not limited to, video, audio, and/or subtitles). In general, the source coding server 12 generates a top level index for a plurality of container files containing stream and/or metadata information, at least a plurality of which are alternate streams. The alternate stream is a stream that encodes the same media content in a different manner. In many cases, the alternate streams encode media content (such as, but not limited to, video content and/or audio content) at different maximum bitrates. In several embodiments, alternate streams of video content are encoded at different resolutions and/or different frame rates. The top level index file and the container file are uploaded to the HTTP server 14. The various playback devices may then request portions of the top-level index file, other index files, and/or container files via network 16 (such as the internet) using HTTP or another suitable stateless protocol.

In the illustrated embodiment, the playback devices include a personal computer 18, a CE player, and a mobile phone 20. In other embodiments, the playback device may include consumer electronics devices such as DVD players, blu-ray players, televisions, set-top boxes, video game consoles, tablets, and other devices capable of connecting to a server via HTTP and playing back encoded media. Although a particular architecture is shown in fig. 1, any of a variety of architectures may be utilized including systems that perform conventional streaming without performing adaptive bitrate streaming, which enables playback devices to request portions of a top level index file and a container file in accordance with embodiments of the present invention.

Playback apparatus

Some processes for providing a method and configuration system according to embodiments of the present invention are performed by a playback apparatus. Related components in a playback device that may perform processing according to embodiments of the present invention are shown in fig. 2. Those skilled in the art will recognize that the playback device may include other components that have been omitted for brevity without departing from the described embodiments of the invention. The playback device 200 includes a processor 205, non-volatile memory 210, and volatile memory 215. The processor 205 is a processor, microprocessor, controller, or combination of processors, microprocessors, and/or controllers that executes instructions stored in the volatile memory 215 or the non-volatile memory 210 to manipulate data stored in the memory. The non-volatile memory 210 may store processor instructions for configuring the playback device 200 to perform processes including processes according to embodiments of the present invention and/or data for processes being utilized. According to some embodiments, the instructions are included in a playback application that performs playback of media content on a playback device. According to various embodiments, the playback device software and/or firmware may be stored in any of a variety of non-transitory computer-readable media as appropriate for a particular application.

Server

Various processes performed within an adaptive streaming system according to different embodiments of the present invention are performed by an HTTP server, a source encoding server, and/or local and network time servers. The relevant components in the server that perform one or more of these processes according to embodiments of the present invention are shown in FIG. 3. Those skilled in the art will recognize that the server may include other components that have been omitted for brevity without departing from the described embodiments of the invention. The server 300 includes a processor 305, a non-volatile memory 310, and a volatile memory 315. The processor 305 is a processor, microprocessor, controller, or combination of processors, microprocessors, and/or controllers that executes instructions stored in the volatile memory 315 or non-volatile memory 310 to manipulate data stored in the memory. Non-volatile memory 310 may store processor instructions for configuring server 300 to perform processes including processes according to embodiments of the present invention and/or data for processes being utilized. According to some embodiments, the instructions to perform the encoding of the media content are part of an encoding application. According to various embodiments, the server software and/or firmware may be stored in any of a variety of non-transitory computer-readable media as appropriate for a particular application. Although a particular server is illustrated in FIG. 3, any of a variety of servers configured to perform any number of processes may be utilized in accordance with embodiments of the present invention. Adaptive buffering in accordance with many embodiments of the present invention is described below.

Adaptive buffer processing

Adaptive buffering in accordance with various embodiments of the present invention is illustrated in fig. 4. Process 400 includes receiving (402) a minimum buffer time from digital video content. Digital video content may be encoded and stored in any of a variety of formats including, but not limited to, DASH standard, MPEG-4, and/or Matroska containers. The playback device may then begin downloading (404) the digital video content.

The minimum buffer time is reduced (406) as a function of the slow motion playback speed factor. In many embodiments, a function of the slow-motion playback speed factor may change the amount of media consumed for playback. In various embodiments, the slower the playback speed, the greater the reduction in minimum buffer time. As an illustrative example, media according to several embodiments of the invention has an MBT of 8 seconds. The media will typically be buffered for 8 seconds before playback begins. If the same media is played back in slow motion at 1/2, the same media can begin playback based on a 4 second reduced MBT. In the alternative, if the same media is played back in slow motion at 1/4, the same media may begin playback based on a 2 second reduced MBT. Additionally, if the media is played back at a slow motion rate of 1/8, the same media may begin playback based on a reduced MBT of 1 second. Those of ordinary skill in the art will readily appreciate that these are merely examples and that different media may be played at different rates.

The digital video content begins playing in slow motion (408). In many embodiments, the digital video content begins playing once the MBT buffer is full. In several other embodiments, playback begins before the MBT buffer is full. As an illustrative example, when the available bandwidth is much higher than the bit rate of the digital video content, playback may begin when the MBT buffer is 75% full. Those skilled in the art will readily appreciate that this is merely an example and that playback of digital video content may depend on various factors in the system including, but not limited to, bandwidth, bit rate, MBT buffer size, playback speed factor, and/or system hardware requirements.

The slow motion digital video content continues to play (410) until a reduced minimum buffer time is reached. Although various processes for adaptively buffering digital video content are described above with respect to fig. 4, any of various processes capable of reducing the minimum buffering time may be utilized according to various embodiments of the present invention as desired for a particular application.

In several embodiments of the present invention, slow-motion playback speed and/or audio playback may increase (ramp up) as the MBT increases. In many embodiments of the present invention, the playback speed and/or audio playback may be ramped up as the MBT increases such that the playback speed and/or audio playback will be at a maximum speed when the MBT is reached.

In other embodiments of the invention, the process may detect when the buffer falls below a threshold and gradually reduce playback speed and/or audio playback until the buffer can be restored. This may attempt to delay a pause in the playback of the video stream and/or audio stream (as opposed to playing and then pausing the video stream and/or audio stream) with the playback gradually decaying. In general, this may occur in the context of delaying a down-switch to a lower bit rate and/or lower quality video stream. Alternatively, if the video stream and/or audio stream is the lowest quality stream, playback may be slowed until better bandwidth can be achieved. In general, embodiments of the invention may delay stream playback pauses as long as possible and gradually attenuate slow motion playback through various implementations in accordance with the invention, rather than from playback to pause.

Although the present invention has been described in certain specific aspects, many additional modifications and variations will be apparent to those of ordinary skill in the art. It is therefore to be understood that the present invention may be practiced otherwise than as specifically described, including various modifications of the implementation, such as with encoders and decoders that support features other than those specified within the particular standard to which they conform, without departing from the scope and spirit of the invention. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims (22)

1. A playback apparatus comprising:

a memory;

a network interface; and

a processor that reads instructions stored in the memory, the instructions directing the processor to:

downloading digital video content in a buffer of a playback device;

receiving a minimum buffer time from the digital video content;

playing the digital video content at slow motion speed using the playback device;

decreasing the minimum buffer time by a slow motion playback speed factor;

continuing to play the digital video content at the slow-motion speed using the playback device until a reduced minimum buffer time is reached; and is

Once a minimum buffer time is reached, the digital video content is played using the playback device at a speed faster than the slow motion speed.

2. The playback device of claim 1, wherein downloading the digital video content further comprises:

requesting a list;

extracting from the manifest a location of a container file containing the encoded media; and is

Requesting a portion of the container file that includes encoded media.

3. The playback apparatus of claim 2, further comprising:

comparing the downloaded encoded media in the buffer to the reduced minimum buffering time; and is

Playing the encoded media using the playback device when the encoded media exceeds the reduced minimum buffer time.

4. The playback device of claim 1, wherein the minimum buffer time is an amount of time to download a minimum amount of media stored in a buffer to avoid interruption of playback.

5. The playback device of claim 1, wherein the minimum buffer time in the digital video content is stored at a location selected from the group consisting of a manifest, an index file, and a file to which the digital video content points.

6. The playback device of claim 5, wherein the digital video content further comprises a manifest and a plurality of container files, each container file containing an alternate digital video stream.

7. The playback device of claim 7, wherein the alternate digital video stream is a digital video stream encoded at a different bit rate.

8. The playback device of claim 1, wherein the digital video content is a DASH standard file and the minimum buffer time is stored in a media presentation description data structure.

9. The playback device of claim 1, wherein reducing the minimum buffering time further comprises reducing the digital video content filling the buffer by playing the available digital video content at a reduced speed.

10. The playback device of claim 1, wherein the slow-motion playback speed factor varies an amount of digital video content consumed for playback.

11. The playback device of claim 1, wherein the slow-motion speed corresponds to the slow-motion playback speed factor.

12. A method for providing playback of digital video content, comprising:

downloading, using a processor that reads instructions stored in a memory, digital video content in a buffer of a playback device;

receiving, using the processor that reads instructions stored in memory, a minimum buffer time from the digital video content;

playing the digital video content at slow motion speed using the playback device and the processor reading instructions stored in memory;

reducing, using the playback device and using the processor that reads instructions stored in memory, the minimum buffer time by a slow-motion playback speed factor; and is

Once the minimum buffer time is reached, playing the digital video content at a speed faster than the slow motion speed using the playback device and the processor reading instructions stored in memory.

13. The method for providing playback of digital video content as recited in claim 12, wherein downloading the digital video content further comprises:

using the processor request manifest to read instructions stored in memory;

extracting, using the processor that reads instructions stored in memory, a location of a container file containing encoded media from the manifest; and is

A processor using the instructions to read stored in memory requests a portion of the container file that includes encoded media.

14. The method for providing playback of digital video content of claim 13, further comprising:

comparing, using the processor that reads instructions stored in memory, the downloaded encoded media in the buffer to the reduced minimum buffering time; and is

Playing, using the playback device, encoded media when the encoded media exceeds the reduced minimum buffer time, using the processor that reads instructions stored in memory.

15. The method for providing playback of digital video content as recited in claim 12, wherein the minimum buffer time is an amount of time to download a minimum amount of media storage in the buffer to avoid interruption of playback.

16. The method for providing playback of digital video content as recited in claim 12, wherein the minimum buffer time in the digital video content is stored at a location selected from the group consisting of a manifest, an index file, and file content to which the digital video content points.

17. The method for providing playback of digital video content as recited in claim 16, wherein the digital video content further comprises a manifest and a plurality of container files, each container file containing an alternate digital video stream.

18. The method for providing playback of digital video content as recited in claim 17, wherein the alternative digital video stream is a digital video stream encoded at a different bit rate.

19. The method for providing playback of digital video content as recited in claim 12, wherein the digital video content is a DASH standard file and the minimum buffer time is stored in a media presentation description data structure.

20. The method for providing playback of digital video content as recited in claim 12, wherein reducing the minimum buffering time further comprises reducing the digital video content filling the buffer by playing available digital video content at a reduced speed.

21. The method for providing playback of digital video content of claim 12 wherein the slow-motion playback speed factor varies an amount of digital video content consumed for playback.

22. The method for providing playback of digital video content of claim 12 wherein the slow-motion speed corresponds to the slow-motion playback speed factor.