GB2395387A

GB2395387A - Method of controlling switching between different bandwidth v ersions of video streams

Info

Publication number: GB2395387A
Application number: GB0226452A
Authority: GB
Inventors: Jonathan Soon Yew Teii; Anthony Richard May
Original assignee: Motorola Inc
Current assignee: Motorola Solutions Inc
Priority date: 2002-11-13
Filing date: 2002-11-13
Publication date: 2004-05-19
Anticipated expiration: 2022-11-13
Also published as: AU2003267427A1; GB2395387B; WO2004045216A1; GB2395387A8; HK1063402A1; GB0226452D0

Abstract

A method of controlling a video streaming device, comprising a switching means for switching between alternative bandwidth versions of one of a plurality of video streams. The switching means employs a plurality of switching methods. Switching method selection is controlled by data encoded within selected frames of each of the alternative bandwidth versions of the video stream. The encoded data indicates which if any of the plurality of switching methods to select at and/or after said selected frames for the desired video stream switch. This enables the video streaming device to employ different switching methods for each frame, enabling the reduction both of prediction errors after switching and data storage requirements.

Description

Video streaming device and method of control for switchable video streams

Technical Field

This invention relates to the area of streaming video transmission. Background

Streaming video involves transmitting video data from a server across a communications channel to a client, where it is played back in real-time as the video data is being received. This communications channel may include a 15 wireless channel, or a network such as a LAN, or the Internet. This channel may not always deliver sufficient bandwidth for the video data. As such, video streams are usually encoded at different bit rates in order to accommodate varying channel conditions.

Several existing video streaming systems that work over the Internet either measure the available bandwidth before the video data is transmitted, or alternatively allow the client device to indicate the available bandwidth. These 25 systems then transmit the appropriate bit stream, that being the bitstream which has a bit rate less than or equal to the available bandwidth.

However, such systems are unable to cope when the available 30 bandwidth changes during transmission of video data. When available bandwidth drops, the video is paused and the client has to wait for its buffer to fill up with video data before playback is resumed.

There is a need for a video-streaming server to be able to switch between alternative bandwidth video streams responsively as channel conditions change. However, implementing such a switch is not a matter of simply 5 changing to the alternative stream midway through transmission, as the different frame types used in hybrid video coding have to be taken into account to ensure that the resulting output at the client is of sufficient quality. As a further consideration, the server has limited resources in terms of storage space and processing power.

Any solution will preferably try to optimise both such resources and quality.

A number of video stream switching methods are known in the art, each corresponding to the different frame types present in a hybrid video codec such as MPEG-4 or H.26L. P and I-Frames are defined as follows: a) A P-frame is an interceded frame that may predict macroblocks (segments of the image) from previous frame(s), using what is termed backward prediction. P-frames are thus sensitive to stream continuity.

b) An I-frame is an intracoded frame that only uses prediction within the same frame, and can essentially be thought of as a single still image.

30 Consequently, there are P-frame and I-frame switching methods as outlined below.

P-frame switching is illustrated in figure 1. In figure 1, video stream two 120 can be assumed to have a lower data 35 rate and hence lower frame rate than stream one 110, and

the figure illustrates switching from stream one to stream two. In this case when a stream-switching request 130 is received, the server simply selects the next nearest frame 124 from the new stream, resulting in the transmitted frame 5 sequence 111, 112, 113, 124, 125, P-frame switching as illustrated in figure 1 will result in display errors. This is because the next nearest frame 124 in the stream two is encoded using predictions based on the 10 previous frame in stream two, 123, but due to the switch 140 must now use predictions from frame 113 of stream one, which does not contain identical information.

In most cases, the decoder would consequently suffer from 15 'drift', which occurs when subsequent P-frames are received at the decoder and the reconstructed image drifts further away from the originally encoded image. Drift is measured in terms of a quality measure, such as PSNR. The stream does eventually recover as the influence of the prediction 20 errors incurred at the switch recedes in time, but the perception by the user is of a period of low-integrity imaging. I-frame switching, by contrast, ensures that no errors 25 occur when a stream switch takes place. I-frame switching requires that I-frames be inserted in parallel within each video stream, often at the start of a new scene. Because I-

frames do not employ backward prediction, one can switch between streams upon reaching the next nearest I-frame in 30 the new stream without incurring a prediction error.

I-frame switching is illustrated in figure 2. In figure 2, stream two 220 can be assumed to have a lower data rate and hence lower frame rate than stream one 210, and the figure 35 illustrates switching from stream one to stream two. In

this case, when a stream-switching request 230 is received, the server must wait until an I-frame is due to occur in stream two 220. At this point the streams switch, 240, resulting in the transmitted frame sequence 211, 212, 213, 5 214, 215, 225, 226,

I-frame switching has a number of disadvantages. I-frames use proportionately more data and so necessarily reduce the overall quality of the encoded stream for a given 10 bandwidth. Moreover, I-frames are typically only included every 5 or 10 seconds, and so the responsiveness of the system to stream switch requests is slow, increasing the likelihood of stream pauses at the client. A final disadvantage is that at low bitrates, H.263 and Mpeg-4 15 encoders generally do not insert I- frames at all, and so an assumed use of I-frame switching for multiple streams would require modifications to the encoders.

A third method exists that attempts to overcome the 20 problems of I- and P-frame switching.

SP-frame switching utilises an additional supplementary data stream that bridges the two video streams during the switching process. The SP video stream is generated to 25 contain data for the nearest decompressed frame in a second stream, based on the previous input frame in a first stream. The result is that the switch from stream one to stream two does not incur prediction errors of the type seen with direct P-frame switching.

SP-frame switching is illustrated in figure 3. In figure 3, stream two 320 can be assumed to have a lower data rate and hence lower frame rate than stream one 310, and the figure illustrates switching from stream one to stream two. In 35 this case when a stream-switching request 330 is received

the server uses the SP-frame corresponding to the frame in stream one as the basis for prediction of the next frame in stream two. This results in the transmitted frame sequence 311, 312, 313, 314, 304, 325, 326,...

Whilst SP-Frame switching produces the best quality reconstructed frames at the decoder for minimal delay, it necessitates the generation of a separate SP-frame stream for each possible switching scenario, increasing the 10 storage requirements at the server.

If one assumes only that a stream switch is permitted between adjacent available bit rate streams, then for n possible bit rates one requires 2(n-1) SP-frame streams. If 15 one tries to mitigate the additional storage cost by only providing a corresponding SP frame every 10 P-frames, for example, this again reduces the responsiveness of the server to bandwidth changes.

20 The three switching methods above constitute 3 possible cost/quality trade-offs; i. P-switching incurs no cost, but impacts quality when there is frame misalignment between streams due to the 25 different bit rates.

ii. I-switching preserves switching quality, but incurs cost in the video stream by needing additional data that impacts on overall quality for a given bandwidth.

iii. SP-switching preserves switching quality, but incurs 30 cost in the server by requiring significant increases in storage and computation.

No one prior art method provides the ideal solution. Thus,

there is a need to optimise this cost/quality trade-off, to 35 improve the efficiency of video stream switching.

Summary of the Invention

In accordance with the present invention, there is provided 5 a method of controlling a video streaming device, as claimed in claim 1, and a video streaming device, as claimed in claim 8.

Brief description of the drawings

FIG. 1 shows the switching strategy for the P-frame switching method.

FIG. 2 shows the switching strategy for the I-frame 15 switching method.

FIG. 3 shows the switching strategy for the SP-frame switching method.

20 FIG. 4 shows the switching strategy for the P-frame switching method when P-Frames between video streams are aligned. Detailed description of the preferred embodiment

In a preferred embodiment of the present invention, selected frames of each video stream are augmented with switching strategy recommendation data. Thus for each selected frame, data indicating the use of P-frame, I frame 30 or SP-frame or another form of switching as the best option for that frame may be encoded, for each desired switching scenario. This would require 2 or less bits of information per scenario, and thus would impose little cost when compared to the typical 2,800 bits of information per P 35 frame.

In a preferred embodiment of the present invention, the generation of switching strategy recommendation data would use the following rules: i. Use P-frame switching when frames are aligned; ii. Use I-frame switching when the next available I-frame is an acceptably short distance from the current frame; 10 iii. Use any alternative switching method supported by the streaming protocol as appropriate; iv. Use SP-frame switching for remaining frames.

The provision of switching guidance need not be limited to 15 the three methods of P-frame, I-frame or SP-frame switching. Therefore rule iii above acknowledges that additional switching methods may benefit from control by the present invention.

20 P-frames are said to be aligned when they have substantially identical time-stamps. Figure 4 illustrates this occurrence, in which switching from stream one 410 to stream two 420 minimises prediction error as frames 413 and 422 are substantially identical for the purposes of 25 backward prediction by frame 423.

Evaluation Evaluations in support of the present invention suggest 30 that a significant number of P-frames are aligned. The table below is based on a 90 second test video sequence containing a number of different scenes:

Switching Scenario Frame alignments 36 kbps 22 kbps 37% 22 kbps 15 kbps 24% Thus rules i. and iv. above may reduce the required number of SP frames by a third for these four switching scenarios.

5 Moreover, the proportion of aligned P-Frames increases with bit rate, and consequently the use of the present invention reduces the number of required SP-frames as SP-frames get larger. 10 Inclusion of I-frames in the video stream, for example at the start of each new scene, would further reduce the required number of SP frames by use of rule ii.

Alternative Embodiment In an alternative embodiment, it may be decided that it is not necessary to be able to switch between streams at every possible frame. For example, if it is known that the client buffer is capable of storing 2 seconds of video, then a 20 switch delay of 0.5 seconds may be acceptable.

In the alternative embodiment, SP frames are therefore only required where there is more than (for example) a 0.5 second gap until the next pair of aligned P-frames or until 25 the next I-frame.

Evaluation Evaluations in support of the present invention suggest 30 that this would additionally reduce the need for SP frames.

The table below is based on a 90 second test video sequence containing a number of different scenes:

Switching Higher No. of Average SP Frame Scenario Frame rate Alignments Interval Rate 36 kbps 8.06 fps 170 0. 53s c 1 in 8 22 kbps 22 kbps 4.94 fps 88 1. 02S 1 in 5 15 kbps The final right-hand column suggests the likely proportion of required SP-Frames relative to the number of Pframes.

5 For the higher bandwidth example, fewer than 1 in 8 P-

frames will require corresponding SP-frames, as P-frame alignment occurs on average every half second. Where there are longer gaps, one frame in every 0. 5s of the gap will have its switching strategy recommendation data encoded to 10 provide an SP-frame switch at that point, and a corresponding SP-frame will be generated. For the lower bandwidth example, roughly 1 in 5 P-frames will require corresponding SP-frames as P-frame alignment occurs on average every second, so requiring typically one SPframe 15 between such alignments.

Again, the addition of I-frames will further reduce the SP-

frame requirement.

20 In summary, a preferred embodiment of the present invention

incorporates switching strategy recommendation data within video frame data, for alternative bit rate video streams.

The switching strategy recommendation data indicates which of a plurality of alternative switching strategies to use 25 for a given frame, based on rules concerning the location of frames that permit low error switching to occur, such as I-frames and aligned P-frames. The benefit of such an arrangement is to maintain quality whilst minimising cost, particularly in terms of SP-frame generation and storage.

30 An alternative embodiment proposes further reductions in

SP-frame numbers, by only recommending SP-frames if the temporal proximity of aligned P-frames or I-frames exceeds a stipulated duration.

Claims

1. A method of controlling a video streaming device, the video streaming device comprising a switching means for 5 switching between alternative bandwidth versions of one of a plurality of video streams, the switching means employing a plurality of switching methods, the method of controlling comprising selecting the switching method in dependence on: the encoding of data within one or more selected 10 frames of each of the plurality of alternative bandwidth versions of the video stream, the encoded data indicating which, if any, of the plurality of switching methods to select at and/or after said selected frame, in order to make the desired video 15 stream switch.

2. A method of controlling a video streaming device according to claim 1, wherein: the switching means employs any of I-frame, P-frame or SP 20 frame switching; and the selection of I-frame, P-frame or SP-frame switching is determined by the encoding of data within the selected frames indicating which, if any, of I-frame, P-frame or SP-

frame switching to select at and/or after said selected 25 frames.

3. A method of controlling a video streaming device according to claim 1 or claim 2, wherein every video frame is selected for the encoding of data.

4. A method of controlling a video streaming device according to any previous claim, wherein switching method selection data is encoded to include method selection information only for the next 35 higher alternative bandwidth video stream corresponding to

the current video stream, where available, and next lower alternative bandwidth video stream corresponding to the current video stream, where available.

5 5. A method of controlling a video streaming device according to any previous claim, wherein SP-frames that have been generated for the purpose of SP-frame switching between a pair of alternative bandwidth versions of the video stream are only stored if 10 the encoded data on either of the aforesaid pair of alternative bandwidth versions of the video stream indicates that the SP frame is required.

6. A method of controlling a video streaming device 15 according to any previous claim, wherein the encoded data indicating which if any of the plurality of switching methods to select is based on the following rules; i. Use P-frame switching when frames are aligned; 20 ii. Use I-frame switching when the next available I-frame is reached, if the next available I-frame is an acceptably short distance from the current frame for the purposes of the application; iii. Use any alternative switching method supported by the 25 streaming protocol as appropriate; iv. Use SPframe switching for remaining frames.

7. A method of controlling a video streaming device according to any of claims 1 to 5, wherein the encoded data 30 indicating which if any of the plurality of switching methods to select is based on the following rules; i. if the next available aligned P-frame is an acceptably short distance from the current frame for the purposes of the application, then use Pframe switching when 35 the next available aligned P-frame is reached;

ii. if the next available I-frame is an acceptably short distance from the current frame for the purposes of the application, then use I-frame switching when the next available I-frame is reached; 5 iii. if the availability of said switching method occurs in an acceptably short time for the purposes of the application, then use any alternative switching method supported by the streaming protocol when said alternative switching method becomes available; 10 iv. if rules i., ii. or iii. are not met, then use SP-

frame switching for each frame where the onset of the subsequent frame would exceed the acceptable time following the last available switching position.

15

8. A video streaming device comprising a switching means for switching between alternative bandwidth versions of one of a plurality of video streams, the switching means employing a plurality of switching methods, and the switching means being adapted to select the bandwidth 20 version of a video stream in dependence on: encoded data within one or more selected frames of each of the plurality of alternative bandwidth versions of the video stream; the encoded data indicating which, if any, of the 25 plurality of switching methods to select at and/or after said selected frames in order to make the desired video stream switch.

9. A video streaming device according to claim 8, wherein 30 the video streaming device is a server.