WO2007068139A1

WO2007068139A1 - System and method for controlling the media stream of the multiparty video communication

Info

Publication number: WO2007068139A1
Application number: PCT/CN2005/002152
Authority: WO
Inventors: Zhengxun Xia; Jindong Wang; Jianzhang Ye; Chengjin Pan
Original assignee: Zte Corporation
Priority date: 2005-12-12
Filing date: 2005-12-12
Publication date: 2007-06-21
Also published as: CN101273631B; CN101273631A

Abstract

The invention discloses a system and a method for controlling the media stream of the multiparty video communication. The system includes a transmitting control path for media stream, a communication terminal and a server. Therein, the communication terminal includes a RTCP unit, a RTP output unit, and a RTP input unit, a TS encoding unit, an I frame encoding unit, a media buffer unit, a TS decoding UNIT, an I frame decoding unit and a stream controlling unit. The server includes an I frame memory unit, a stream controlling unit and a communication sub-unit. The communication sub-unit includes a RTCP transmitting unit, a RTP output unit and a RTP input unit, an I frame extraction unit, a RTP de-packeting unit, a media buffer unit and a media source unit. The invention also discloses a method for transmitting the static picture mixed with the TS stream and a method for seamless switching and controlling the media stream, which reduce the burden of the server and the communication terminal and provide enough information in order to reduce the process of the multiparty video communication.

Description

TECHNICAL FIELD The present invention relates to the field of multimedia technologies, and relates to a multi-party video communication media stream control system and method, and more particularly to a static image and a real-time stream in real-time multi-party video communication.

(Transport Stream) (hereinafter referred to as TS stream) system and method for hybrid transmission and system and method for media stream control and handover. BACKGROUND With the development of network technology, a new media technology emerges as the times require. This is streaming media technology. Streaming media refers to continuous time-based media, such as audio, video or multimedia files, that use streaming technology in the network. Streaming media technology has been greatly used in video telephony and video conferencing systems. In the existing video communication system, each communication terminal communicates through a streaming media server. In a multi-party video communication, a communication terminal usually has a plurality of communication links to receive audio and video data of other communication terminals. As a communication center, it is very burdensome. As a communication terminal, it may not care about the video information of all the communication terminals participating in the communication. Therefore, on the basis of this, the industry has proposed several methods to reduce the burden on the server, for example, in the video conference system, each communication terminal. Only the audio and video data of the communication terminal that is speaking is received. Although the above method partially relieves the burden on the server, there are still the following shortcomings -

1. The switching procedure of the communication terminal switching from one media source to another is complicated;

2. In a multi-party video communication process, if the communication terminal only receives data from a certain communication source, the integrity of the information will be lost. SUMMARY OF THE INVENTION In view of the above circumstances, an object of the present invention is to provide a multi-party video communication media stream control system and method. The invention fully considers the characteristics of the multi-party video communication system, and proposes a System and method for hybrid transmission of static images and TS streams and a system and method for seamlessly switching and controlling media streams, which can provide sufficient information while reducing the load of servers and communication terminals, thereby simplifying the process of multi-party video communication .

The present invention mainly proposes a system and method for solving the following three problems:

1. How to identify these two kinds of data when the still image and the TS stream are simultaneously transmitted;

2. When the multi-party communication is established, the communication terminal receives the static image of the other communication terminal, and selects the main media source concerned by it;

3. How to seamlessly switch the main media source.

To achieve the above object, a multi-party video communication media stream control system of the present invention includes a media stream transmission control path for transmitting audio and video data and a control signal; and a communication terminal responsible for transmitting and receiving the audio and video data and the control signal; The server is responsible for providing the media stream service for the video communication terminal in response to the above control signal; wherein the communication terminal includes: Real Time Transport Control Protocol (RTCP) unit, Real Time Transport Protocol (hereinafter) RTP) output unit and RTP input unit, TS coding unit, intra-framed frame (I-frame, also called intra picture) (hereinafter referred to as I frame) coding unit, media buffer unit, TS decoding unit, I a frame decoding unit and a flow control unit; the server includes: an I frame storage unit, a flow control unit, and a communication subunit, wherein the communication subunit includes: an RTCP transmission unit, an RTP output unit, and an RTP input unit, an I frame extraction unit, and an RTP Unpacking unit, media buffer Meta and media source unit.

The RTP output unit of the communication terminal is responsible for transmitting the audio and video data of the communication terminal to the RTP input unit of the communication subunit corresponding to the communication terminal in the server; the RTP input unit of the communication terminal is responsible for receiving from the server Audio and video data of other communication terminals;

The TS coding unit of the communication terminal is responsible for TS stream coding of the communication terminal; the I frame coding unit of the communication terminal is responsible for periodically encoding the static image of the communication terminal, and forming I frame data output to the RTP output unit of the communication terminal, and using a specific Payload type in the above RTP packet to mark the I frame data of the communication terminal; The media buffer unit of the communication terminal is responsible for receiving data of the RTP input unit of the communication terminal;

The TS decoding unit of the communication terminal is responsible for extracting the TS stream from the media buffer unit and decoding it;

The I frame decoding unit of the communication terminal is configured to extract the RTP packet having the specific Payload type from the media buffer unit, and decode the I frame data to form a still image; the flow control unit of the communication terminal is responsible for the flow with the server. The control unit communicates, notifies the server of the main media source currently used by the communication terminal or the instruction of the TS coding unit that starts the communication terminal of the server;

The I frame storage unit of the server is responsible for storing I frame data of each communication terminal; the flow control unit of the server is responsible for controlling the link of the audio and video streams of each communication terminal; the communication subunit of the server is responsible for establishing a communication link with the communication terminal. Transmitting the audio and video data and the control signal, and outputting the video I frame data to the I frame storage unit of the server, and acquiring the media from the RTP unpacking unit of the I frame storage unit of the server and other communication subunits of the server Source data is input to a media source unit of the communication subunit;

The RTP output unit of the communication subunit of the server is responsible for transmitting audio and video data of other communication terminals to the communication terminal corresponding to the communication subunit;

The RTP input unit of the communication subunit of the server is responsible for receiving audio and video data of the communication terminal;

The I frame extracting unit of the communication subunit of the server is responsible for extracting I frame data according to a specific Payload type of the RTP packet, and storing it in the I frame storage unit of the server; if the I frame extracting unit is not found For the above type of RTP packet, the I frame data is directly extracted from the data in the RTP unpacking unit of the communication subunit of the server; for example: if the current video adopts the MPEG4 encoding format, the vop-coding-type or group_of may be adopted. The -vop_start_code field finds the I frame data; The RTP unpacking unit of the communication subunit of the server is responsible for removing the RTP header of the received audio and video data after filtering out the data packet required by the I frame extracting unit of the communication subunit of the server, and outputting the RTP header. Go to the media buffer unit of the communication subunit;

The media buffer unit of the communication subunit of the server is responsible for receiving data output by the RTP unpacking unit of the communication subunit of the server;

The media source unit of the communication subunit of the server is responsible for acquiring audio and video data required by the communication terminal corresponding to the communication subunit from the I frame storage unit of the server and the media buffer unit of the communication subunit corresponding to the other communication terminal.

In addition, the RTP output unit of the communication terminal is respectively connected to the TS coding unit and the I frame coding unit of the communication terminal;

The RTP input unit of the communication terminal is connected to the media buffer unit of the communication terminal; the media buffer unit of the communication terminal is respectively connected to the TS decoding unit and the I frame decoding unit of the communication terminal;

The I frame extracting unit of the communication subunit of the server is respectively connected to the I frame storage unit of the server, the RTP input unit of the communication subunit of the server, and the RTP unpacking unit; the RTP unpacking unit of the communication subunit of the server respectively Connecting to the RTP input unit, the I frame extracting unit, and the media buffer unit of the communication subunit of the server;

The media source units of the communication subunits of the server are respectively connected to the RTP output unit of the communication subunit of the server, the I frame storage unit of the server, and the flow control unit.

The communication terminal marks the I frame data of the communication terminal by defining a specific load type (hereinafter referred to as a Payload type) in the RTP packet, and the RTP packet of the type includes only the I frame data of the audio and video data of the communication terminal; Because it is a communication system, the communication terminal and the server are two-way audio and video stream communication; when the multi-party video communication is established, the server includes a plurality of the communication subunits, and each of the communication subunits and each of the communication terminals Corresponding; After the multi-party video communication is established, the above communication terminal and the server can transmit audio and video data through at least the RTP/RTCP transmission channel.

To achieve the above object, a multi-party video communication media stream control method of the present invention, the method for hybrid transmission of a static image and a TS stream includes the following steps: Step 101: The RTP output\input unit of the communication terminal establishes a pair of data communication links with the RTP input/output unit of the communication subunit corresponding to the communication terminal in the server, wherein the static image and the TS stream pass through the pair. Data communication link transmission;

Step 102: The I frame encoding unit of the communication terminal periodically encodes the static image into I frame data, and transmits the static image to the RTP output unit of the communication terminal.

Step 103: If the RTP output unit of the communication terminal receives the I frame data in the I frame coding unit, package it into an RTP packet with a specific Payload type, and send it immediately; if the RTP output unit of the communication terminal receives The data of the TS coding unit of the communication terminal is encapsulated into an RTP packet, and then sent according to a time stamp (hereinafter referred to as TimeStamp); wherein the RTP packet of the specific Payload type is different from the RTP packet of the TS stream; 104: If the RTP input unit of the communication subunit corresponding to each communication terminal in the server receives the RTP packet of the specific Payload type, extract the I frame data and store it in the I frame storage in the server. In the unit; if the TS stream data is received, the packet header is removed by the RTP unpacking unit of the communication subunit, and then stored in the media buffer unit of the communication subunit;

Step 105: The media source unit timing of the communication subunit corresponding to each communication terminal in the server acquires I frame data of another communication terminal from the I frame storage unit, and the main media source determined by each communication terminal Obtaining the TS stream data in a media buffer unit of the corresponding communication subunit;

Step 106: If the RTP output unit of the communication subunit corresponding to each communication terminal in the server receives the RTP packet of the specific Payload type, send it immediately; if the RTP packet of the TS stream is received, press The above TimeStamp is sent;

Step 107: If the RTP input module of the communication terminal receives the RTP packet of the specific Payload type, transmit it to the I frame decoding unit of the communication terminal to form a still image of another communication terminal; if the TS is received The streamed RTP packet is transmitted to the TS decoding unit of the communication terminal to form a TS stream of the primary media source.

The data communication link in the above step 101 can be, for example, an RTP/RTCP transmission. Road.

To achieve the above object, a multi-party video communication media stream control method of the present invention, the method for selecting a primary media source for a still image received through a communication terminal includes the following steps:

Step 201: When initially starting multi-party video communication, the communication terminal only performs I frame coding, and the I frame data is periodically sent to the server through an RTP packet of a specific Payload type;

Step 202: The server receives the I frame data of each communication terminal, and stores the data in the I frame storage unit of the server.

Step 203: The communication subunit corresponding to each communication terminal in the server acquires I frame data of another communication terminal from the I frame storage unit, and sends the I frame data to the communication terminal. Step 204: The communication terminal is configured according to the foregoing a static image of the communication terminal, selecting a primary media source, and then the flow control unit of the communication terminal notifying the flow control unit of the server of the primary media source selected by the communication terminal;

Step 205: The server checks whether the primary media source selected by the communication terminal is transmitting a TS stream, and if not, notifies the flow control unit of the primary media source to start the TS coding unit; Step 206: The flow control unit of the server sends a control The signal is supplied to the media source unit of the communication subunit corresponding to the communication terminal to acquire the TS stream data from the media buffer unit of the communication subunit corresponding to the main media source.

The primary media source is another communication terminal different from the communication terminal.

To achieve the above object, a multi-party video communication media stream control method of the present invention, the method for switching a communication terminal from a current primary media source to another primary media source includes the following steps: Step 301: Flow control of the communication terminal 4 The unit notifies the server's flow control unit to switch its current primary media source from 4, A ₂ , ..., 4 to 4;

Step 302: The server checks whether the primary media source is transmitting the TS stream, and if not, notifies the flow control unit of the primary media source to activate the TS encoding unit of the primary media source.

Step 303: The flow control unit of the server sends a control signal to the communication terminal. 4 The media source unit of the corresponding communication subunit obtains the TS stream data from the media buffer unit of the communication subunit corresponding to the above main media source _y . The current primary media sources 4, ..., 4 and the primary media source are communication terminals different from the communication terminal; the subscripts x, y, and n are different from each other.

Compared with the existing video communication system, the invention has the following advantages:

1. When the multi-party video communication is established in the initial stage, the communication terminal only periodically transmits the video I frame data, thereby shortening the time for establishing the multi-party video communication;

2. The communication terminal can receive the TS stream data of the main media source of the communication terminal concerned by the current communication terminal while displaying the static image of the other communication terminal that is updated regularly, and can save the bandwidth while ensuring the maximum amount of information acquired;

_3. Compared with other media stream switching technologies, the present invention establishes an RTP/RTCP transmission channel independent of the media source, and does not need to perform TimeStamp and Synchronization Source Identifier (SSRC) conversion during transmission. The above and other objects, features and advantages of the present invention will become more <RTIgt; BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a system configuration diagram of the present invention;

2 is a schematic diagram of an application environment according to a preferred embodiment of the present invention; FIG. 3 is a schematic diagram of system processing during three-party communication in the system of the present invention;

4 is a schematic flowchart of a process when a communication terminal periodically displays a still image of another communication terminal according to the present invention;

FIG. 5 is a flowchart of processing when the communication terminal A of the present invention confirms the communication terminal B as its main media source;

FIG. 6 is a flowchart of processing when the communication terminal A of the present invention switches its main media source from the communication terminal B to the communication terminal C. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings.

Figure 1 is a block diagram of the system of the present invention. As shown in FIG. 1, the system of the present invention includes a media stream transmission control path, a communication terminal, and a server, wherein the communication terminal includes: an RTCP unit, an RTP output unit, and an RTP input unit, a TS coding unit, an I frame coding unit, and a media. The buffer unit, the TS decoding unit, the I frame decoding unit and the flow control unit; the server includes: an I frame storage unit, a flow control unit and a communication subunit, wherein the communication subunit comprises: an RTCP transmission unit, an RTP output unit, and an RTP input Unit, I frame extraction unit, RTP unpacking unit, media buffer unit and media source unit.

2 is a schematic diagram of an application environment according to a preferred embodiment of the present invention. As shown in Figure 2, communication terminals A, B, and C are connected to the video communication system server through the mobile network, the Internet (Internet), and the intranet (intranet) respectively. When multi-party video communication starts (the system processing diagram is shown in Figure 3). The server creates three communication sub-units respectively corresponding to the three communication terminals A, B, and C; each of the communication terminals establishes an RTCP control link and an RTP input/output link for each of the communication sub-units to transmit Audio and video data, and a flow control link is established to transmit primary media source control information. The communication terminal A receives the audio and video data sent from the server, extracts the I frame data therein to form a static image of the timing update of the other communication terminal, and determines that the main media source is the communication terminal B according to the display screen, so the above The communication terminal B needs to simultaneously output TS stream data and timed still image data, and the communication terminal A receives the TS stream data of the communication terminal B while receiving the static image data of the timing of the communication terminals B and C. When the communication terminal A selects the communication terminal C as its main media source, the server can smoothly switch the TS stream from the communication terminal B to the communication terminal C without the communication terminal A performing the switching operation. The operation is seamless switching.

In combination with the above real-time example, the present invention must have the following functions after being successfully implemented:

1. The I frame data sent by the communication terminals B and C, the communication terminal A can receive and display; 2. After the communication terminal A determines that the communication terminal B is its main media source according to the display image, the communication terminal B starts to transmit its TS stream, and the communication terminal A can receive the static image sent by the communication terminal C at the same time. The TS stream of the communication terminal B;

3. If the communication terminal A selects the communication terminal C as its main media source, the server can switch the TS stream from the communication terminal B to the communication terminal C without the above-mentioned communication terminal A being aware of it.

In order to achieve the above purpose, and in order to save bandwidth and speed up the communication establishment, when the multi-party video communication is established, each communication terminal participating in the communication only periodically transmits the I frame data, and each communication terminal can display the static image of the timing update of the other communication terminals. The general process of this process is shown in Figure 4. Please refer to Figure 2, Figure 3 and Figure 4 for the specific implementation steps as follows - Step 1: When multi-party video communication is established, the server creates a corresponding communication sub-unit for each communication terminal. , the communication terminal and the communication subunit establish an RTP/RTCP transmission channel;

Step 2: The I frame encoding unit of the communication terminal VIII, B, and C periodically (for example, every 10 seconds) acquires the current still image of the communication terminal, encodes it into I frame data, and transmits it to the above communication terminals. RTP. output unit;

Step 3: When receiving the data of the I frame coding unit, the RTP output unit of the communication terminals A, B, and C encapsulates an RTP header for the data, where the SSRC field of the RTP header and the RTP header of the TS stream data of the communication terminal SSRC, which marks the source of the data; meanwhile, the Payload field of the RTP header is defined as a specific 7-bit data to mark the static image data packet from the TS stream;

Step 4: After the RTP output unit of the communication terminal adds the RTP header to the static image data, immediately sends the RTP header to the communication subunit A corresponding to each communication terminal, B, C in the server through the RTP transmission channel. , B, C RTP input unit;

Step 5: The I frame extracting unit of the communication subunits A, B, and C corresponding to each of the communication terminals A, B, and C in the server periodically periodically receives the Payload of the RTP header from the RTP input unit of the communication subunit. Field extracting still image data and transmitting it to the I frame storage unit of the above server; Step 6: The I frame storage unit of the server distinguishes static image data from different communication terminals according to different SSRC fields in the RTP header of each data packet;

Step 7: The media source units of the communication subunits A, B, and C corresponding to the communication terminals A, B, and C in the server periodically acquire static image data of other communication terminals from the I frame storage unit. And immediately transmit it to the RTP output unit of its communication subunit, the RTP output unit immediately transmits such data to the RTP input unit of the communication terminal corresponding to its communication subunit;

Step 8. The media buffer unit of the communication terminals A, B, and C receives data from the RTP input unit of the communication terminal, extracts still image data according to the Payload field of the RTP packet, and distinguishes other data according to the SSRC field of the RTP packet. Data of different communication terminals to form complete I frame data of different communication terminals;

Step 9: The I frame decoding unit of the communication terminals A, B, and C decodes the I frame data, and distinguishes different sources of the decoded image according to the SSRC field, so that the communication terminals VIII, B, and C can See the static image of the timing update of other communication terminals.

For the communication terminal A, it is now possible to see the static images of the communication terminals B and C that are regularly updated, and to select the main media source B of interest according to the static images of the B and C of the communication terminal, so that the communication is performed. The terminal A can receive the TS stream of the communication terminal B while receiving the static image updated by the timing of the communication terminals B and C. The general flow of the process is shown in FIG. 5, please refer to FIG. 2, FIG. 3 and FIG. 5. The specific implementation steps of this process are as follows:

Step 1: The communication terminal A selects the communication terminal B as its current main media source, and the flow control unit of the communication terminal A sends a control signal to the flow control unit of the server to request the current primary media source of the communication terminal A to be the communication terminal B;

Step 2: After receiving the request that the primary media source is the communication terminal B, the flow control unit of the server first checks whether the TS stream access count of the current communication terminal B is 0, and if the communication terminal B The TS stream access count is 0, indicating that the communication terminal B starts to establish multi-party video communication, and the flow control unit of the server sends a TS stream start request to the flow control unit of the communication terminal B; otherwise, step 6 is performed; Step 3: After receiving the TS stream start request sent by the server, the flow control unit of the communication terminal B notifies the TS coding unit of the communication terminal to start working, and the TS coding unit is working on the current communication terminal. The audio and video data of B is encoded, and the encoded data is transmitted to the RTP output unit of the communication terminal B, and the RTP output unit adds the RTP header to the encoded data, and transmits it according to TimeStamp;

Step 4: After receiving the TS stream data sent by the communication terminal B, the RTP input unit of the communication subunit B in the server transmits the TS stream data to the RTP unpacking unit of the communication subunit, where the Payload field according to the RTP packet is used. The difference is static image data or TS stream data;

Step 5: The RTP unpacking unit of the communication subunit B in the server removes the RTP header of the TS stream data and stores the data in the media buffer unit of the communication subunit. Step 6: The flow control unit notification of the server The media source unit of the communication subunit A takes data from the media buffer unit of the communication subunit B, and the TS stream access count of the communication terminal B is incremented by one; • the I frame data of the TS stream is started when the data is started. Start fetching, if it is MPEG4 compression encoding, you can find the I frame by vop-coding_type or group-of-vop-start_code field;

Step 7: The media source unit of the communication subunit A of the server obtains the TS stream data from the media buffer unit of the communication subunit B, and transmits the TS stream data to the RTP output unit of the communication subunit;

Step 8: The RTP output unit of the communication subunit A of the server adds an RTP header to the TS stream data according to the RTP session attribute currently established with the communication terminal A, that is, the SSRC field of the RTP header is the communication terminal A and the communication. Subunit A establishes the SSRC of the RTP session, wherein the TimeStamp and the serial number (hereinafter referred to as SqucneceNumber) are independent of the source of the data source; the RTP output unit of the communication subunit transmits the TS stream data according to the TimeStamp;

Step 9: The RTP input unit of the communication terminal A receives the TS stream data and transmits it to the TS decoding unit of the communication terminal, and the TS decoding unit decodes the real-time audio and video stream data of the communication terminal B, and Display images and play sounds. Thus, the communication terminal A can receive and play the real-time audio and video data of the communication terminal B while receiving the static image updated by the timing of the communication terminals B and C. When the communication terminal A wants to switch the current main media source from the communication terminal B to the communication terminal C, the communication terminal C only needs to be selected according to the static image, and the server uses the main media source of the communication terminal A from the communication terminal. B switches to the above communication terminal (for the above communication terminal A, shortly after it makes a selection, the display window of the TS stream will display the image of the communication terminal C, and play the sound of the communication terminal C, the process roughly processes such as Figure 6. Please refer to Figure 2, Figure 3 and Figure 6. The specific implementation steps of this process are as follows:

Step 1: The flow control unit of the communication terminal A sends a flow switching instruction to the flow control unit of the server, requesting that its main media source be switched from the communication terminal B to the communication terminal C;

Step 2: After receiving the flow switching instruction of the communication terminal A, the flow control unit of the server first checks the current TS flow access count of the communication terminal C, and if the TS flow access count of the communication terminal C is > 0, the communication is indicated. If the terminal C has started to transmit its TS stream, step 3 is performed. If the TS stream access count of the communication terminal C is 0, indicating that the communication terminal C does not transmit the TS stream, the flow control unit of the server provides the communication terminal C. The flow control unit sends a TS stream start request, as described above, the communication terminal C starts to send its TS stream, and the media buffer unit of the communication subunit C stores the real-time audio and video data of the communication terminal C;

Step 3: The flow control unit of the server reduces the TS flow access count of the communication terminal B by one, and checks whether the current TS terminal access count of the communication terminal B is 0. If the count is 0, it indicates that no communication terminal needs the communication terminal. In the TS stream of B, the flow control unit of the server notifies the flow control unit of the communication terminal B to stop its current TS stream coding operation, and the flow control unit of the communication terminal B sends a stop real-time coding to the TS coding unit of the communication terminal. The instruction, the TS coding unit of the communication terminal B stops working; otherwise, step 4 is performed; step 4, the flow control unit of the server notifies that the media source unit of the communication subunit A is taken from the media buffer unit of the communication subunit C Data, the TS stream access count of the communication terminal C is incremented by 1; the data is started from the I frame data of the TS stream, and if it is MPEG4 compression code, it can be vopt-coding-type or group-of-vop. — start— code The PT/CN2005/002152 field finds the I frame;

Step 5: The media source unit of the communication subunit A of the server obtains the TS stream data from the media buffer unit of the communication subunit C and transmits the TS stream data to the RTP output unit of the communication subunit;

Step 6: The RTP output unit of the communication subunit A of the server adds an RTP header to the TS stream data according to the RTP session attribute currently established with the communication terminal A, that is, the SSRC field of the RTP header is the communication terminal A. The SSRC, the TimeStam and the SequenceNumber of the communication subunit A are independent of the source of the data source, and the RTP output unit of the communication subunit transmits the TS stream data according to the TimeStamp; Step 7. The RTP input unit of the communication terminal A receives After the TS stream data is transmitted to the TS decoding unit of the communication terminal, the TS decoding unit decodes the real-time audio and video stream data of the communication terminal C, and displays an image and a playback sound.

Thus, the server can smoothly switch the main media source of the communication terminal A from the communication terminal B to the communication terminal C; The media stream switching method according to the present invention is characterized by:

1. Establish an independent RTP/UTCP transmission channel for the terminal, independent of the media source;

2. For the characteristics of real-time streaming, when the audio and video streams are switched, the switching can be completed by selecting different data sources, without changing the TimeStamp, SequenceNumber, and SSRC in the RTP session;

3. Switching to one or more streams to the same media source is the same for media sources.

Of course, the present invention may also have other various embodiments, such as transmitting static image data by using a Transmission Control Protocol (hereinafter referred to as TCP protocol), or using static image data and TS stream data differently. Transmission channel transmission. A person skilled in the art can make various changes and modifications in accordance with the present invention without departing from the spirit and scope of the invention, but the corresponding changes and modifications should be included in the appended claims. The scope of protection.

Claims

Rights request

A multi-party video communication media stream control system, comprising a media stream transmission control path, responsible for transmitting audio and video data and control signals; a communication terminal, responsible for transmitting and receiving the above audio and video data and control signals; and a server, responsible for responding to the above control Signal, providing a media stream service for a video communication terminal, characterized in that

The communication terminal includes: a real-time transmission control protocol unit, a real-time transmission protocol output unit, and a real-time transmission protocol input unit, wherein the real-time transmission protocol output unit is responsible for transmitting audio and video data of the communication terminal to the server corresponding to the communication terminal. a real-time transmission protocol input unit of the communication subunit, the real-time transmission protocol input unit is responsible for receiving audio and video data of other communication terminals from the server;

The real-time stream coding unit is responsible for real-time stream coding of the communication terminal;

The intra-coded frame coding unit is configured to periodically encode the static image of the communication terminal, form the intra-coded frame data to be output to the real-time transmission protocol output unit of the communication terminal, and mark with a specific load type in the real-time transmission protocol packet. Intra-coded frame data of the communication terminal;

a media buffer unit, configured to receive data of a real-time transmission protocol input unit of the communication terminal;

a real-time stream decoding unit, configured to extract a real-time stream from the media buffer unit and decode the same;

The intra-coded frame decoding unit is configured to extract the above-mentioned real-time transmission protocol packet with a specific payload type from the media buffer unit, and decode the intra-coded frame data to form a static image;

a flow control unit, configured to communicate with the flow control unit of the server, to notify the server of the primary media source currently used by the communication terminal or to receive an instruction of the real-time stream coding unit of the communication terminal of the server;

The above servers include: An intra-coded frame storage unit is responsible for storing intra-coded frame data of each of the communication terminals;

a flow control unit, configured to control a link of the audio and video streams of each of the communication terminals; and a communication subunit, configured to establish a communication link with the communication terminal to transmit the audio and video data and the control signal, and is responsible for intraframe coded frame storage to the server The unit outputs the video intraframe encoded frame data, and the media source for inputting the media source data into the communication subunit from the real-time transport protocol unpacking unit of the intra-coded frame storage unit of the server and the other communication sub-units of the server In the unit, wherein the communication subunit includes:

a real-time transmission control protocol transmission unit, a real-time transmission protocol output unit, and a real-time transmission protocol input unit, wherein the real-time transmission protocol output unit is responsible for transmitting audio and video data of the other communication terminals to a communication terminal corresponding to the communication sub-unit, the real-time The transmission protocol input unit is responsible for receiving audio and video data of the communication terminal;

The intra-coded frame extracting unit is configured to extract the intra-coded frame data according to a specific load type of the real-time transport protocol packet and store it in the intra-coded frame storage unit of the server; if the intra-coded frame If the extracting unit does not find the real-time transport protocol packet of the above type, it extracts the intra-coded frame data directly from the data in the real-time transport protocol unpacking unit of the communication sub-unit of the server;

The real-time transmission protocol unpacking unit is configured to remove the real-time transmission protocol header of the received audio and video data after filtering out the data packet required by the intra-frame coding frame extraction unit of the communication sub-unit of the server, and output the same to the real-time transmission protocol header of the received audio and video data. a media buffer unit of the communication subunit; a media buffer unit configured to receive data output by a real-time transport protocol unpacking unit of the communication subunit of the server; and a media source unit responsible for an intra-coded frame storage unit from the server Obtaining audio and video data required by the communication terminal corresponding to the communication subunit in the media buffer unit of the communication subunit corresponding to the other communication terminal;

The real-time transmission protocol output unit of the communication terminal is respectively connected to the real-time stream coding unit and the intra-frame coding frame coding unit of the communication terminal;

Real-time transmission protocol input unit of the above communication terminal and media buffer of the above communication terminal Unit connection

The media buffer unit of the communication terminal is respectively connected to the real-time stream decoding unit and the intra-frame coded frame decoding unit of the communication terminal; and

The intra-coded frame extracting unit of the communication sub-unit of the server is respectively connected to the intra-coded frame storage unit of the server, the real-time transport protocol input unit of the communication sub-unit of the server, and the real-time transport protocol unpacking unit;

The real-time transport protocol unpacking unit of the communication subunit of the server is respectively connected to the real-time transport protocol input unit, the intra-coded frame extracting unit and the media buffer unit of the communication subunit of the server;

The media source units of the communication subunits of the server are respectively connected to a real-time transmission protocol output unit of the communication subunit of the server, an intra-coded frame storage unit of the server, and a flow control unit.

2. The system of claim 1 wherein

The communication terminal marks the intra-coded frame data of the communication terminal by defining a specific load type in the real-time transmission protocol packet, and the real-time transmission protocol packet of the type only includes the intra-coded frame data of the audio and video data of the communication terminal;

The above communication terminal and the server are two-way audio and video stream communication;

When the multi-party video communication is established, the server includes a plurality of the communication subunits, and each of the communication subunits corresponds to each of the communication terminals;

After the multi-party video communication is established, the above communication terminal and the server transmit the audio and video data through the real-time transmission protocol/real-time transmission control protocol transmission channel.

A multi-party video communication media stream control method, characterized in that the method for hybrid transmission of a static image and a real-time stream comprises the following steps:

Step 101: The real-time transmission protocol output/input unit of the communication terminal establishes a pair of data communication links with the real-time transmission protocol input/output unit of the communication sub-unit corresponding to the communication terminal in the server, wherein the static image and the real-time stream are All transmitted through this pair of data communication links; Step 102: The intra-coded frame coding unit of the communication terminal periodically encodes the static image into intra-frame encoded frame data, and transmits the same to the real-time transmission protocol output unit of the communication terminal.

Step 103: If the real-time transmission protocol output unit of the communication terminal receives the intra-coded frame data in the intra-frame coding frame coding unit, package the data into a real-time transmission protocol packet with a specific payload type, and send it immediately; The real-time transmission protocol output unit of the communication terminal receives the data of the real-time stream coding unit of the communication terminal, and then encapsulates it into a real-time transmission protocol packet, and sends it according to a time stamp; wherein the real-time transmission protocol package of the specific load type a real-time transport protocol packet that is different from the above-mentioned real-time stream;

Step 104: If the real-time transport protocol input unit of the communication sub-unit corresponding to each communication terminal in the server receives the real-time transport protocol packet of the specific load type, extract the intra-frame encoded frame data and store the same. Go to the intra-coded frame storage unit in the server; if the real-time stream data is received, the packet header is removed by the real-time transport protocol unpacking unit of the communication subunit, and then stored in the media buffer unit of the communication subunit Inside;

Step 105: The media source unit timing of the communication subunit corresponding to each communication terminal in the server is used to acquire intraframe encoded frame data of other communication terminals from the intraframe encoded frame storage unit, and determined from each communication terminal. Obtaining the real-time stream data in a media buffer unit of a communication subunit corresponding to the main media source;

Step 106: If the real-time transmission protocol output unit of the communication sub-unit corresponding to each communication terminal in the server receives the real-time transmission protocol packet of the specific load type, immediately send; if the real-time flow is received in real time The transport protocol packet is sent according to the above time stamp; '

Step 107: If the real-time transmission protocol input module of the communication terminal receives the real-time transmission protocol packet of the specific payload type, transmit the same to the intra-coded frame decoding unit of the communication terminal to form a static image of another communication terminal; Receiving the real-time transport protocol packet of the real-time stream, transmitting the real-time transport protocol packet to the real-time stream decoding unit of the communication terminal to form a real-time stream of the main media source.

The method according to claim 3, characterized in that the data communication link in the above step 101 is a real-time transmission protocol/real-time transmission control protocol transmission channel.

A multi-party video communication media stream control method, characterized in that the method for selecting a main media source through a still image received by a communication terminal comprises the following steps:

Step 201: When initially starting multi-party video communication, the communication terminal only performs intra-coded frame coding, and the intra-coded frame data is periodically sent to the server through a real-time transmission protocol packet of a specific load type;

Step 202: The server receives the intra-frame encoded frame data of each communication terminal, and stores it in an intra-framed frame storage unit of the server.

Step 203: The communication subunit corresponding to each communication terminal in the server acquires intraframe encoded frame data of other communication terminals from the intraframe encoded frame storage unit and sends the intraframe encoded frame data to the communication terminal;

Step 204: The communication terminal selects a primary media source according to the displayed static image of the other communication terminal, and then the flow control unit of the communication terminal notifies the flow control unit of the server of the primary media source selected by the communication terminal;

Step 205: The server checks whether the primary media source selected by the communication terminal is transmitting a real-time stream, and if not, notifying the flow control unit of the primary media source to start the real-time stream coding unit;

Step 206: The flow control unit of the server sends a control signal to the media source unit of the communication subunit corresponding to the communication terminal to obtain real-time stream data from the media buffer unit of the communication sub-unit corresponding to the main media source.

6. The method of claim 5 wherein said primary media source is another communication terminal different from said communication terminal.

A multi-party video communication media stream control method, characterized in that a method for a communication terminal to switch from a current primary media source to another primary media source comprises the following steps:

Step 301: The flow control unit of the communication terminal 4 notifies the flow control unit of the server to treat it as The former main media source consists of 4, A ₂ , . . . , „switch to ^ ; '

Step 302: The server checks whether the primary media source is transmitting a real-time stream, and if not, notifying the flow control unit of the primary media source to activate a real-time stream encoding unit of the primary media source;

Step 303: The flow control unit of the server sends a control signal to the media source unit of the communication subunit corresponding to the communication terminal, to obtain real-time stream data from the media buffer unit of the communication subunit corresponding to the main media source. .

8. The method of claim 7 wherein

The current main media source 4, A ₂ 4 and the main media source are different communication terminals from the communication terminal A _x ;

The above subscripts x, y and n are different from each other.