CN118264839A - Method, device, medium and equipment for synchronously playing multiple live streams - Google Patents

Abstract

The embodiment of the disclosure provides a method, a device, a medium and equipment for synchronously playing a multi-path live stream. One embodiment of the method comprises the following steps: acquiring at least two paths of live streams, wherein key frames of each path of live stream carry synchronous information, and the synchronous information corresponds to the actual acquisition time of the frames; determining the synchronous information of non-key frames corresponding to the key frames of each path of live stream based on the synchronous information in the key frames of each path of live stream and the display time stamp PTS of each frame; and controlling the playing of each path of live stream based on the synchronous information of the currently played video frames of each path of live stream. According to the embodiment of the disclosure, synchronous playing of the multi-path live stream can be realized by carrying the synchronous information in the key frame.

Description

Method, device, medium and equipment for synchronously playing multiple live streams

Technical Field

The embodiments of the present disclosure relate to the field of video processing technology, and in particular to a method, device, medium and equipment for synchronously playing multiple live streams.

Background technique

With the rapid development of computer technology and Internet technology, videos and live broadcasts have gradually been integrated into people's daily lives, and are quietly changing people's production and lifestyles. In practice, in live broadcasts of concerts and events, there may be scenarios where multiple live streams are played simultaneously. For example, in live broadcasts of events, the event content and commentary content are played simultaneously, or in concerts, the content of multiple cameras is played simultaneously. In such scenarios, multiple live broadcasts need to be able to maintain synchronization, but the timing of freezes and other problems in each live broadcast during playback is different. Without interference, it is basically impossible to ensure that multiple live broadcasts are always synchronized. This will result in a very poor user experience and may even lead to major adverse consequences.

Summary of the invention

The embodiments of the present disclosure describe a method and device for synchronous playback of multiple live streams, which can achieve synchronous playback of multiple live streams by carrying synchronization information in key frames.

According to a first aspect, a method for synchronously playing multiple live streams is provided, the method comprising: obtaining at least two live streams, wherein a key frame of each live stream carries synchronization information, and the synchronization information corresponds to an actual acquisition time of the frame; based on the synchronization information in the key frame of each live stream and a display timestamp PTS of each frame, determining the synchronization information of non-key frames of each live stream corresponding to the key frame; based on the synchronization information of a currently played video frame of each live stream, controlling the playback of each live stream.

In one embodiment, the method further includes: when reading a key frame, caching the mapping relationship between the synchronization information of the key frame and the display timestamp PTS; and the method of determining the synchronization information of the non-key frame video frames corresponding to the key frame of each live stream based on the synchronization information in the key frame of each live stream and the display timestamp PTS of each frame includes: when reading a non-key frame, calculating the synchronization information of the non-key frame based on the cached mapping relationship and the display timestamp PTS of the non-key frame. Thus, the synchronization information of each non-key frame is determined based on the mapping relationship between the pre-cached key frame synchronization information and the display timestamp PTS.

In one embodiment, the at least two live streams include a first live stream and a second live stream, the first live stream is set as a master live stream, and the second live stream is set as a slave live stream; and the controlling the playing of each live stream based on the synchronization information of the currently played video frames of each live stream includes: based on the synchronization information of the currently played video frames of the master live stream and the slave live stream, controlling the playing of the slave live stream to be synchronized with the playing of the master live stream. Thus, by controlling the playing of the slave live stream, the synchronous playing of the slave live stream and the master live stream is achieved.

In one embodiment, the player instance that plays the main live stream is a main player instance, and the player instance that plays the slave live stream is a slave player instance; and, based on the synchronization information of the main live stream and the video frame currently played by the slave live stream, the playback of the slave live stream is controlled to be synchronized with the playback of the main live stream, including: when the first synchronization information of the next video frame to be played by the slave player instance is later than the second synchronization information of the video frame currently played by the main player instance, and the first difference between the first synchronization information and the second synchronization information is less than or equal to the preset first threshold, the slave player instance increases the playback speed; when the first synchronization information is earlier than the second synchronization information, and the second difference between the first synchronization information and the second synchronization information is less than or equal to the preset second threshold, the slave player instance reduces the playback speed. In this way, the playback synchronization of the main live stream and the slave live stream is achieved when the gap between the first synchronization information and the second synchronization information is small.

In one embodiment, the above-mentioned control of playing the above-mentioned slave live stream based on the synchronization information of the video frames currently played by the above-mentioned main live stream and the above-mentioned slave live stream to synchronize with the playing of the above-mentioned main live stream also includes: when the above-mentioned first synchronization information is later than the above-mentioned second synchronization information, and the above-mentioned first difference is greater than the above-mentioned first threshold, or when the above-mentioned first synchronization information is earlier than the above-mentioned second synchronization information, and the above-mentioned second difference is greater than the above-mentioned second threshold, the above-mentioned slave player instance re-pulls the slave live stream. In this way, when the playing gap between the main player instance and the slave player instance is large, synchronization can be quickly achieved.

In one embodiment, the above-mentioned re-pulling the slave live stream includes: re-pulling the slave live stream based on the difference between the above-mentioned first synchronization information and the second synchronization information, thereby making the re-pulled slave live stream more accurate.

In one embodiment, the synchronization information of each live stream is located in the supplemental enhancement information SEI; and the synchronization information in the key frame of each live stream is determined based on the time code generated by the time code generator or the UTC timestamp generated by the NTP server. Thus, the synchronization information of each video frame of each live stream can be written into the SEI of each video frame.

According to the second aspect, a method for synchronous playback of multiple live streams is provided, which is executed by a director station, wherein the method includes: obtaining at least two live streams, wherein each live stream carries the actual acquisition time of each frame; based on the actual acquisition time of each frame in each live stream, determining the synchronization information of each live stream; writing the synchronization information of each live stream into the key frame of each live stream, so as to perform synchronous playback of the at least two live streams.

According to a third aspect, a device for synchronous playback of multiple live streams is provided, comprising: an acquisition unit, configured to acquire at least two live streams, wherein a key frame of each live stream carries synchronization information, and the synchronization information corresponds to an actual acquisition time of the frame; a determination unit, configured to determine, based on the synchronization information in the key frames of each live stream and a display timestamp PTS of each frame, synchronization information of non-key frames of each live stream corresponding to the above key frames; and a control unit, configured to control playback of each live stream based on the synchronization information of a currently played video frame of each live stream.

According to a fourth aspect, a device for synchronous playback of multiple live streams is provided, which is deployed on a director's station, wherein the device includes: a live stream acquisition unit, configured to acquire at least two live streams, wherein each live stream carries the actual acquisition time of each frame; a synchronization information determination unit, configured to determine the synchronization information of each live stream based on the actual acquisition time of each frame in each live stream; and a synchronous playback unit, configured to write the synchronization information of each live stream into a key frame of each live stream, for synchronous playback of the at least two live streams.

According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method as described in any one of the first aspect or the second aspect is implemented.

According to the sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute any one of the methods in the first aspect or the second aspect.

According to the seventh aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of any one of the first aspect or the second aspect is implemented.

According to the method and device for synchronous playback of multiple live streams provided by the embodiments of the present disclosure, at least two live streams can be obtained first, and the key frames of each live stream carry synchronization information, and the synchronization information can correspond to the actual acquisition time of the frame. Then, based on the synchronization information in the key frames of each live stream and the display timestamp PTS of each frame, the synchronization information of the non-key frames corresponding to the key frames of each live stream can be determined. Finally, based on the synchronization information of the currently played video frames of each live stream, the playback of each live stream can be controlled. Therefore, the synchronous playback of multiple live streams can be achieved by carrying synchronization information in the key frames.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram showing an application scenario in which an embodiment of the present disclosure can be applied;

FIG2 is a schematic diagram showing a flow chart of a method for synchronously playing multiple live streams according to an embodiment;

FIG3 is a schematic diagram showing a method of writing SEI information into an I frame of a GOP;

FIG4 is a schematic flow chart showing a method for synchronously playing multiple live streams according to yet another embodiment;

FIG5 is a schematic diagram showing an example of determining synchronization information of non-key frames corresponding to key frames;

FIG6 shows a schematic block diagram of an apparatus for synchronously playing multiple live streams according to an embodiment;

FIG7 shows a schematic block diagram of an apparatus for synchronously playing multiple live streams according to yet another embodiment;

FIG8 shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present application.

Detailed ways

It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

The technical solution provided by the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the inventions. It should also be noted that, for ease of description, only the parts related to the relevant inventions are shown in the accompanying drawings. It should be noted that, in the absence of conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

As mentioned above, in the scenario of playing multiple live streams at the same time, if the multiple live content is not synchronized, it will cause a very poor user experience. In some implementations, in order to achieve the requirement of playing multiple live streams at the same time in the aforementioned scenario, certain requirements can be made for video encoding or packaging. For example, MV-HEVC (Multiview High Efficiency Video Coding) can be used to achieve multi-view synchronous playback, but this method requires the decoder to support MV-HEVC encoding, and MV-HEVC is not currently widely supported.

To this end, the embodiment of the present disclosure provides a method for synchronous playback of multiple live streams, which can achieve synchronous playback of multiple live streams without relying on specific encoding, encapsulation, etc. FIG. 1 shows a schematic diagram of an application scenario in which the embodiment of the present disclosure can be applied. As shown in FIG. 1, in the scenario shown in FIG. 1, first, live stream 1, live stream 2, ... live stream n, a total of n live streams can be input into the director station 101, wherein each live stream carries the actual acquisition time of each frame. The director station 101 can determine the synchronization information of each live stream based on the actual acquisition time of each frame in each live stream. Afterwards, the director station 101 can write the synchronization information of each live stream into the key frame of each live stream, thereby obtaining live stream 1', live stream 2', ... live stream n'. Here, the key frame can refer to an I frame, that is, an intra coded frame (Intra Coded Picture). The player 102 at the user end can pull live stream 1', live stream 2', ... live stream n' through CDN (Content Delivery Network). The player 102 can determine the synchronization information of the non-key frames corresponding to the key frames of each live stream based on the synchronization information in the key frames of each live stream and the presentation time stamp PTS (Presentation Time Stamp) of each frame. Finally, the player 102 can control the playback of each live stream based on the synchronization information of the currently played video frame of each live stream, thereby achieving synchronous playback of n live streams. Therefore, by carrying synchronization information in key frames, synchronous playback of multiple live streams is achieved.

2, which shows a schematic flow chart of a method for synchronously playing multiple live streams according to an embodiment. The method can be executed by a director station, where the director station can be a local director station or a cloud director station.

As shown in FIG. 2 , the method for synchronously playing multiple live streams may include the following steps 201 to 203, specifically:

Step 201, obtaining at least two live streams.

In this embodiment, several live streams acquired or rebroadcasted by a device may be input into a director's station, and each live stream may carry the actual acquisition time of each video frame.

As an example, in a scenario where a time code generator can be used, multiple video capture devices corresponding to multiple live streams can be connected to the same time code generator, which can provide accurate time code signals that can be received and used by multiple video capture devices. Here, the time code can be the time code recorded by the video capture device for each image when capturing image signals.

As another example, in a scenario where a time code generator cannot be used, the streaming devices corresponding to multiple live streams can be connected to the same local area network, and an NTP (Network Time Protocol) server can be established in the local area network. The NTP server can provide accurate UTC (Coordinated Universal Time) timestamps for streaming devices in the local area network by synchronizing with the international standard time source, and uniformly synchronize the time for streaming devices in the local area network.

Step 202: Determine the synchronization information of each live stream based on the actual acquisition time of each frame in each live stream.

In this embodiment, the director station can synchronize the various live streams. Specifically, the synchronization information of each live stream can be determined based on the actual acquisition time of each frame in each live stream. For example, the director station can generate synchronization information based on the time code or UTC timestamp of each live stream. For example, the format of the time code and the UTC timestamp can be converted to a predetermined format, and the information obtained after the format conversion is used as synchronization information. In addition, the director can also observe the differences between the various live streams through the director station, and add corresponding delays to each live stream according to the differences. As a result, the director station can adjust the synchronization information of each live stream according to the corresponding delay added by the director to each live stream, and complete the preliminary synchronization.

Step 203: write the synchronization information of each live stream into the key frame of each live stream, so as to perform synchronous playback of at least two live streams.

In this embodiment, the director can write the synchronization information of each video frame of each live stream into the SEI (Supplemental Enhancement Information) of each video frame to obtain a synchronization SEI. Here, SEI is text data inserted into the audio and video stream to convey additional information, which can be accurately synchronized with the relevant audio and video content and reach the client through the transmission link. In practice, in order to avoid the destruction of the synchronization SEI sequence in subsequent transcoding and other links as much as possible (for example, during the transcoding process, if there is a change in frame rate, the one-to-one correspondence between the synchronization SEI and the video frame may be destroyed, which in turn causes the synchronization SEI timing abnormality), when the director writes the synchronization SEI, the synchronization SEI can be written only into the key frame for synchronous playback of multiple live streams.

As an example, a key frame may refer to an I frame in a GOP (Group of Pictures), that is, an intra coded frame (Intra Coded Picture). Here, GOP may represent a group of continuous pictures. In MPEG (Moving Picture Experts Group) encoding, GOP is a basic access unit, consisting of continuous picture frames, which may include I frames, P frames and B frames. Among them, I frame may represent an intra coded frame, which can be understood as a complete picture. P frame may be used to represent a forward prediction frame, which records the change relative to the I frame and represents the difference with the previous frame. B frame may represent a bidirectional interpolation frame, which represents the difference between the previous and next frames. As shown in FIG. 3, FIG. 3 shows a schematic diagram of writing SEI information in the I frame of GOP. In the example shown in FIG. 3, the image in the video needs to be processed by acquisition, encoding, directing and transcoding. When the director writes the synchronization SEI, the synchronization SEI is only written in the I frame. It can be understood that due to limited pages, only two GOPs, GOP N and GOP N+1, are shown in the example shown in FIG. In practice, the number of GOPs included in a video is much greater than two.

Through the method described in FIG. 2 , multiple live streams with synchronization information written into key frames can be played synchronously on the client based on the synchronization information written into the key frames.

Please refer to Figure 4, which shows a schematic flow chart of a method for synchronously playing multiple live streams according to another embodiment. The method can be executed by a client. As shown in Figure 4, the method for synchronously playing multiple live streams can include the following steps 401-403, specifically:

Step 401, obtaining at least two live streams.

In this embodiment, the client can pull at least two live streams through the network, wherein the key frame of each live stream carries synchronization information, and the synchronization information corresponds to the actual acquisition time of the key frame. Here, the synchronization information carried in the key frame of each live stream can be written by the method described in Figure 2. The synchronization information of each live stream can be located in the supplementary enhancement information SEI. The synchronization information in the key frame of each live stream can be determined based on the time code generated by the time code generator or the UTC timestamp generated by the NTP server.

Step 402: Based on the synchronization information in the key frames of each live stream and the display timestamp PTS of each frame, the synchronization information of the non-key frames corresponding to the key frames of each live stream is determined.

In this embodiment, when writing synchronization information to the live stream, in order to avoid the synchronization SEI disorder problem that may be introduced by transcoding, the synchronization SEI is only written in the I frame of the live stream. In order to achieve frame-level synchronization, it is necessary to determine the synchronization information of the non-key frames corresponding to the key frames. Here, the non-key frame corresponding to a certain key frame may refer to a non-key frame in the same GOP as the key frame. For example, assuming that a certain GOP includes I frames, P frames, B frames, B frames and B frames in sequence, then the non-key frames corresponding to the key frames (i.e., I frames) in the GOP are P frames, B frames, B frames and B frames. In this example, the synchronization information of the non-key frames corresponding to the key frames of each live stream can be determined based on the synchronization information in the key frames of each live stream and the display timestamp PTS of each frame.

In some implementations, when a player instance used by a client to play a live stream reads a key frame, a mapping relationship between the synchronization information of the key frame and the display timestamp PTS can be cached. The mapping relationship can refer to the corresponding relationship between the synchronization information of the key frame and the display timestamp PTS. Based on this, the above step 402 can be specifically performed as follows: when reading a non-key frame, the synchronization information of the non-key frame is calculated based on the cached mapping relationship and the display timestamp PTS of the non-key frame.

For example, assuming that an I frame belongs to GOP N, the synchronization information corresponding to the I frame is T, and the display timestamp PTS corresponding to the I frame is t, then when the player instance reads the I frame, it can cache the correspondence between T and t. When reading a non-key frame corresponding to the I frame, the synchronization information of the non-key frame can be calculated based on the correspondence between T and t and the display timestamp PTS of the non-key frame. For example, the synchronization information T' of the non-key frame can be calculated using the following formula:

T’=T+(t’-t).

Among them, t' can represent the display timestamp PTS of the non-key frame. As shown in Figure 5, Figure 5 shows a schematic diagram of an example for determining the synchronization information of the non-key frame corresponding to the key frame. In the example shown in Figure 5, two GOPs, GOP N and GOP N+1, are included. The non-key frames in each GOP can determine the synchronization information according to the mapping relationship between the key frames corresponding to them. It can be understood that the number of GOPs shown in Figure 5, the number of frames contained in each GOP, and the synchronization information and display timestamp PTS corresponding to each frame, etc., are only schematic, and the number of GOPs in the video, the number of frames contained in each GOP, and the synchronization information and display timestamp PTS corresponding to each frame are limited.

In practice, the acquisition time of the mapping relationship between the synchronization information of the key frame and the display timestamp PTS is earlier than the acquisition time of the non-key frame, and due to the existence of the cache, multiple sets of mapping relationships may be cached in the player instance at the same time. For example, when obtaining GOP N, the data of GOP N+1 and GOP N+2 may have been cached, so that the mapping relationship of the key frames in GOP N+1 and GOP N+2 has been obtained. Therefore, when calculating the synchronization information of a non-key frame in GOP N, it is necessary to find the key frame corresponding to the non-key frame. For example, in the cached mapping relationships, all mapping relationships with a display timestamp PTS less than the display timestamp PTS of the non-key frame can be used as the target mapping relationship, and the key frame corresponding to the mapping relationship with the largest display timestamp PTS among all the target mapping relationships can be used as the key frame corresponding to the non-key frame.

Through this implementation, the synchronization information of each non-key frame can be determined based on the mapping relationship between the synchronization information of the pre-cached key frame and the display timestamp PTS.

Step 403: Control the playback of each live stream based on the synchronization information of the currently played video frame of each live stream.

In this embodiment, if multiple live streams need to be played simultaneously in the same interface, multiple player instances are usually required, and each player instance can independently load and play one live stream, thereby achieving the effect of displaying multiple live streams simultaneously on the same interface. In order to achieve synchronization of multiple live streams, the playback of each live stream can be controlled based on the synchronization information of the currently played video frame of each live stream. For example, multiple live streams can be registered to the same synchronization group, and one of the multiple live streams is selected as the main live stream, and the other live streams are synchronized with the main live stream by adjusting the playback speed.

For example, assuming that at least two live streams include a first live stream and a second live stream, wherein the first live stream is set as a master live stream and the second live stream is set as a slave live stream. Based on this, the above step 403 may specifically include: based on synchronization information of video frames currently played by the master live stream and the slave live stream, controlling the playback of the slave live stream to be synchronized with the playback of the master live stream.

In this example, the playback progress of the main live stream is used as a reference for the playback progress of the slave live stream, and the playback speed is not adjusted by itself. Thus, by controlling the playback of the slave live stream, the synchronous playback of the slave live stream and the main live stream is achieved.

In some implementations, assuming that the player instance playing the main live stream is the main player instance, and the player instance playing the slave live stream is the slave player instance, then the above-mentioned control of the playback of the slave live stream based on the synchronization information of the video frames currently played by the main live stream and the slave live stream may specifically include the following cases a), b) and c), specifically:

a) When the first synchronization information of the next video frame to be played by the slave player instance is later than the second synchronization information of the video frame currently played by the main player instance, and the first difference between the first synchronization information and the second synchronization information is less than or equal to a first threshold, the slave player instance increases the playback speed.

For example, assuming that the synchronization information of the video frame currently played by the main player instance is 10s (seconds), and the synchronization information of the next video frame to be played by the slave player instance is 8s, the playback progress of the slave player instance is later than the playback progress of the main player instance. Here, the playback progress may refer to the synchronization information corresponding to the frame played by the player instance at a certain moment. For example, if the synchronization information of the video frame currently played by the main player instance is 10s, then the current playback progress of the main player instance is 10s. If the playback progress difference between the two is less than or equal to the preset catch-up threshold, the playback speed of the slave player instance can be increased. For example, the playback speed is increased according to a preset ratio. For another example, the playback speed is increased according to the size of the playback progress difference between the two, and the larger the difference, the greater the increase in the playback speed. Here, the catch-up threshold may be a manually set value.

b) when the first synchronization information is earlier than the second synchronization information and a second difference between the first synchronization information and the second synchronization information is less than or equal to a preset second threshold, reducing the playback speed from the player instance.

For example, assuming that the synchronization information of the video frame currently played by the main player instance is 10s, and the synchronization information of the next video frame to be played by the slave player instance is 12s, the playback progress of the slave player instance is earlier than that of the main player instance. If the difference in playback progress between the two is less than or equal to the slow-motion threshold, the playback speed of the slave player instance can be reduced. For example, the playback speed can be reduced according to a preset ratio. For another example, the playback speed can be reduced according to the size of the difference in playback progress between the two. The larger the difference, the greater the reduction in playback speed.

From the above description, it can be seen that a) and b) achieve synchronization of playback of the main live stream and the slave live stream when the playback gap between the first synchronization information and the second synchronization information is small.

c) When the first synchronization information is later than the second synchronization information and the first difference is greater than the first threshold, or when the first synchronization information is earlier than the second synchronization information and the second difference is greater than the second threshold, re-pull the live stream from the player instance.

Here, when the difference in playback progress between the slave player instance and the master player instance is greater than the catch-up threshold or the slow-motion threshold, the slave player instance can achieve fast synchronization by re-pulling the slave live stream. For example, the latest slave live stream can be re-pulled from the network. For another example, the slave live stream can be pulled based on the second synchronization information of the video frame currently played by the master player instance, for example, the video frame whose synchronization information in the slave live stream is after the second synchronization information is pulled. Through this implementation method, synchronization can be quickly achieved when the playback gap between the master player instance and the slave player instance is large.

In some implementations, the synchronization information of the slave live stream re-pulled from the player instance may be too late. For example, assume that the synchronization information of the video frame currently played by the main player instance is 10s, and the synchronization information of the video frame of the slave live stream re-pulled from the player instance is 15s. Obviously, the synchronization information of the live stream pulled from the player instance is too late, and direct playback will cause it to be out of sync with the main live stream. In order to avoid the synchronization information of the slave live stream pulled from the player being too late, the slave live stream can be re-pulled based on the difference between the first synchronization information and the second synchronization information. For example, if the first synchronization information is later than the second synchronization information, and the time difference t1 between the two is greater than the first threshold, the slave live stream is re-pulled by offsetting t1 backward from the player instance. For another example, if the first synchronization information is earlier than the second synchronization information, and the time difference t2 between the two is greater than the second threshold, the slave live stream is re-pulled by offsetting t2 forward from the player. Through this implementation, fast synchronization can be achieved when the playback gap between the master player instance and the slave player instance is large, and the slave live stream re-pulled from the slave player instance can be made more accurate.

Reviewing the above process, in the above embodiment of the present disclosure, at least two live streams can be obtained first, and the key frames of each live stream carry synchronization information, which can correspond to the actual acquisition time of the frame. Then, based on the synchronization information in the key frames of each live stream and the display timestamp PTS of each frame, the synchronization information of the non-key frames corresponding to the key frames of each live stream can be determined. Finally, based on the synchronization information of the currently played video frames of each live stream, the playback of each live stream can be controlled. Therefore, by carrying synchronization information in the key frames, the synchronous playback of multiple live streams can be achieved.

According to another embodiment, a device for synchronously playing multiple live streams is provided. The device for synchronously playing multiple live streams can be deployed on a client.

FIG6 shows a schematic block diagram of an apparatus for synchronously playing multiple live streams according to an embodiment. The apparatus shown in FIG6 is used to execute the method shown in FIG4. As shown in FIG6, the apparatus 600 for synchronously playing multiple live streams includes: an acquisition unit 601, configured to acquire at least two live streams, wherein the key frames of each live stream carry synchronization information, and the synchronization information corresponds to the actual acquisition time of the frame; a determination unit 602, configured to determine the synchronization information of the non-key frames corresponding to the above key frames of each live stream based on the synchronization information in the key frames of each live stream and the display timestamp PTS of each frame; and a control unit 603, configured to control the playback of each live stream based on the synchronization information of the currently played video frame of each live stream.

In some optional implementations of this embodiment, the above-mentioned device 600 also includes: a cache unit (not shown in the figure), configured to cache the mapping relationship between the synchronization information of the key frame and the display timestamp PTS when reading the key frame; and the determination unit 602 is further configured to: when reading the non-key frame, calculate the synchronization information of the non-key frame based on the cached mapping relationship and the display timestamp PTS of the non-key frame.

In some optional implementations of this embodiment, the at least two live streams include a first live stream and a second live stream, the first live stream is set as a main live stream, and the second live stream is set as a slave live stream; and the control unit 603 includes: a playback control unit (not shown in the figure), configured to control the playback of the slave live stream based on synchronization information of the main live stream and the video frames currently played by the slave live stream, so as to be synchronized with the playback of the main live stream.

In some optional implementations of the present embodiment, the player instance that plays the main live stream is a master player instance, and the player instance that plays the slave live stream is a slave player instance; and the playback control unit is further configured as: when the first synchronization information of the next video frame to be played by the slave player instance is later than the second synchronization information of the video frame currently played by the main player instance, and the first difference between the first synchronization information and the second synchronization information is less than or equal to a preset first threshold, the slave player instance increases the playback speed; when the first synchronization information is earlier than the second synchronization information, and the second difference between the first synchronization information and the second synchronization information is less than or equal to a preset second threshold, the slave player instance reduces the playback speed.

In some optional implementations of the present embodiment, the playback control unit is further configured to: when the first synchronization information is later than the second synchronization information and the first difference is greater than the first threshold, or when the first synchronization information is earlier than the second synchronization information and the second difference is greater than the second threshold, the live stream is re-pulled from the player instance.

In some optional implementations of this embodiment, the above-mentioned re-pulling the slave live stream includes: re-pulling the slave live stream based on the difference between the above-mentioned first synchronization information and the second synchronization information.

In some optional implementations of this embodiment, the synchronization information of each live stream is located in the supplemental enhancement information SEI; and the synchronization information in the key frame of each live stream is determined based on the time code generated by the time code generator or the UTC timestamp generated by the NTP server.

According to another embodiment, another device for synchronously playing multiple live streams is provided. The device for synchronously playing multiple live streams can be deployed in a director station.

FIG7 shows a schematic block diagram of an apparatus for synchronously playing multiple live streams according to another embodiment. The apparatus shown in FIG7 is used to execute the method shown in FIG2. As shown in FIG7, the apparatus 700 for synchronously playing multiple live streams includes: a live stream acquisition unit 701, configured to acquire at least two live streams, wherein each live stream carries the actual acquisition time of each frame; a synchronization information determination unit 702, configured to determine the synchronization information of each live stream based on the actual acquisition time of each frame in each live stream; and a synchronous playback unit 703, configured to write the synchronization information of each live stream into the key frame of each live stream, so as to perform the synchronous playback of the at least two live streams.

The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

According to another aspect of the embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in FIG. 2 or 4 .

According to yet another embodiment, there is further provided an electronic device, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method described in FIG. 2 or 4 is implemented.

The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible, or may be advantageous.

Referring to Fig. 8, a schematic diagram of the structure of an electronic device 800 suitable for implementing the embodiments of the present application is shown. The electronic device shown in Fig. 8 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

As shown in Figure 8, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.

Typically, the following devices can be connected to the I/O interface 805: input devices 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 can allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 8 shows an electronic device 800 with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or provided alternatively. Each box shown in FIG. 8 may represent one device, or may represent multiple devices as needed.

In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method provided by the present disclosure.

It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least two live streams, wherein the key frames of each live stream carry synchronization information, and the synchronization information corresponds to the actual acquisition time of the frame; based on the synchronization information in the key frames of each live stream and the display timestamp PTS of each frame, determines the synchronization information of the non-key frames of each live stream corresponding to the key frames; based on the synchronization information of the currently played video frames of each live stream, controls the playback of each live stream.

Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or electronic device. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims (12)

1. A method for synchronously playing multiple live streams, the method comprising:

Acquiring at least two paths of live streams, wherein key frames of each path of live stream carry synchronous information, and the synchronous information corresponds to the actual acquisition time of the frames;

Determining the synchronous information of non-key frames corresponding to the key frames of each path of live stream based on the synchronous information in the key frames of each path of live stream and the display time stamp PTS of each frame;

and controlling the playing of each path of live stream based on the synchronous information of the currently played video frames of each path of live stream.

2. The method of claim 1, wherein the method further comprises:

when a key frame is read, the mapping relation between the synchronous information of the key frame and the display time stamp PTS is cached; and

The determining the synchronization information of the video frames of the non-key frames of each live stream corresponding to the key frames based on the synchronization information in the key frames of each live stream and the display time stamp PTS of each frame comprises the following steps:

when the non-key frame is read, the synchronous information of the non-key frame is calculated based on the cached mapping relation and the display time stamp PTS of the non-key frame.

3. The method of claim 1, wherein the at least two live streams include a first live stream and a second live stream, the first live stream being set as a master live stream and the second live stream being set as a slave live stream; and controlling playing of each path of live stream based on the synchronization information of the currently played video frame of each path of live stream, including:

And controlling the playing of the slave live stream based on the synchronous information of the video frames currently played by the master live stream and the slave live stream so as to be synchronous with the playing of the master live stream.

4. A method according to claim 3, wherein the player instance playing the master live stream is a master player instance and the player instance playing the slave live stream is a slave player instance; and controlling playing of the slave live stream based on the synchronization information of the video frames currently played by the master live stream and the slave live stream so as to synchronize with the playing of the master live stream, wherein the method comprises the following steps:

When the first synchronization information of the next video frame to be played by the slave player instance is later than the second synchronization information of the video frame currently played by the master player instance, and the first difference value between the first synchronization information and the second synchronization information is smaller than or equal to a preset first threshold value, the slave player instance increases the playing speed;

And when the first synchronization information is earlier than the second synchronization information and the second difference value between the first synchronization information and the second synchronization information is smaller than or equal to a preset second threshold value, the slave player instance reduces the playing speed.

5. The method of claim 4, wherein the controlling the playing of the slave live stream to synchronize with the playing of the master live stream based on synchronization information of video frames currently played by the master live stream and the slave live stream, further comprises:

The slave player instance re-pulls the slave live stream if the first synchronization information is later than the second synchronization information and the first difference is greater than the first threshold, or if the first synchronization information is earlier than the second synchronization information and the second difference is greater than the second threshold.

6. The method of claim 5, wherein the re-pulling the slave live stream comprises:

And re-pulling the slave live stream based on the difference value of the first synchronization information and the second synchronization information.

7. The method of claim 1, wherein the synchronization information of each live stream is located in supplemental enhancement information SEI; and the synchronization information in the key frames of each live stream is determined based on the time code generated by the time code generator or the UTC time stamp generated by the NTP server.

8. A method of synchronized playback of a multi-channel live stream, the method performed by a director, wherein the method comprises:

acquiring at least two paths of live broadcast streams, wherein each path of live broadcast stream carries the actual acquisition time of each frame;

determining synchronous information of each path of live broadcast stream based on actual acquisition time of each frame in each path of live broadcast stream;

and writing the synchronous information of each path of live broadcast stream into the key frames of each path of live broadcast stream so as to synchronously play the at least two paths of live broadcast streams.

9. A device for synchronously playing multiple live streams, comprising:

the acquisition unit is configured to acquire at least two paths of live streams, wherein key frames of each path of live stream carry synchronous information, and the synchronous information corresponds to the actual acquisition time of the frames;

A determining unit configured to determine synchronization information of non-key frames corresponding to key frames of each live stream based on synchronization information in key frames of each live stream and display time stamps PTS of each frame;

and the control unit is configured to control the playing of each path of live stream based on the synchronous information of the currently played video frames of each path of live stream.

10. A device for synchronously playing multiple paths of live streams, deployed on a broadcast guiding station, wherein the device comprises:

the live broadcast stream acquisition unit is configured to acquire at least two paths of live broadcast streams, wherein each path of live broadcast stream carries the actual acquisition time of each frame;

the synchronization information determining unit is configured to determine synchronization information of each path of live stream based on actual acquisition time of each frame in each path of live stream;

And the synchronous playing unit is configured to write the synchronous information of each path of live stream into the key frames of each path of live stream so as to be used for synchronously playing the at least two paths of live streams.

11. A computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of any of claims 1-8.

12. An electronic device comprising a memory having executable code stored therein and a processor, which when executing the executable code, implements the method of any of claims 1-8.