CN112930687B - Media stream processing method and device, storage medium and program product - Google Patents

Abstract

A media stream processing method and device, storage medium and program product, the method includes the following steps: the front end of the Internet webpage pulls the media stream, acquires the structured information carried in the media stream (S102), then generates a display time stamp of the structured information according to the frame structure of the media stream, and the display time stamp is aligned with the playing time of the media stream (S104), so that the front end of the Internet webpage performs format encapsulation on the media stream, plays the encapsulated media stream by using an H5 player (S106), and further synchronously draws the structured information on the current playing picture according to the display time stamp (S108). The method provides a technical scheme that synchronous drawing of structured information and video frames can be realized without installing plug-ins, eliminates the technical barrier that synchronous drawing cannot be realized without plug-ins, and has higher flexibility.

Description

A media stream processing method and device, storage medium and program product

technical field

The present application relates to the field of media streams, for example, to a media stream processing method and device, a storage medium, and a program product.

Background technique

With the popularity of Internet applications, the data transmitted on the network is not limited to text or graphics, and the transmission of media streams such as sound and video brings new enjoyment to people.

Currently, playing media streams on a browser generally relies on browser controls or plug-ins. Specifically, by installing a plug-in in the browser, and directly pulling the media stream through the plug-in, and decoding the pulled media stream to obtain a frame picture, and then drawing structured information using frame synchronization, the structured information and video are realized. Frames are played synchronously.

However, existing media stream processing methods require plug-ins to be installed in the browser, and some browsers also have restrictions on the version of the plug-in, and if traditional plug-ins are not used, other methods are required to synchronize structured information and video frames draw.

Contents of the invention

Embodiments of the present disclosure provide a media stream processing method and device, a storage medium, and a program product, so as to provide a technical solution for synchronous rendering of structured information and video frames without installing plug-ins, eliminating the need for plug-ins that cannot be realized The technical barrier of synchronous drawing has high flexibility.

An embodiment of the present disclosure provides a media stream processing method, including:

The web front end pulls the media stream, and obtains the structured information carried in the media stream;

The Web front end generates a display timestamp PTS of the structured information according to the frame structure of the media stream, and the display timestamp is aligned with the playing time of the media stream;

The Web front end encapsulates the format of the media stream, and uses the H5 player to play the encapsulated media stream;

The web front end synchronously draws the structured information on the current playing screen according to the display time stamp.

An embodiment of the present disclosure also provides a media stream processing device, including:

An acquisition module, configured to pull media streams and acquire structured information carried in the media streams;

A generating module, configured to generate a display time stamp PTS of the structured information according to the frame structure of the media stream, and the display time stamp is aligned with the playing time of the media stream;

A playback module, configured to encapsulate the media stream in a format, and play the encapsulated media stream with an H5 player;

The drawing module is configured to synchronously draw the structured information on the current playing screen according to the display time stamp.

An embodiment of the present disclosure also provides a computer, including the above media stream processing apparatus.

An embodiment of the present disclosure also provides a computer-readable storage medium, storing computer-executable instructions, where the computer-executable instructions are configured to execute the above-mentioned media stream processing method.

An embodiment of the present disclosure also provides a computer program product, the computer program product includes a computer program stored on a computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, the The computer executes the above media stream processing method.

An embodiment of the present disclosure also provides an electronic device, including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is caused to execute the above media stream processing method.

In the technical solution provided by the embodiments of the present disclosure, the media stream is pulled through the Web front end and the structured information of the media stream is obtained, and then, the frame structure of the media stream is used to generate the display time stamp of the structured information, thereby, using the H5 player When playing the media stream, the aforementioned structured information is drawn synchronously according to the display timestamp. In this way, without installing any plug-ins in the browser, and without decoding frame pictures, the aforementioned structured information and media streams can be drawn synchronously, eliminating the need for no plug-ins However, the technical barrier of synchronous rendering cannot be realized, and it has high flexibility. Moreover, this implementation method occupies a small amount of CPU, and there is no problem of CPU consumption. It can achieve multi-channel video playback and avoid the problem of freezing.

Description of drawings

One or more embodiments are exemplified by the corresponding drawings, and these exemplifications and drawings do not constitute a limitation to the embodiments, and elements with the same reference numerals in the drawings are shown as similar elements, The drawings are not limited to scale and in which:

FIG. 1 is a schematic flowchart of a media stream processing method provided by an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of an interaction flow of a media stream processing method provided by an embodiment of the present disclosure;

FIG. 3 is a schematic flowchart of another media stream processing method provided by an embodiment of the present disclosure;

FIG. 4 is a schematic flowchart of another media stream processing method provided by an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of a media stream processing device provided by an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of a computer provided by an embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to understand the characteristics and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. The attached drawings are only for reference and description, and are not intended to limit the embodiments of the present disclosure. In the following technical description, for purposes of explanation, numerous details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, one or more embodiments may be practiced without these details. In other instances, well-known structures and devices may be shown simplified in order to simplify the drawings.

First of all, in order to facilitate understanding, each noun involved in the embodiments of the present disclosure will be described in detail.

Web (Internet web page) front-end: it may be embodied as a Web front-end processor.

H5: HTML5 is developed by the World Wide Web Consortium (W3C), and its goal is to replace the HTML4. Standards are achieved to match contemporary online demands. When talking about HTML5 in a broad sense, it actually refers to a set of technology combinations including Hyper Text Markup Language (Hyper Text Markup Language) HTML, Cascading Style Sheets (CSS) and JavaScript (a literal scripting language). .

MSE (Media Source Extensions): Media source extensions are a new browser interface (Web API) supported by mainstream browsers such as Chrome, Safari, and Edge. MSE complies with W3C standards and allows JavaScript to dynamically construct <video> and <audio> media streams. It defines objects that allow JavaScript to transmit media stream fragments to an HTMLMedia Element (media unit).

PTS (Presentation Time Stamp): It is the display time stamp. This display time stamp can be used as the basis for the player to play information or media stream. For example, the player can determine when to display the frame corresponding to the display time stamp according to the display time stamp. data.

RTSP (Real Time Streaming Protocol): Real-time streaming protocol, which is an application layer protocol in the Transmission Control Protocol (Transmission Control Protocol, TCP) / Internet Protocol (Internet Protocol, IP) protocol system, developed by Columbia University and Netscape The IETFRFC standard submitted by the company and Real Networks.

RTMP (Real Time Messaging Protocol): Real-time message transmission protocol, which is based on TCP, is a protocol family, including real-time message transmission protocol (Real Time Messaging Protocol, RTMP) basic protocol and RTMPT/RTMPS/RTMPE and other variants. RTMP is a network protocol designed for real-time data communication. It is mainly used for audio, video and data communication between the Flash/AIR (Adobe Integrated Runtime) platform and media streaming/interactive servers that support the RTMP protocol. Software that supports this protocol includes Adobe Media Server/UltrantMedia Server/red5, etc.

FLV (FLASH VIDEO): The FLV media stream format is a video format developed with the introduction of Flash MX technology. Because the files it forms are smaller and the loading speed is faster, it is possible to watch video files on the Internet. Its appearance effectively solves the problem of exporting SWF (shock wave flash, a special design software Flash) after the video file is imported into Flash. special format) file size is huge, and it cannot be used well on the network.

http-flv: flv media stream transmitted based on http protocol.

websocket-flv: flv media stream transmitted based on websocket protocol.

NPAPI (Netscape Plugin Application Programming Interface): Netscape Plugin Application Programming Interface, which is a plug-in interface used by Gecko engine browsers such as Netscape Navigator, Mozilla Suite, Mozilla SeaMonkey, and Mozilla Firefox, and webkit engine browsers such as Apple Safari and Google Chrome .

PPAPI (Pepper Plugin API): There are security risks for NPAPI.

The embodiments of the present disclosure refer to the aforementioned nouns later, and all of them represent the aforementioned meanings, and details are not repeated here.

In view of the foregoing problems existing in the prior art, the embodiments of the present disclosure provide the following solutions: the web front end pulls the media stream and obtains the structured information, and then, by adding a display time stamp to the structured information, the H5 player Synchronously draw structured information when playing media streams to achieve synchronous playback.

An embodiment of the present disclosure provides a method for processing a media stream. Referring to Figure 1, the method includes:

S102. The web front end pulls the media stream, and acquires structured information carried in the media stream.

S104. The web front end generates a display time stamp PTS of the structured information according to the frame structure of the media stream, and aligns the display time stamp with the playing time of the media stream.

S106, the web front end encapsulates the format of the media stream, and uses the H5 player to play the encapsulated media stream;

S108. The web front end synchronously draws the structured information on the current playing screen according to the display time stamp.

In the embodiment of the present disclosure, the media streams involved in the foregoing S102 and S104 are http-flv streams or websocket-flv streams, where http and websocket are protocol names, and flv is a media stream format. The essence of the format encapsulation described in step S106 is to convert the format of the media stream so that the encapsulated media stream can meet the playback requirements of the H5 player. In a possible design, the encapsulated media stream may be in FMP4 format.

Before the web front end pulls the data stream, that is, before step S102 is executed, the streaming media server needs to push the aforementioned media stream, and the algorithm server also needs to calculate the structural information of the media stream.

Specifically, please refer to the interactive schematic diagram shown in FIG. 2. Before performing S102, the method further includes the following steps:

S1011. The streaming media server pulls the real-time streaming protocol RTSP stream.

S1012. The streaming media server decodes the RTSP stream to obtain video frame units.

S1013. The streaming media server sends the video frame unit to the algorithm server.

S1014, the algorithm server processes the video frame unit to obtain the structured information of the RTSP stream.

S1015. The algorithm server sends the structured information to the streaming media server.

S1016. The streaming media server encapsulates the RTSP stream and structured information to obtain a media stream.

S1017. The streaming media server pushes the media stream.

In the embodiments of the present disclosure, there is no special limitation on whether the streaming media server, the algorithm server, and the Web front end are integrated together. The three can be independent, or at least two can be integrated into one device or device.

In addition, the prior art also involves an H5 solution that does not rely on plug-ins to play media streams. Specifically, it transmits low-rate streams in MPEG1 encoding format through websockets, and uses a central processing unit (Central Processing Unit, CPU) software Solve the frame picture and use canvas to draw synchronously. However, this implementation method uses the CPU to softly decode the frame picture, resulting in a high CPU usage, resulting in a limited number of channels for playback, and does not support the mainstream h264 encoding scheme, requiring the streaming media server to convert the encoding to MPEG1.

In contrast, as shown in Figure 2, the technical solution provided by the embodiment of the present disclosure is to decode the RTSP stream by the streaming media server to obtain the video frame unit, without decoding by the Web front end, and there is no occupation of the CPU, so there is no There is a problem of CPU being consumed by performance. Therefore, multi-channel video playback is achieved and the freeze problem is effectively solved.

In the interaction process shown in FIG. 2 , in S1016 , the RTSP stream and the structured information are encapsulated, and the result of the encapsulation is to form the aforementioned http-flv stream or websocket-flv stream in the embodiment of the present disclosure.

Based on the push stream from the streaming media server, the web front-end can pull the media stream from the address where the stream is pushed.

Specifically, the embodiment of the present disclosure provides the following implementation manner of S102:

The web front end pulls the media stream through XHR2/Fetch or websocket.

Among them, XHR2/Fetch or websocket are used as network requests to request binary data from streaming media servers to obtain media streams.

In addition, based on the protocol-format of the media stream encapsulated in S1016 is different, the positions where the structured information is encapsulated in the media stream may be different. In a possible design, if the RTSP stream and the structured information are encapsulated into a media stream conforming to the H.264 specification, such as the http-flv stream, the structured information can be encapsulated into the supplemental enhancement information (Supplemental enhancement) of the http-flv stream. information, SEI) unit. Among them, H.264 is a digital video compression format jointly proposed by the International Organization for Standardization and the International Telecommunication Union.

In contrast, the encapsulation positions of structured information in media streams based on different protocols and formats are different. When the web server executes the acquisition step of structured information, the web front end can obtain the information carried in the media stream according to the protocol and format of the media stream. structured information.

That is, first determine the protocol-format of the media stream, and then determine the encapsulation position of the structured information according to the preset correspondence between the protocol-format and the encapsulation position, and then obtain the structured information at the encapsulation position. For example, in the aforementioned possible design, since the web front end pulls the http-flv stream, it can be determined that the SEI unit in the http-flv stream is used to encapsulate structured information, and therefore, the structured information can be obtained in the SEI unit .

Existing media stream processing methods are implemented with the help of plug-ins, such as NPAPI plug-ins, PPAPI plug-ins, or ADOBEFLASH plug-ins, all need to decode frame pictures after pulling media streams, and use frames to draw structured information synchronously, which not only requires browsing If the plug-in is installed in the browser, the decoding of frame pictures further leads to an increase in the amount of computation and a reduction in processing efficiency.

Based on this, in the embodiment of the present disclosure, since there is no need to install a plug-in in the browser, but to generate a display time stamp for the structured information in the aforementioned acquired media stream, there is no need to decode the media stream into a frame picture, Then, when the H5 player plays the media stream, the structured information can be drawn synchronously.

Its specific implementation process can refer to the process shown in Figure 3. At this time, the step S104 can be:

S1042. According to the frame structure of the media stream, the web front end parses out the SEI unit of the structured data and the video frame unit of the media stream.

S1044. The web front end generates a display time stamp for the SEI unit according to the playing time of the video frame unit, so that the SEI unit is aligned with the video frame unit.

In the embodiment of the present disclosure, the final presentation result is that the media stream and the structured information are played synchronously. Therefore, it is required that the display time stamp of the SEI unit is aligned with the playback time of the video frame unit.

For example, if the video frame unit is played at a time interval of 40ms per frame, when generating the display timestamp of the SEI unit, it can be generated for each SEI unit at an interval of 40ms. The video frame unit corresponding to the SEI unit has the same display time stamp at the playing time.

Based on the aforementioned settings, the embodiment of the present disclosure uses the H5 player to play the media stream, and at the same time, draws structured information above the current playing screen according to the display timestamp generated above to realize synchronous playback.

According to the display timestamp, when synchronously drawing structured information, you can draw a three-dimensional image or a two-dimensional image on the canvas as needed. The canvas is a browser DOM (document object model) object, and the picture it presents The overlay is displayed on the current playing screen. That is to say, compared with the picture played in the H5 player, the picture presented by the canvas is closer to the viewer.

In one possible design, two-dimensional images can be implemented with a canvas. Canvas is a 2D (two-dimensional) drawing protocol, which is rendered pixel by pixel, and 2D image drawing is realized through JavaScript. Therefore, in the preceding process, if the synchronous rendering of structured information is realized through canvas, a canvas element can be added to the H5 player, so that canvas can draw 2D images on the canvas through JavaScript.

Or, in another design, the rendering of three-dimensional images can be realized through the Web GL (Web Graphics Library, Web Graphics Library) protocol. And Web GL is a 3D (three-dimensional) drawing protocol. This drawing protocol eliminates the trouble of developing a special rendering plug-in for web pages. It can be used to create website pages with complex 3D structures, and can even be used to design 3D web games, etc. wait.

When implementing synchronous drawing, the web front end can call the current time of the media stream at a preset frequency, and then perform synchronous drawing of structured information on the canvas based on the display timestamp, and overlay and align the current playing screen. Wherein, the preset frequency can be set as required, for example, it can be set to call time at a frequency of 60 frames per second.

However, in the process of synchronous drawing, considering the possible problem that the playback of the media stream and the structured information may not be synchronized, the embodiment of the present disclosure also provides a possible design as shown in FIG. Deviation correction processing of simplified information ensures synchronous drawing with media stream.

As shown in Figure 4, the method also includes:

S110, the web front end performs deviation correction processing on the drawing of the structured information according to the first frequency, the first frequency is greater than the second frequency, and the second frequency is the frame playback frequency of the media stream.

In the design shown in FIG. 4 , the second frequency is the frame playback frequency of the media stream, and its time interval is the frame interval of the media stream, which is related to the frame structure design of the media stream. While the first frequency is greater than the second frequency, the time interval of the first frequency is smaller than the time interval of the first frequency, that is, smaller than the frame interval of the media stream. That is, deviation correction is performed on the rendering of the structured information at a higher frequency, so that the played media stream and the structured information can be presented synchronously.

In addition, considering the time-consuming calculation of alignment drawing, in the actual implementation, a time interval slightly longer than the specific calculation time can be used for deviation correction processing. Among them, the correction processing can be realized through the MATH.ceil function.

The MATH.ceil function is a function in the lua function (the 1ua function is a function in JavaScript), which consists of: MATH.ceil(x) returns the smallest integer greater than or equal to the parameter x, that is, the floating-point number is rounded up. In the implementation scenario involved in the embodiments of the present disclosure, it can be expressed as: MATH.ceil(video.currentTime/40)*40 is aligned with the PTS, where video.currentTime represents the current playing moment of the media stream.

For example, in a possible implementation scenario, if the frame playback interval of the media stream is 40 ms, 10 ms per frame may be used as the first frequency for deviation correction processing.

It can be seen from the foregoing description that the technical solutions provided by the embodiments of the present disclosure do not need to install plug-ins in the browser, which can be applied to the development needs of future browsers, and realize synchronous drawing by displaying timestamps without the need for media streams to be modified by the Web front end. Decode frame pictures, which can effectively reduce the amount of computation, help shorten the processing time, and reserve more time for deviation correction processing to ensure the synchronous playback and alignment of structured information and media streams.

The embodiment of the present disclosure also provides a media stream processing device. Please refer to FIG. 5, the media stream processing device 500 includes:

An acquisition module 51, configured to pull media streams and acquire structured information carried in the media streams;

The generation module 52 is used to generate the display time stamp PTS of the structured information according to the frame structure of the media stream, and the display time stamp is aligned with the playing time of the media stream;

Playing module 53, is used for carrying out format encapsulation to described media stream, utilizes H5 player to play the media stream after the encapsulation;

The drawing module 54 is configured to synchronously draw structured information on the current playing screen according to the display time stamp.

In a possible design, the generation module 52 is specifically used for:

According to the frame structure of the media stream, parse out the SEI unit of the structured data and the video frame unit of the media stream;

According to the playback moment of the video frame unit, a display timestamp is generated for the SEI unit, so that the SEI unit is aligned with the video frame unit.

In another possible design, the media stream processing device further includes:

The deviation correction module (not shown in FIG. 5 ) is configured to perform deviation correction processing on the rendering of the structured information according to a first frequency, the first frequency is greater than the second frequency, and the second frequency is the frame playback frequency of the media stream.

In another possible design, the drawing module 54 is specifically used for:

Draw a three-dimensional image or a two-dimensional image on the canvas according to the display time stamp, wherein the canvas overlay is displayed on the current playing screen.

In another possible design, the acquisition module 51 is specifically used for:

pull media stream;

Obtain the structured information carried in the media stream according to the protocol and format of the media stream.

In another possible design, the media stream is: http-flv media stream or websocket-flv media stream.

In another possible design, the acquisition module 51 is specifically used for:

Pull media stream through XHR2/Fetch or websocket.

The media stream processing apparatus 500 shown in FIG. 5 is set at the front end of the Web.

In addition, an embodiment of the present disclosure also provides a computer, please refer to FIG. 6 , the computer 600 includes the above-mentioned media stream processing apparatus 500 .

An embodiment of the present disclosure also provides a computer-readable storage medium, storing computer-executable instructions, where the computer-executable instructions are configured to execute the above media stream processing method.

An embodiment of the present disclosure also provides a computer program product, the computer program product includes a computer program stored on a computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, the The computer executes the above media stream processing method.

The above-mentioned computer-readable storage medium may be a transitory computer-readable storage medium, or a non-transitory computer-readable storage medium.

An embodiment of the present disclosure also provides an electronic device, the structure of which is shown in FIG. 7 , the electronic device includes:

At least one processor (processor) 73, take a processor 73 as an example in Fig. 7; And memory (memory) 71, can also comprise communication interface (Communication Interface) 72 and bus. Wherein, the processor 73, the communication interface 72, and the memory 71 can communicate with each other through a bus. Communication interface 72 may be used for information transfer. The processor 73 may invoke logic instructions in the memory 71 to execute the media stream processing method of the foregoing embodiments.

In addition, the logic instructions in the above-mentioned memory 71 can be implemented in the form of software function units and can be stored in a computer-readable storage medium when sold or used as an independent product.

As a computer-readable storage medium, the memory 71 can be used to store software programs and computer-executable programs, such as program instructions/modules corresponding to the methods in the embodiments of the present disclosure. The processor 73 executes the function application and the media stream by running the software programs, instructions and modules stored in the memory 71, that is, implements the media stream processing method in the above method embodiment.

The memory 71 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required by a function; the data storage area may store data created according to the use of the terminal device, and the like. In addition, the memory 71 may include a high-speed random access memory, and may also include a non-volatile memory.

The technical solutions of the embodiments of the present disclosure can be embodied in the form of software products, which are stored in a storage medium and include one or more instructions to make a computer device (which can be a personal computer, a server, or a network equipment, etc.) to perform all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc. A medium that can store program code, or a transitory storage medium.

When used in the present application, although the terms 'first', 'second', etc. may be used in the present application to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without changing the meaning of the description, a first element could be called a second element, and likewise, a second element could be called a first element, as long as all occurrences of "first element" are renamed consistently and all occurrences of "Second component" can be renamed consistently. The first element and the second element are both elements, but may not be the same element.

The terms used in the present application are used to describe the embodiments only and are not used to limit the claims. As used in the examples and description of the claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well unless the context clearly indicates otherwise . Similarly, the term "and/or" as used in this application is meant to include any and all possible combinations of one or more of the associated listed ones. Additionally, when used in this application, the term "comprise" and its variants "comprises" and/or comprising (comprising) etc. refer to stated features, integers, steps, operations, elements, and/or The presence of a component does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groupings of these.

The aspects, implementations, implementations or features of the described embodiments can be used alone or in any combination. Aspects of the described embodiments can be implemented by software, hardware or a combination of hardware and software. The described embodiments may also be embodied by a computer-readable medium storing computer-readable code comprising instructions executable by at least one computing device. The computer readable medium can be associated with any data storage device that can store data that can be read by a computer system. By way of example, computer readable media may include read only memory, random access memory, CD-ROM, HDD, DVD, magnetic tape, optical data storage devices, and the like. The computer readable medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

The above technical description may refer to the accompanying drawings, which form a part hereof, and in which are shown by way of description implementations in accordance with the described embodiments. While these embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments, these embodiments are not limiting; as such, other embodiments can be used and without departing from the scope of the described embodiments Circumstances can also be changed. For example, the order of operations described in the flow diagrams is non-limiting, and thus the order of two or more operations illustrated in and described with respect to the flow diagrams may be changed according to several embodiments. As another example, in several embodiments, one or more operations illustrated in and described with respect to the flowcharts are optional, or may be deleted. Additionally, certain steps or functionality may be added to the disclosed embodiments, or the order of two or more steps permuted. All such variations are considered to be encompassed by the disclosed embodiments as well as the claims.

Additionally, terminology was used in the above technical description to provide a thorough understanding of the described embodiments. However, undue detail is not required to implement the described embodiments. Thus, the foregoing description of the embodiments has been presented for purposes of illustration and description. The embodiments presented in the foregoing description, and examples disclosed according to these embodiments, are provided individually to add context and to facilitate understanding of the described embodiments. The above description is not intended to be exhaustive or to limit the described embodiments to the precise form of the disclosure. Several modifications, alternatives, and variations are possible in light of the above teachings. In some instances, well known process steps have not been described in detail in order not to unnecessarily obscure the described embodiments.

Claims (15)

1. A method for processing a media stream, comprising:

the method comprises the steps that a media stream is pulled by the front end of an Internet webpage Web, and structured information carried in the media stream is obtained;

the Web front end generates a display time stamp PTS of the structured information according to the frame structure of the media stream, wherein the display time stamp is aligned with the playing time of the media stream;

the Web front end performs format encapsulation on the media stream and plays the encapsulated media stream by using an H5 player;

the Web front end synchronously draws the structural information on the current playing picture according to the display time stamp;

the Web front end generates a display time stamp PTS of the structured information according to the frame structure of the media stream, including:

the Web front end analyzes an SEI unit of structured data and a video frame unit of the media stream according to the frame structure of the media stream;

and the Web front end generates the display time stamp for the SEI unit according to the playing time of the video frame unit, so that the display time stamp of the SEI unit is aligned with the playing time of the video frame unit.

2. The method according to claim 1, wherein the method further comprises:

and the Web front end performs deviation rectifying processing on the drawing of the structured information according to a first frequency, wherein the first frequency is larger than a second frequency, and the second frequency is the frame playing frequency of the media stream.

3. The method according to claim 1, wherein the Web front-end synchronously drawing the structured information on the current playing screen according to the display time stamp, including:

and the Web front end draws a three-dimensional image or a two-dimensional image on a canvas according to the display time stamp, wherein the canvas is displayed on the current playing picture in an overlaying manner.

4. The method of claim 1, wherein the Web front end pulls a media stream to obtain structured information carried in the media stream, comprising:

the Web front end pulls the media stream;

and the Web front end acquires the structural information carried in the media stream according to the protocol and the format of the media stream.

5. The method according to claim 1 or 4, wherein the media stream is: http-flv media stream or websocket-flv media stream.

6. The method of claim 5, wherein the Web front end pulling the media stream comprises:

the Web front end pulls the media stream through XHR2/Fetch or websocket.

7. A media stream processing device, comprising:

the acquisition module is used for pulling the media stream and acquiring the structural information carried in the media stream;

the generation module is used for generating a display time stamp PTS of the structured information according to the frame structure of the media stream, wherein the display time stamp is aligned with the playing time of the media stream;

the playing module is used for carrying out format encapsulation on the media stream and playing the encapsulated media stream by using an H5 player;

the drawing module is used for synchronously drawing the structural information on the current playing picture according to the display time stamp;

the generating module is specifically configured to:

analyzing SEI units of structured data and video frame units of the media stream according to the frame structure of the media stream;

and generating the display time stamp for the SEI unit according to the playing time of the video frame unit, so that the display time stamp of the SEI unit is aligned with the playing time of the video frame unit.

8. The apparatus of claim 7, wherein the apparatus further comprises:

and the deviation rectifying module is used for rectifying the drawing of the structured information according to a first frequency, wherein the first frequency is larger than a second frequency, and the second frequency is the frame playing frequency of the media stream.

9. The apparatus of claim 7, wherein the rendering module is specifically configured to:

and drawing a three-dimensional image or a two-dimensional image on a canvas according to the display time stamp, wherein the canvas is displayed in an overlaying manner on the current playing picture.

10. The apparatus of claim 7, wherein the obtaining module is specifically configured to:

pulling the media stream;

and acquiring the structural information carried in the media stream according to the protocol and the format of the media stream.

11. The apparatus according to claim 7 or 10, wherein the media stream is: http-flv media stream or websocket-flv media stream.

12. The apparatus of claim 11, wherein the obtaining module is specifically configured to:

the media stream is pulled through XHR2/Fetch or websocket.

13. A computer comprising the apparatus of any one of claims 7-12.

14. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,,

the memory stores instructions executable by the at least one processor, which when executed by the at least one processor, cause the at least one processor to perform the method of any of claims 1-6.

15. A computer readable storage medium, characterized in that computer executable instructions are stored, said computer executable instructions being arranged to perform the method of any of claims 1-6.

Images