JP2008527472A - How to process multimedia streams - Google Patents

Abstract

  Methods, machine readable media, systems and devices are disclosed. In one embodiment, the method determines whether one or more components comprising the multimedia stream require real-time processing, at least one of the one or more components performs real-time processing. If needed, including assigning one or more components that require real-time processing to one or more real-time processes and assigning any remaining components to one or more non-real-time processes .

Description

  The present invention relates to processing multimedia streams. More particularly, the present invention relates to intelligent selection of algorithmic processes for processing multimedia streams.

  Interactions with multimedia streams, such as music downloads, video recordings, and telephone conversations, are prevalent in society. The types of electronic devices designed to send and receive these streams are as varied as the format of the streams themselves. When a user of one electronic device transmits information to a user of another electronic device, the formats need to be compatible with each other in order for communication to take place effectively. In many cases, the formats used by the two electronic devices are not interchangeable, and therefore an intermediate device must be used to convert information that travels between the two devices. Devices called transcoders convert one type of multimedia stream to another, so people can easily communicate with each other and use these different devices and different multimedia stream formats as they are. Transcoding is currently used in a variety of different scenarios, such as for unidirectional, bidirectional, or multidirectional streams.

  Unidirectional transcoding is used in situations where one device transmits a multimedia stream to another device. Video messaging and video streaming are examples of unidirectional transcoding. FIG. 1A shows an example of a transaction that may require unidirectional transcoding of a multimedia stream. In this example, video stream 100 is transmitted from first computer 102 to second computer 104. Video from the first computer 102 can have many different possibilities, including MPEG2, MPEG4, GSM (global system for mobile communications) or AMR (adaptive multi-rate), among others. Encoded into one of the following formats. The second computer 104 receives and decodes the streaming video and displays it on the video monitor.

  Another example of unidirectional transcoding is voicemail. A person may want to record a message in his voicemail system to inform the caller that he is not in the office. The person may wish to play the message to decide whether he wants to save the message or record again. The person You can speak through a 711 phone, and the voicemail system may record the message in a WAV file. Again, a transcoding step is necessary.

  Bidirectional transcoding is used when two devices send streams to each other, potentially during a video call or audio (eg, telephone) call. FIG. 1B shows an example of a bi-directional video call that may require bi-directional transcoding of a multimedia stream. In this example, video stream 110 is transmitted from first computer 112 to second computer 114 and from second computer 114 to first computer 110.

  Multi-directional transcoding is used when more than two devices send streams to each other, for example during a three-party video conference. FIG. 1C shows an example of a three-party video conference that may require multi-directional transcoding. In this example, multiple computers send and receive video streams between each other, just as in the example of FIG. 1B. In this example, a first computer 122, a second computer 124, and a portable mobile computer / cell phone 126 send and receive a stream 120 between each other.

  In many cases, each of the computers shown in FIGS. 1A, 1B, and 1C may encode and decode multimedia streams in different formats. For example, the two computers (130, 132) of FIG. 1C may encode and decode a stream in MPEG4 format, but the mobile computer / cell phone 134 may or may not be able to encode or decode MPEG4. Only GSM format streams are encoded and decoded. In this case, a transcoding device may be used to intercept the stream coming from each of the three devices and convert the stream into a unique format for each destination device.

  A video stream is just one of many possible examples of where a transcoder may be needed. For example, if a person using a landline phone calls a friend on a mobile phone, there is a necessary truss coding step. Standard landline phones are generally G.P. It operates using the 711 format. Friends answering calls typically have GSM, AMR or other mobile phone voice format technologies. Thus, in order for two friends to hear each other, a transcoding step is required while they are speaking.

  The processing power required for many of these transcoding examples varies greatly. Some examples require real-time transcoding, while others do not. “Real-time” refers to the approximate speed required for the transcoder to transcode and output the multimedia stream at the same rate that the untranscoded multimedia stream reaches the transcoder. . Therefore, transcoding an audio stream in real time requires transcoding audio information at a rate at which a person actually speaks. Thus, two friends talking to each other over landline and mobile phone lines are talking to each other in real time and have a form that is transcoded in real time because audio delay cannot be tolerated. There must be. Three-party video conferencing not only has three streams that need to be transcoded in real time, but the stream contains much more information per packet than just pure audio component packets. Since it consists of packets, it requires more processing power.

  Currently, real-time transcoding systems typically use very powerful processors, such as servers with dedicated digital signal processors, to ensure that any form of information in a packet can be transcoded in real time. . In many cases this amount of processing power is not necessary. The person who records the office absence message and then examines the record is not operating in real time. There is a delay in the range of multiple seconds between when a person speaks into the phone and when he hears the recorded voice while examining.

  Some transcoding systems perform transcoding in a non-real-time offline manner. A user on a personal computer can convert a VHS tape to an AVI file for storage on a DVD. Finally, there are implementations in which no transcoding is performed, for example a stream of raw video, audio or audio format can be transmitted over the network. This type of stream is not efficient because the raw stream is not compressed and requires a great deal of network bandwidth.

  The present invention has been shown by way of example and is not limited by the figures of the accompanying drawings. In the drawings, like reference numbers indicate like elements.

  An embodiment of an effective method for processing a multimedia stream is disclosed. In the following description, a number of specific details are set forth. However, it will be understood that embodiments may be practiced without these specific details. In other instances, well-known elements, applications and techniques will not be discussed in detail so as not to obscure the present invention.

  FIG. 2 is a block diagram of one embodiment of a computer system that includes a multimedia stream processing management system (MSPMS). The computer system includes a processor 200, a memory controller hub (MCH) 202, and an I / O controller hub (ICH) 210. MCH 202 and ICH 210 include a chipset. The processor 200 is coupled to the MCH 202 via a host bus. MCH 202 is coupled to system memory 204. In different embodiments, the system memory may be synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), Rambus DRAM (RDRAM: Rambus). DRAM), or one of many other forms of main system memory.

  In one embodiment, the computer system includes an MSPMS 206 that is stored in the system memory 204 and that is operated / executed by the processor 200. In different embodiments, the MSPMS 206 is associated with the speed required for processing (eg, real time or non-real time), each of the different components including the stream (eg, audio component, audio component, video component, data component and control component). Multiple management decisions regarding multimedia stream processing may be controlled, such as priority and assignment of specific devices that actually process the multimedia stream. In different embodiments, there are many other possible processing devices that actually process the multimedia stream, especially digital signal processors, network servers, remote desktop computers, or diagrams. 2 may include the computer system (including processor 200).

  The MCH 202 is also coupled to the graphics module 208. The ICH 210 is coupled to the hard drive 212 and the I / O bus 214. The I / O bus 214 is coupled to a network interface card that interfaces with the network. In different embodiments, the network can be a local area network, a public telephone system, the Internet, a wide area network, a wireless network, or any other network that can deliver data between computing devices.

  FIG. 3 illustrates one embodiment of a multimedia stream processing management system (MSPMS). In this embodiment, the MSPMS processing logic 300 is located between the multimedia stream source 302 and the destination 318. The source 302 generates and encodes the multimedia stream and sends it to the destination device 318 over the network. A stream is composed of a plurality of consecutive packets.

  Each packet may include multiple components. Each component of the packet consists of some type of information about the stream. For example, a packet for an audio / video stream can include an audio component and a separate video component. Different examples are specifically mentioned, although there are also data components that send instructions in the stream, general audio components including music and background noise information (for specific audio information), and other types of control information. And other components that also make up the multimedia stream, such as control components that may include security information, authentication information, and quality of service (QOS) information. Each component in the packet contains a predetermined amount of information about that component. The amount of information per component per packet is usually measured in time. For example, the audio component in individual packets of a multimedia stream can consist of 100 milliseconds of recorded audio. Apart from the component information, each packet includes header information, which includes address information to allow the packet to be routed to the correct destination.

  Returning to FIG. 3, for example, source 302 is sending a video stream in MPEG 4 format to a destination 318 that requires MPEG 2 format, which must then be transcoded before arriving at destination 318. In this embodiment, MSPMS processing logic 300 intercepts the stream. The stream enters the stream unwrapper unit 304. The stream unwrapper unit 304 unwraps the stream into its individual components. Individual components of the stream are routed to the process allocation unit 306. In the current example, the video component is unwrapped and sent to the process allocation unit 306. In one embodiment, the header address information is sent directly from the stream unwrapper unit 304 to the stream wrapper unit 316.

  The process allocation unit 306 then assigns each of the one or more stream components to processing logic that includes one or more algorithm processes (308, 310, 312, 314). An “algorithm process” is a method or process that can modify one or more components of a multimedia stream in some way. Examples of algorithmic processes include, among other things, a method for transcoding multimedia stream components, a method for excluding certain components of a multimedia stream, a method for delaying processing of components of a multimedia stream, A method for buffering a real-time transport protocol (RTP) stream is included.

  Often, more than one algorithmic process must be used to complete a particular task. For example, the user may wish to leave a voicemail. The user calls the telephone number and the voicemail system answers the call. When the user leaves a message, the first algorithm process may buffer the actual voice data coming through the RTP stream, and the second algorithm process may either The dial tone may be monitored to detect when the button is pressed (dual-tone multi-frequency (DTMF) detection), and the third algorithm process is more The RTP stream may be transcoded into a format that is easy to store. In this case, the RTP stream buffer and DTMF detection algorithm process must be operated in real time to generate a valid user interaction, otherwise voicemail will not be useful and transcoding The process simply retrieves the information from the buffer when it is available (on an available basis). Thus, the RTP and DTMF algorithm processes may have higher priority than the transcoding process, and those processes are managed accordingly by the MSPMS.

  In one embodiment, prioritization of algorithm processes may be performed based on the importance of components with respect to user perception. In the above example, voicemail user interaction prioritizes the ability to maintain a clear record of the user's voice. RTP stream buffering is therefore more than transcoding the stream to a different format because removing any part of the RTP stream results in loss of information and makes the user's voice inaudible in some or all of the messages. Is obviously much more important. Thus, in one embodiment, there are N algorithm processes, each of which may perform processing of stream components in different ways. In one embodiment, each algorithm process prioritizes the processing of components transmitted to it based on the importance of each component. Thus, in this embodiment, if the process assignment unit 306 assigns more than one component to a single algorithm prioritization process, the two or more components are processed in order of their priority. In different embodiments, the priority may be the network transmission rate, the relative importance of the different components to allow the user to understand the stream, the bandwidth consumption of the components in relation to other components, or any number of It can be determined based on an algorithm that takes into account factors, such as other factors.

  For example, in video conferencing, if video components are not available, audio communication is still a viable alternative, but if audio components are not available, the efficiency of the communication is simply accompanied by a video supply and dramatically Decrease. In another example, the video component in a multi-component packet multimedia stream typically consumes the most bandwidth and computing power, so the algorithm process can simply transcode the video in real time. Or it may be able to transcode all of the other components (probably audio, voice, data and control components) in real time, so four components take precedence over just one component Sometimes. In addition, if resources are not available to process all components in real time or even in a delayed (non-real time) manner, another option is to exclude a certain percentage of lower priority components It is. Thus, if the video component is of the lowest priority, the decision to assign the component to a particular algorithm process that processes only every other (or every second, every third, etc.) packetized component is a process. Can be performed by the allocation unit 306, which results in a lower frame rate.

  In another embodiment, two or more of the N algorithm processes execute on a single computer system. In this embodiment, each of the two or more components also has its own priority level assigned. Thus, in one example, if the multimedia stream has a video component and an audio component, each assigned to a separate algorithm process by the process assignment unit 306, the audio component process It may have a higher operational priority than the process. In this case, if the computer system does not have sufficient resources to run both processes simultaneously, the video component process will process the audio component of one or more multimedia stream packets. Will be delayed while completing. In another embodiment, each of the N algorithm processes is executed on its own thread in the local computer system. In this embodiment, the local computer system may assign each thread a specific priority that matches the priority of the component assigned to each process thread.

  In another embodiment, two or more of the N algorithm processes are executed on two or more separate computer systems. In this embodiment, process allocation unit 306 may use one or more of the algorithmic processes that run on the computer system with the most available resources. In another embodiment, a component may be assigned to more than one algorithm process. In this embodiment, the process allocation unit 306 divides the information in a single component in the stream into multiple parts and assigns each part to a separate algorithm process. In one embodiment, the process allocation unit 306 alternately transmits component information between the two algorithm processes at the per-packet level (ie, the component of the first packet is transmitted to the algorithm process 1 and the second Component of the second packet is sent to algorithm process 2, the component of the third packet is sent to algorithm process 3, etc.).

  In the current example, process allocation unit 306 allocates video components to an algorithmic process that transcodes from MPEG4 to MPEG2. As the video component of each packet is processed by one or more algorithmic processes, the video component (ie, a new MPEG2 format video stream) is sent to the stream wrapper unit 316. In one embodiment, the stream wrapper unit 316 encapsulates the individual components back into a packetized form that is sent to the destination. In one embodiment, the stream wrapper unit 316 provides information such as the destination address of each component received by the stream wrapper unit 316 as the output of one or more algorithm processes, the stream unwrapper unit 304. Get directly from. In another embodiment, the stream wrapper unit 316 receives information such as component-based priority information and component-based packet combination information directly from the process allocation unit 306. Per-component priority information allows the stream wrapper unit 316 to prioritize the wrapping and transmission of different components arriving from one or more of the algorithm processes. Component-by-component packet combination information allows the stream wrapper unit 316 to incorporate multiple components into a single packet, or individual components to separate packets for reasons such as higher priority. Allows to be wrapped and transmitted.

  In the current example, stream wrapper unit 316 receives the newly transcoded MPEG2 video component, wraps the component again with destination address information and any other relevant information, and sends the stream to destination 318. To do.

  In one embodiment, when deciding which algorithm process to assign a given component to, the process assignment unit 306 includes, among other information, the network of each computer system on which the algorithm process is executed, among other things. It takes into account information including connectivity and throughput, the amount of free resources available to each computer system, and the priority of each algorithm process within that respective computer system. In this embodiment, the process allocation unit 306 receives information about each computer system from the algorithmic processes that are executed within each computer system (dashed arrows in FIG. 3). Using computer system information about each algorithm process, the process allocation unit 306 (and generally MSPMS) can use the in-stream to minimize or eliminate any perceived performance degradation of the stream. Can be determined as efficiently as possible. Therefore, as indicated above, MSPMS can process the components of the stream in real-time, non-real-time, or not at all, depending on the component priority and available system resources. It may not be processed.

  FIGS. 4-9 illustrate multiple examples of MSPMS processing logic decisions for processing multimedia streams with video and audio components. In these examples, this function indicates that each component in the stream requires transcoding, but in different embodiments, partial transcoding is acceptable for some or all of the video stream. In some cases, it may be acceptable that no transcoding is performed. In addition, the processing of stream components includes many other functions other than just transcoding, such as RTP stream buffering, DTMF detection, or any one of many other multimedia stream processing functions. obtain. Accordingly, the transcoding embodiments of FIGS. 4-9 are merely exemplary embodiments, and they can be used in place of any number of other processes performed with a multimedia stream. Further, although these examples describe a single packet in the stream, embodiments potentially include repeating the process for all packets in the stream.

  FIG. 4 illustrates one embodiment of a method for transcoding multimedia stream packets using a single algorithm process. In this embodiment, the source 500 sends a multimedia stream packet 502 over the network. Packet 502 includes a header 504 with all routing and address information regarding the destination of the packet. In this embodiment, the packet includes a video component 506 and an audio component 508. In one embodiment, the video component includes one video frame (ie, a still image) and the audio component includes one audio frame (ie, 100 milliseconds of audio information). In one embodiment, the header 404 is removed and a packet 410 containing only video and audio components is sent to the algorithm process 412. Algorithm process 412 transcodes the information in each component and outputs a new packet containing only the transcoded video and audio components. The header 404 can be reattached to the transcoded packet to bypass the algorithm process 414 and generate a new packet 416 that is sent to the destination 418. In this embodiment, a single algorithm process 412 can transcode (in real-time) a multimedia stream of packets having video and audio components. In one embodiment, the header 404 contains information provided by the stream wrapper unit (FIGS. 3, 316) from the stream unwrapper unit (FIGS. 3, 304) and the process allocation unit (FIG. 3, 306). Modified based on. In another embodiment, the header 404 is not removed from the packet and the packet is sent to the algorithm process with the complete header 404.

  FIG. 5 illustrates one embodiment of a method for transcoding multimedia stream packets using multiple algorithm processes. In this embodiment, the source 500 sends a multimedia stream packet 502 over the network. Packet 502 includes a video component and an audio component (as described above in FIG. 4). In one embodiment, the header is removed and each different component in the packet is separated. Video component 504 is sent to algorithm process # 1 (508) for video transcoding, and audio component 506 is sent to algorithm process # 2 (510) for audio transcoding. In different embodiments, algorithm processes # 1 (508) and # 2 (510) may be located on the same computing device or on different computing devices. When algorithm processes # 1 (508) and # 2 (510) complete transcoding of their respective video and audio components, the header that bypassed the transcoding process (512, 514) was transcoded. Separate video and audio component packets (516, 518) may be reattached. Video component packet 516 and audio component packet 518 arrive at destination 520 separately. In one embodiment, the header is based on information provided by the stream wrapper unit (FIGS. 3, 316) from the stream unwrapper unit (FIGS. 3, 304) and the process allocation unit (FIG. 3, 306). Will be corrected. In another embodiment, the header is not removed from the packet and the packet is sent to the algorithm process with the header complete. Thus, the algorithm process itself can modify the header.

  FIG. 6 illustrates another embodiment of a method for transcoding multimedia stream packets using multiple algorithm processes. In this embodiment, the source 600 sends a multimedia stream packet 602 over the network. Packet 602 includes a video component and an audio component (as described above in FIG. 4). The header is removed and different components in the packet are separated. Video component 604 is sent to algorithm process # 1 (608) for video transcoding, and audio component 606 is sent to algorithm process # 2 (610) for audio transcoding. When algorithm processes # 1 (608) and # 2 (610) complete their respective video and audio component transcoding, they reattach the video and audio components in multi-component packet 614. The header that bypassed the transcoding process is reattached to the multi-component packet 614 and the packet is sent to the destination 616. In this embodiment, if any transcoded component packet has to wait more than a predetermined amount of time for the corresponding packet from the other algorithm process, the packet 614 To minimize any bottlenecks in the process, the stream wrapper unit (FIGS. 3, 316) is simply sent with one component to the destination. In one embodiment, the header is based on information provided by the stream wrapper unit (FIGS. 3, 316) from the stream unwrapper unit (FIGS. 3, 304) and the process allocation unit (FIG. 3, 306). Will be corrected. In another embodiment, the header is not removed from the packet and the packet is sent to the algorithm process with the header complete. Thus, the algorithm process itself can modify the header.

  FIG. 7 illustrates one embodiment of a method for transcoding multimedia stream packets using one algorithm process and one buffer. In this embodiment, the source 700 sends a multimedia stream packet 702 over the network. Packet 702 includes video and audio components (as described above in FIG. 4). The header is removed and different components in the packet are separated. In this embodiment, audio component 704 takes precedence over video component 706, so audio component 704 is sent directly to algorithm process 710, and video component 706 is transmitted by algorithm process 710 to transcode audio component 704. Sent to buffer 708 to wait for completion. When the transcoding process of audio component 704 is complete, buffer 708 releases the stored video component 706 to algorithm process 710 for transcoding. When the transcoding of the audio or video component is complete, the transcoded component is reattached to the header (712, 714) and the individual video or audio component packet (716, 718) is sent to the destination 720. In one embodiment, the header is based on information provided by the stream wrapper unit (FIGS. 3, 316) from the stream unwrapper unit (FIGS. 3, 304) and the process allocation unit (FIG. 3, 306). Will be corrected. In another embodiment, the header is not removed from the packet and the packet is sent to the algorithm process with the header complete. Thus, the algorithm process itself can modify the header.

  FIG. 8 illustrates one embodiment of a method for transcoding multimedia stream packets using one algorithm process and multiple buffers. In this embodiment, the source 800 sends a multimedia stream packet 802 over the network. Packet 802 includes a video component and an audio component (as described above in FIG. 4). The header is removed and different components in the packet are separated. In this embodiment, there is only one algorithm process 812. Furthermore, the priority of the components can be changed in real time based on the specific situation. In this case, the priority of the video component and the priority of the audio component may increase or decrease relative to each other during the transcoding operation. Thus, two buffers are used to allow any component to have the ability to wait for the algorithm process 812 to finish transcoding other components.

  In this embodiment, video component 804 is sent to buffer # 1 (808) and audio component 806 is sent to buffer # 2 (810). In this embodiment, algorithm process 812 takes a single component one at a time from either buffer # 1 (808) or buffer # 2 (810). The decision as to which buffer to take depends on the priority assigned to the individual components waiting in each buffer. When the algorithm process 812 finishes transcoding individual components, the transcoded components are reattached to the header (814 or 816) and the individual video or audio component packets (818 or 820) are sent to the destination 822. Sent.

  In one embodiment, the header is based on information provided by the stream wrapper unit (FIGS. 3, 316) from the stream unwrapper unit (FIGS. 3, 304) and the process allocation unit (FIG. 3, 306). Will be corrected. In another embodiment, the header is not removed from the packet and the packet is sent to the algorithm process with the header complete. Thus, the algorithm process itself can modify the header.

  FIG. 9 illustrates one embodiment of a method for transcoding multimedia stream packets using multiple algorithm processes and multiple buffers. In this embodiment, source 900 transmits a multimedia stream packet 902 over the network. Packet 902 includes video and audio components (as described above in FIG. 5). The header is removed and different components in the packet are separated. In this embodiment, there are two algorithm processes (912, 914). Furthermore, the priority of the components can be changed in real time based on the specific situation. In one embodiment, algorithm processes # 1 (912) and # 2 (914) are on the same computer system but in separate processes. In this case, the priority of the video transcoding process and the priority of the audio transcoding process increase or decrease not only with respect to each other but also with respect to other processes running on the computer system. Can do.

  For example, a computer system in which both algorithm processes reside will receive a worm from the Internet and a process that executes a virus protection software program may be a video transcoding algorithm process or an audio transcoding algorithm process. Temporarily takes precedence over either. Thus, in this case, each algorithm process itself will allow both components to have the ability to wait for the algorithm process (912, 914) to regain high priority on the system. It has a buffer.

  In this embodiment, video component 904 is sent to buffer # 1 (908) and audio component 906 is sent to buffer # 2 (910). In this embodiment, algorithm process # 1 (912) takes individual video components from buffer # 1 (908), and algorithm process # 2 (914) uses individual audio components from buffer # 2 (910). I take the. When any algorithm process finishes transcoding the individual component, the transcoded component is reattached to its header (916 or 918) and the individual video or audio component packet (920 or 922) is sent to the destination. Sent to 924.

  In one embodiment, the header is based on information provided by the stream wrapper unit (FIGS. 3, 316) from the stream unwrapper unit (FIGS. 3, 304) and the process allocation unit (FIG. 3, 306). Will be corrected. In another embodiment, the header is not removed from the packet and the packet is sent to the algorithm process with the header complete. Thus, the algorithm process itself can modify the header.

  FIG. 10 illustrates one embodiment of an algorithmic process that excludes individual components based on component priority and processing power limitations. In this embodiment, a stream of video components 1000 (ie, video frames) and an audio component stream 1002 (ie, audio frames) are input to an algorithm process 1004. Accordingly, the algorithm process 1004 of this embodiment is similar to the algorithm process shown in FIG. In this embodiment, the algorithm process 1004 does not have the processing power to keep up with every audio frame and every video frame. Thus, in the absence of other systems with algorithmic processes that can perform part of the transcoding operation, certain frames must be excluded to maintain the real-time nature of the stream.

  If the audio component has a higher transcoding priority than the video component, certain video frames must be excluded (ie removed). Thus, in one embodiment, a stream of four video frames 1000 and a stream of four audio frames 1002 are input to an algorithm process 1004. The algorithm process 1004 transcodes all audio frames and sends them out in sequence in the audio output stream 1006. The algorithm process 1004 also uses the remaining processing power to transcode as many video frames as possible and send them in sequence in the video output stream 1008. In this case, the algorithm process 1004 excludes every other video frame in order to maintain the real-time aspect of the stream. Thus, the audio stream is clear and complete and the video stream has its frame rate reduced by half.

  FIG. 11 is a flowchart of one embodiment of a method for assigning components of a multimedia stream to real-time and non-real-time algorithm processes. The method is implemented by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (eg, executed on a general purpose computer system or a dedicated machine), or a combination of both. Referring to FIG. 11, the method begins with processing logic determining whether one or more components comprising a multimedia stream require real-time processing (block 1100). If the processing logic determines that real-time processing is required (block 1102), the processing logic assigns one or more components that require real-time processing to one or more real-time algorithm processes (block 1104). ). Finally, processing logic assigns any remaining components to one or more non-real time algorithm processes (block 1106) and the method is terminated.

  FIG. 12 is a flowchart of another embodiment of a method for assigning components of a multimedia stream to a real-time process. The method is implemented by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (eg, executed on a general purpose computer system or a dedicated machine), or a combination of both. Referring to FIG. 12, the method begins with processing logic unwrapping a packet into its one or more components (block 1200). Processing logic then determines the amount of processing required to process each of the one or more components that require real-time processing within each packet in real time (block 1202). Finally, processing logic assigns each of the one or more components that require real-time processing within each packet to a separate real-time process (block 1204) and the method ends.

  FIG. 13 is a flowchart of one embodiment of a method for excluding components of a multimedia stream from the transcoding process. The method is implemented by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (eg, executed on a general purpose computer system or a dedicated machine), or a combination of both. Referring to FIG. 13, the method begins with processing logic unwrapping a packet into its one or more components (block 1300). Processing logic then determines whether all components of the unwrapped packet can be transcoded in real time (block 1302). If so, the method is terminated. Otherwise, processing logic checks for the first component (block 1304). Once the component is examined, processing logic determines whether the component can be excluded (block 1306).

  The determination of whether a component can be excluded can be made based on a number of factors, such as the processing priority of the component and the number of consecutive component frames excluded. For example, as discussed in FIG. 10, if an algorithm process does not have the processing power to process everything in real time, a decision must be made as to which components to exclude and when to exclude them. I must. Removing video components by half of their standard frame rate by excluding every other frame is more objectionable than removing 15 frames consecutively and then transcoding 15 frames consecutively. Few. The end result is the same number of frames, but the quality of the viewing experience is different for those two separate options.

  Referring to FIG. 13, if a component can be excluded, processing logic excludes the component (block 1308). Processing logic then determines whether all components have been examined (block 1310). If so, the method is terminated. Otherwise, processing logic checks the next component (block 1312). Eventually, all components of the unwrapped packet are examined and the method ends. In another embodiment, determining whether to exclude one or more components from a stream may include a decision regarding a multimedia stream process other than simply transcoding. Thus, transcoding is used as an example and may be replaced with one or more other functions that may be performed on a multimedia stream.

  FIG. 14 is a flowchart of an embodiment of a method for skipping transcoding of a stream. The method is implemented by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (eg, executed on a general purpose computer system or a dedicated machine), or a combination of both. Referring to FIG. 14, the method begins with processing logic determining the format of the stream transmitted from the source device (block 1500). Processing logic then determines the type of stream required by the destination device (block 1502). Processing logic then determines whether the two forms are the same (block 1504). If the formats are not the same, processing logic transcodes the stream (block 1506) and the process ends. Otherwise, if the formats are the same, processing logic does not transcode the stream, but instead sends the stream directly from the source to the destination (block 1508), and the method ends.

  FIG. 15 assigns components of a multimedia stream that do not require real-time processing but that require processing prior to the determined maximum time delay to processes that can be processed within the determined maximum time delay. 2 is a flowchart of an embodiment of a method. The method is implemented by processing logic that may include hardware (circuitry, dedicated logic, etc.), software (eg, executed on a general purpose computer system or a dedicated machine), or a combination of both. Referring to FIG. 15, the method begins with processing logic determining a maximum delay allowed for each component in a packet that does not require real-time processing (block 1500). In different embodiments, the determination of the maximum allowable processing delay depends on the type of transaction in which the multimedia stream is used, the speed of the transmission medium (e.g., the required processing speed for voice mail attached within the email is electronic May be based on multiple factors, including those that are limited to the inherent delay of the mail, as well as other factors. The processing logic then assigns each of the components that do not require real-time processing to a separate process, and each process provides at least the amount of processing necessary to process each respective component within an acceptable delay time. (Block 1502), the method ends.

  In many of the exemplary embodiments described above, the packets were primarily dealing with video and audio components. In other embodiments, the packet includes any one or more of video, audio, voice, data, control, or any other possible packet type. Further, the exemplary embodiments have referred to packets that contain only two components. In other embodiments, a packet may include any number of possible components (ie, one or more components). Finally, in many exemplary embodiments, a single algorithm process was responsible for processing all components of a single type. In other embodiments, more than one algorithm process may be assigned to a single type of component. Thus, in many embodiments, two or more algorithm processes exchange processing roles, and each algorithm process therefore replaces every other packetized component of a single type with every other ( Or every second, every third, etc.).

  Accordingly, embodiments for an effective method for processing a multimedia stream are disclosed. These embodiments have been described with reference to specific exemplary embodiments thereof. However, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments without departing from the broader spirit and scope of the embodiments described herein. It is. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

FIG. 6 illustrates an example of a transaction that may require unidirectional transcoding of a multimedia stream. FIG. 2 illustrates an example of a bidirectional video call that may require bidirectional transcoding of a multimedia stream. FIG. 6 illustrates an example of a three-party video conference that may require multi-directional transcoding. 1 is a block diagram of one embodiment of a computer system with a multimedia stream processing management system (MSPMS). 1 illustrates one embodiment of a multimedia stream processing management system (MSPMS). FIG. FIG. 3 illustrates one embodiment of a method for transcoding multimedia stream packets using a single algorithm process. FIG. 3 illustrates one embodiment of a method for transcoding multimedia stream packets using multiple algorithm processes. FIG. 6 illustrates another embodiment of a method for transcoding multimedia stream packets using multiple algorithm processes. FIG. 6 illustrates one embodiment of a method for transcoding multimedia stream packets using one algorithm process and one buffer. FIG. 6 illustrates one embodiment of a method for transcoding multimedia stream packets using one algorithm process and multiple buffers. FIG. 3 illustrates one embodiment of a method for transcoding multimedia stream packets using multiple algorithm processes and multiple buffers. FIG. 6 illustrates one embodiment of an algorithm process that excludes individual components based on component priority and processing capability limitations. 2 is a flowchart of one embodiment of a method for assigning components of a multimedia stream to real-time and non-real-time algorithm processes. 6 is a flowchart of another embodiment of a method for assigning components of a multimedia stream to a real-time process. 2 is a flowchart of one embodiment of a method for excluding components of a multimedia stream from a transcoding process. 6 is a flowchart of an embodiment of a method for skipping transcoding of a stream. An implementation of a method for assigning a component of a multimedia stream that does not require real-time processing, but that requires processing before a determined maximum time delay, to a process that can be processed within the determined maximum time delay It is a flowchart of a form.

Claims (50)

Determining whether one or more components comprising the multimedia stream require real-time processing;
Assigning said one or more components requiring real-time processing to one or more real-time processes if at least one of said one or more components requires real-time processing;
Assigning any remaining components to one or more non-real-time processes;
A method comprising the steps of:   The method of claim 1, wherein the multimedia stream includes a plurality of packets, each packet having all packetized portions of the one or more components.   The method of claim 2, wherein each of the one or more components includes one or more of video information, audio information, audio information, data information, and control information. Real-time processing
Determining the rate at which the packets are transmitted by the source device of the stream;
Processing each of the one or more components comprising each packet at least at a rate at which the packets are transmitted by the source device of the stream;
The method of claim 2 comprising: Assigning the one or more components of the stream in need of real-time processing to one or more real-time processes comprises:
Unwrapping each packet in the stream into its one or more components;
Determining the amount of processing required to process each of the one or more components that require real-time processing in each packet in real time;
Assigning each of the one or more components requiring real-time processing within each packet to a separate process;
The method of claim 4 comprising:   The method of claim 5, wherein the process includes a thread that is executable on the computing device.   The method of claim 5, further comprising assigning each of the one or more components in each packet to a process on a separate computing device.   4. The method of claim 3, further comprising assigning a priority level to a component, wherein the priority level is calculated based on the importance of the component information. Assigning any remaining components to one or more non-real-time processes includes
Determining the maximum delay allowed for each of the components that do not require real-time processing in each packet;
Assigning each of the components that do not require real-time processing within each packet to a separate process, each process having the amount of processing required to process each component within the allowable delay time. Providing at least a stage, and
The method of claim 8 further comprising:   Sequentially processing a plurality of said components of a packet, the order of the processed components being in order of priority level of the components, with the highest priority level component being processed first and the lowest priority level component 9. The method of claim 8, wherein is processed last.   11. The method of claim 10, further comprising preventing information loss of low priority level packet components by temporarily storing the stream of packets in a storage device in real time immediately after packet generation. . Determining whether at least one component from at least one packet can be excluded from processing if not all of the components of the stream are processable in real time;
If at least one component can be excluded, excluding said component;
The method of claim 8 further comprising:   The method of claim 12, further comprising excluding the component having the lowest priority level.   The method of claim 13, further comprising setting a maximum number of consecutive packets including the same excluded component. Determining the format of the stream transmitted from the source device;
Determining the format of the stream required by the destination device;
If the source device stream format is the same as the required destination device stream format, transmitting the stream directly between the source device and the destination device without processing the stream When,
The method of claim 1 further comprising:   9. The method of claim 8, further comprising assigning two or more components to the same process, wherein the priority of each component determines the order of processing. Determining whether one or more components comprising the multimedia stream require real-time transcoding;
If at least one of the one or more components requires real-time transcoding, assigning the one or more components requiring real-time transcoding to one or more real-time processes When,
Assigning any remaining components to one or more non-real-time processes;
The method of claim 1 further comprising: When executed by a machine on a machine-readable medium having instructions thereon, the machine
Determining whether one or more components comprising the multimedia stream require real-time processing;
Assigning said one or more components requiring real-time processing to one or more real-time processes if at least one of said one or more components requires real-time processing;
Assigning any remaining components to one or more non-real-time processes;
A machine-readable medium, characterized in that a method comprising:   The machine-readable medium of claim 18, wherein the multimedia stream includes a plurality of packets, each packet having all packetized portions of the one or more components.   The machine readable machine of claim 19, wherein each of the one or more components includes a type of information from a list including video information, audio information, audio information, data information, and control information. Medium. Real-time processing
Determining the rate at which the packets are transmitted by the source device of the stream;
Processing each of the one or more components comprising each packet at least at a rate at which the packets are transmitted by the stream source device;
The machine-readable medium of claim 19, comprising: Assigning the one or more components of the stream in need of real-time processing to one or more real-time processes comprises:
Unwrapping each packet in the stream into its one or more components;
Determining the amount of processing required to process each of the one or more components comprising each packet in real time;
Assigning each of the one or more components in each packet to a separate process;
The machine-readable medium of claim 21, comprising:   21. The machine-readable medium of claim 20, further comprising assigning a priority level to a component, wherein the priority level is calculated based on the importance of the component information. Assigning any remaining components to one or more non-real-time processes includes
Determining the maximum delay allowed for each of the components that do not require real-time processing in each packet;
Assigning each of the components that do not require real-time processing in each packet to a separate process, each process processing necessary to process each respective component within the allowable delay time Providing at least an amount;
24. The machine-readable medium of claim 23, further comprising:   Sequentially processing a plurality of said components of a packet, wherein the order of the processed components is in order of priority of components, the highest priority level component is processed first, and the lowest priority level component is 24. The machine readable medium of claim 23, which is processed last. Determining whether at least one component from at least one packet can be excluded from processing if not all of the components of the stream are processable in real time;
If at least one component can be excluded, excluding said component;
24. The machine-readable medium of claim 23, further comprising:   27. The machine-readable medium of claim 26, further comprising excluding the component having the lowest priority level.   28. The machine-readable medium of claim 27, further comprising setting a maximum number of consecutive packets that include the same excluded component. Determining the format of the stream transmitted from the source device;
Determining the format of the stream required by the destination device;
If the source device stream format is the same as the required destination device stream format, transmitting the stream directly between the source device and the destination device without processing the stream When,
The machine-readable medium of claim 18, further comprising: With bus,
A processor coupled to the bus;
A network interface card coupled to the bus;
A memory coupled to the processor when executed by the processor;
Determine whether one or more components comprising the multimedia stream require real-time processing;
Assigning the one or more components requiring real-time processing to one or more real-time processes if at least one of the one or more components requires real-time processing;
Assign any remaining components to one or more non-real-time processes;
Memory adapted to store instructions; and
A system characterized by including.   The system of claim 30, wherein the multimedia stream includes a plurality of packets, each packet having all packetized portions of the one or more components.   32. The system of claim 31, wherein each of the one or more components includes a type of information from a list that includes video information, audio information, audio information, data information, and control information. Real-time processing
Determining the rate at which the packets are transmitted by the source device of the stream;
Processing each of the one or more components comprising each packet at least at a rate at which the packets are transmitted by the stream source device;
35. The system of claim 32, comprising: Assigning the one or more components of the stream in need of real-time processing to one or more real-time processes comprises:
Unwrapping each packet in the stream into its one or more components;
Determining the amount of processing required to process each of the one or more components comprising each packet in real time;
Assigning each of the one or more components in each packet to a separate process;
34. The system of claim 33, comprising:   The system of claim 34, wherein the process includes a thread that is executable on the computing device.   35. The system of claim 34, further comprising assigning each of the one or more components in each packet to a process on a separate computing device.   The system of claim 32, further comprising assigning a priority level to a component, wherein the priority level is calculated based on the importance of the component information.   Sequentially processing a plurality of said components of a packet, wherein the order of the processed components is in order of priority level of the components, the highest priority level component is processed first, and the lowest priority level component 38. The system of claim 37, wherein is processed last.   39. The system of claim 38, further comprising preventing information loss of low priority level packet components by temporarily storing the stream of packets in a storage device in real time immediately after packet generation. . Determining whether at least one component from at least one packet can be excluded from processing if not all of the components of the stream are processable in real time;
If at least one component can be excluded, excluding said component;
40. The system of claim 38, further comprising:   41. The system of claim 40, further comprising excluding the component having the lowest priority level.   42. The system of claim 41, further comprising setting a maximum number of consecutive packets that include the same excluded component. Determining the format of the stream transmitted from the source device;
Determining the format of the stream required by the destination device;
If the source device stream format is the same as the required destination device stream format, transmitting the stream directly between the source device and the destination device without processing the stream When,
32. The system of claim 30, further comprising: Assigning any remaining components to one or more non-real-time processes includes
Determining the maximum delay allowed for each of the components that do not require real-time processing in each packet;
Assigning each of the components that do not require real-time processing in each packet to a separate process, each process requiring the amount of processing required to process each component within the allowable delay time Providing at least
35. The system of claim 34, further comprising: A stream unwrapper unit operable to unwrap the first multimedia stream into its individual components;
A process allocation unit operable to allocate each of the one or more components to one or more processes;
Algorithm processing logic operable to process each of the one or more individual components;
A stream wrapper unit operable to wrap at least one of the processed one or more individual components into a second multimedia stream;
The apparatus characterized by including.   46. The apparatus of claim 45, wherein the process allocation unit is further operable to determine a priority for each component and assign a highest priority process to the component having the highest priority.   46. The process allocation unit is further operable to send information about the priority of wrapping each of the one or more individual components to the stream unwrapper unit. The device described.   The algorithm processing logic is further operable to transmit information regarding resources available to the algorithm processing logic for processing each of the one or more individual components. 45. The apparatus according to 45.   48. The apparatus of claim 47, wherein the stream wrapper unit is further operable to wrap each of the one or more individual components in order of priority of the components.   46. The apparatus of claim 45, wherein the stream wrapper unit is further operable to transmit each of the wrapped one or more components to a destination device.

Images