CN100452705C - Embedded Linux multimedia signal acquisition and processing apparatus and its transmission method - Google Patents

Abstract

The present invention relates to an embedded Linux multimedia signal collection processing device and a relative transmission method. The device comprises an embedded server, a plurality of client terminals and a cloud platform, wherein the embedded server is connected with the cloud platform via an RS485 bus; the embedded server and the client terminals carry out interactive control and data transmission via the RTP/RTCP, and use the RSVP to realize QoS resource preparation. The present invention adopts safety strategy to prevent the server from being attacked and ensure the safe control to the server and the safety of flow media in the network transmission. The network self-adaptive strategy of the client terminals leads the device to realize the balance of the bandwidth occupation and the replay acceptability in a limit bandwidth network. The present invention can be applied to the remote monitoring field, and especially, the present invention realizes safe, stable and reliable real-time interactive acoustic-virtual monitoring at unattended areas and remote areas with adverse environment.

Description

An embedded Linux multimedia signal acquisition and processing device and transmission method thereof

technical field

The invention belongs to the field of multimedia signal acquisition and processing, and in particular relates to an embedded Linux multimedia signal acquisition and processing device and a transmission method using network transmission.

Background technique

With the rapid development of computer technology and Internet technology, embedded systems have become the new darling of the IT industry, and the development of network multimedia systems under the embedded platform will have broad prospects for development. Streaming media technology is the product of the combination of network and multimedia. Its international standards and basic research are becoming the focus of close attention of the industry and scientific research institutions. Now typical streaming media systems include Microsoft Windows Media, RealSystem, Apple QuickTime, etc., but the size of these software is more than tens of megabytes, which is not suitable for combining with embedded systems, and requires high-performance PC server support, and requires high. However, the current simple network monitoring system has the following limitations: the security of information cannot be guaranteed, there is no QoS, the code rate cannot be adjusted adaptively to the network conditions, and it does not provide multi-channel media synchronization, and it cannot provide camera control .

Contents of the invention

The purpose of the present invention is to provide an embedded Linux multimedia signal acquisition and processing device, which combines three technologies of multimedia, network streaming media transmission and embedded system, can easily deploy remote audio and video real-time monitoring system, realize remote control, network resources Reservation, variable bit rate compression coding, adaptive network streaming media transmission, and confidentiality of information transmission.

Another object of the present invention is to provide a transmission method of the above-mentioned embedded Linux multimedia signal acquisition and processing device.

Embedded Linux multimedia signal acquisition processing device of the present invention, comprises: an embedded server, several clients and a cloud platform, embedded server and cloud platform are connected by RS485 bus; Real-time transmission control protocol (RTP/RTCP) for interactive control and data transmission, use resource reservation protocol (RSVP) to realize quality of service (QoS) resource reservation; embedded server includes audio and video collector, compression encoder and embedded system , the audio and video collector converts the audio and video signals from analog signals to digital signals, and sends the data to the compression encoder, and the compression encoder obtains the data through MPEG4/H.263/H.264 and G.729 compression encoding and transmits it for embedded systems.

The client includes an audio decoder, a video decoder, a streaming media data receiving buffer, a network communication status estimator and a decision maker for adjusting the code rate, and the streaming media data receiving buffer receives the data of the embedded server and sends the data to the audio Decoder and video decoder, the network communication status estimator obtains the network status statistical parameters from the streaming media data receiving buffer and sends the estimated network communication status parameters to the decision maker for adjusting the bit rate, and the decision maker for adjusting the bit rate controls The embedded server changes the encoding bitrate.

Between the embedded server and the client, the non-connection-oriented user packet protocol (UDP) is used at the transport layer to ensure the real-time transmission and reduce the occupied transmission bandwidth; the RTP/RTCP protocol is used at the application layer to realize streaming media Real-time transmission.

The client can run on different PCs at the same time, and log in to the same embedded server at the same time. When logging in and controlling the server, you need to provide a user name and password. Legal users can log in to the server and receive the RTP media sent by the server. Data packets and application packets (APP) using RTCP control the working status of the server.

The transmission method of above-mentioned embedded Linux multimedia signal acquisition processing device is:

Step 1: When logging in, the client software sends the source description packet of the real-time transport control protocol to the embedded server; the embedded server verifies the user name and IP address according to the access security level, and if the verification fails, the embedded server records the user name and IP address If the same user name or IP address fails to verify multiple times in a short period of time, the user name or IP address will be recorded in the blacklist and the service will be stopped for customers using the user name or IP address. If the verification is successful, the user will be registered Name and IP address to the user list and assign access rights;

The source description packet (SDES) contains the user name, password, encryption public key and resource reservation information of the resource reservation protocol (RSVP) for client login. The access security level of the embedded server is divided into four levels, which are respectively no access control, limited user name, limited IP address, limited user name and IP address. The access authority is divided into two levels, namely monitoring level and control level.

Step 2: After the user logs in to the embedded server, the embedded server starts to send streaming media data to the clients in the user list. If the user requests the server to encrypt the streaming media data sent to it, the server is performing streaming media data When sending, the encryption operation will be performed; the streaming media data receiving buffer of the client receives and processes the streaming media data packets: for the streaming media data packets arriving before the playback time limit, determine its position in the streaming media data receiving buffer, rearrange Streaming media data packets arriving out of order; discarding streaming media data packets arriving after the playback time limit; streaming media data receiving buffer regularly counts network transmission status, including streaming media data packet arrival peak value, arrival average value and loss rate;

Step 3: When transmitting streaming media data, the embedded server generates streaming media data packets in RTP format, and generates media timestamps, which are written into each streaming media data packet; when the client plays back, the audio is decoded The device calculates the global media time and the private media fine-tuning time set by the user. The video decoder uses these two times to calculate the private media timestamp. The video decoder reads the data in the streaming media data buffer with the private media timestamp to realize video Play synchronously with audio;

The method for generating the media time stamp is: read the time of the 64bit embedded system, the high 32bit of the embedded system time is a value in seconds (S), and the low 32bit is a value in microseconds (uS) , take the low 16bit value of the second value and the 16bit value of the 23rd to 8th digit in the middle of the microsecond value, and combine them to form a 32bit media timestamp, which simplifies the calculation of the media timestamp;

In the fourth step, the client calls the network communication status estimator regularly, reads the network status statistics parameters from the streaming media data receiving buffer, and then sends the loopback time test packet of the RTCP APP to obtain the loopback time parameters; the network communication status estimator estimates the network status The remaining bandwidth, the average value of delay jitter, the standard deviation of delay jitter, and the effective bandwidth of streaming media data; the client regularly calls the decision maker for adjusting the code rate, and adaptively sends the APP to control the compression and encoding quality of the embedded server to achieve user Satisfactory acceptability.

The embedded server of the present invention provides the following control interface to the client: 1. encoding quality control; 2. encoding frame rate control; 3. video acquisition format D1/CIF/QCIF switching control; ⑤ RSVP QoS resource reservation service interface. Among them, the above-mentioned first three functions are implemented by the embedded server calling the device driver, the fourth function is realized by the embedded server sending instructions to the PTZ through the RS485 interface, and the fifth function is implemented by the embedded server calling the RSVP API support library to control the background The running RSVP daemon completes.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

(1) the compression encoder of the device of the present invention uses the compression coding of MPEG4/H.263/H.264 and G.729 to realize low code rate compression, which reduces the occupation of bandwidth during network transmission;

(2) The embedded server of the present invention adopts a blacklist security strategy, so that the server is free from attacks; adopts a hierarchical access security strategy to ensure the security of server authorization; adopts a streaming media encryption strategy to ensure the security of streaming media in network transmission ;

(3) The network adaptive strategy of the client end of the present invention has guaranteed the sequential arrival of data packet transmission and the adaptability of data traffic to Ethernet communication conditions, so that the device realizes bandwidth occupation and playback acceptability when performing data transmission in a limited bandwidth network the balance;

(4) The client of the present invention drives video playback synchronously based on the audio playback time, which improves the acceptability of media stream playback, and the simplified media timestamp generation method can reduce the calculation amount of media timestamps and improve the accuracy of synchronization;

(5) The device is simple, easy to deploy, and has low power consumption, and can provide various control interfaces to realize interactive operations.

Description of drawings

Fig. 1 is the structural representation of embedded Linux multimedia signal acquisition processing device of the present invention;

Fig. 2 is the structural representation of embedded server of the present invention;

Fig. 3 embedded server and client functional logic structure and data flow schematic diagram of the present invention;

FIG. 4 is a schematic diagram of the client media synchronization method of the present invention.

Detailed ways

In order to better understand the present invention, the present invention will be further described below in conjunction with the accompanying drawings.

As shown in Fig. 1 and Fig. 2, embedded Linux multimedia signal acquisition processing device of the present invention comprises: an embedded server 1, several clients 2 and a cloud platform, embedded server 1 and cloud platform are connected by RS485 bus ; Embedded server 1 and client 2 conduct interactive control and data transmission through RTP/RTCP, and use RSVP to realize QoS resource reservation; Embedded server 1 includes audio and video collectors, compression encoders and embedded systems, audio and video collectors Convert audio and video signals from analog signals to digital signals, and send the data to the compression encoder. The compression encoder obtains the data through MPEG4/H.263/H.264 and G.729 compression encoding and transmits it to the embedded system.

Client 2 includes an audio decoder, a video decoder, a streaming media data receiving buffer, a network communication status estimator and a decision maker for adjusting the code rate, and the streaming media data receiving buffer receives data from the embedded server 1 and sends the data to the audio Decoder and video decoder, the network communication status estimator obtains the network status statistical parameters from the streaming media data receiving buffer and sends the estimated network communication status parameters to the decision maker for adjusting the bit rate, and the decision maker for adjusting the bit rate controls The embedded server 1 changes the code rate.

Between the embedded server 1 and the client 2, the non-connection-oriented user packet protocol (UDP) is used in the transport layer to ensure the real-time transmission and reduce the occupied transmission bandwidth; the RTP/RTCP protocol is used in the application layer to realize streaming Real-time transmission of media.

The embedded server 1 provides a video input port, an audio input port, a PTZ control interface and an RJ45 port to complete signal conversion from analog to digital, digital signal compression encoding and network transmission. Among them, the input of the video input port is the baseband full TV signal; the input of the audio input port is the audio analog signal; the control interface of the pan/tilt is an industrial control bus port conforming to RS485, the embedded server 1 is the master end, and the pan/tilt is the slave end; The embedded server 1 is connected to the Ethernet through the RJ45 port, and the remote client 2 runs on a PC compatible machine installed with Windows, initiates a connection to the embedded server 1 through RTCP, and can send control information, and can also receive embedded Streaming media data encapsulated by server 1 in RTP format. Both the embedded server 1 and the client 2 communicate with the router providing resource reservation through RSVP, to realize network bandwidth resource reservation, and realize QoS of network communication quality of service guarantee.

Audio and video grabbers convert audio and video signals from analog to digital. Among them, the audio signal is sampled after 8KHz low-pass filtering. The input analog video signal is a baseband full TV signal, which is converted into a digital video signal conforming to the ITU-656 format by the decoder chip of the audio and video collector. The data acquisition buffer control software running in the Field Programmable Gate Array (FPGA) is responsible for organizing and storing these digital signals in RAM, waiting for the upper-layer compression coding software to read.

Compression coding software of MPEG4/H.263/H.264 and G.729 runs in the compression encoder. Both are high-compression-ratio compression coding algorithms with a large amount of calculation, which are completed by a dedicated digital signal processor (DSP). Among them, the algorithm software of MPEG4/H.263/H.264 has been specially optimized for DSP in order to realize real-time compression coding. When the compression encoder is working, for the upper layer, it must operate in accordance with the data reading interface and control interface operating procedures specially designed for the DSP host peripheral interface (HPI), accept the control of the upper layer driver, and send data according to the operating procedures ; For the lower layer, according to the current working state of the compression encoder, control the data collector and read the data.

The embedded system's operating system, device driver, RSVP API support library, and streaming media server program run together on the embedded system based on the ARM core microprocessor to provide streaming media real-time transmission services. The device driver conceals the specific operating characteristics of the compression encoding device for the operating system of the embedded system, and is represented as an abstract general-purpose character device, which provides a system call interface for data reading and device control to the operating system; the operating system manages all The RSVP protocol resource reservation function completed by the streaming media server program is actually by calling the RSVPAPI The interface function of the support library is completed by the RSVP background daemon. The streaming media server program provides streaming media real-time transmission service to the remote client.

In the embedded server of the present invention as shown in Figure 3 and client functional logic structure and data flow direction schematic diagram, streaming media server program 3 is processed by service main control thread 4, RTCP listening thread 5, RTP data sending thread 6, RSVP message Thread 7 composition. The main control thread 4 is responsible for processing and receiving control information (such as console management information) from the local, as well as managing the status of other threads and exchanging various data information. Various messages and data flows in the streaming media server program 3 include: RTCP thread control flow 8 , RTP thread control flow 9 , RSVP control flow 10 , and data flow 11 between RTP threads and RTCP threads. The RTP data sending thread 6 is responsible for obtaining the streaming media data from the device driver, packing the streaming media data and adding the RTP header, sending the generated RTP data stream 13 to the client, and generating various statistical information for the RTCP listening thread 5 to call . The RTCP listening thread 5 obtains RTP statistical data through the data flow 11 between the RTP thread and the RTCP thread, analyzes and periodically generates the RTCP packet flow 12 to send to the remote client, and feeds back to the service main control thread 4, so as to adjust the performance of the RTP service in time . The RSVP message processing thread 7 is responsible for processing the RSVP data stream 14 to ensure the quality of service. Client 15 is composed of client main control thread 16 , RTCP communication thread 17 , estimator and decision maker thread 18 , RTP data receiving thread 19 , RSVP message processing thread 20 and media playing thread 21 . The main control thread 16 of the client is responsible for processing and receiving control information from the local (for example, control management information transmitted by the user image interface), as well as managing the status of other threads and exchanging various data information. Various messages and data flows in the client 15 include: RTCP thread control flow 22, decision maker control RTCP thread message flow 23, decision maker control RTCP receive buffer message flow 24, RSVP thread control message flow 25, play media data flow 27 and media player to estimator message flow 26. The RTP data receiving thread 19 is responsible for receiving streaming media data from the network and accessing its streaming media data receiving buffer. The RTCP communication thread 17 receives the control message of the client main control thread 16, realizes the communication request and the control pan-tilt action of the user to log in and log out; in addition, this thread receives the control message flow of the estimator and the decision maker thread 18, realize A request that controls how the embedded server is encoded. The RSVP message processing thread 20 is responsible for processing the RSVP control data stream 14 to ensure the quality of service.

Server RTCP monitoring thread 5 receives the RTCP communication thread 17 of client 15 and sends the SDES of RTCP to streaming media server program 3, and streaming media server program 3 receives this type of data grouping, and according to the security level of embedded server, judges to Which information of the customer information needs to be verified, for example, the verification of the user name and password or the verification of the IP address of the customer's login is required. The access security level of the embedded server is divided into four levels, which are no access control, restricted user name, restricted IP address, and restricted user name and IP address. If the embedded server works in a mode that requires authentication and the authentication fails, the same user name or IP address that fails three consecutive authentications within one minute will cause the user name or IP address to be recorded in the blacklist and stop the service. If the authentication is successful Just register the user name and IP address to the user list and assign access rights. Access permissions are divided into two levels, monitoring level and control level. Clients with monitoring level can only receive multi-channel streaming media sent by the embedded server. Clients with control level can receive multi-channel streaming media sent by the embedded server. Streaming media can also control the audio and video encoders of the embedded server and the actions of the PTZ.

The RTP data sending thread 6 is responsible for sending the RTP data stream 13 to the client, reading MPEG4/H.263/H.264 data or G.729 data from the device driver, generating streaming media data packets in RTP format, and generating media Timestamp, the media timestamp is written into each streaming media data packet. The method of generating the media time stamp is: read the time of the 64bit embedded system. The high 32bit of the embedded system time is a value in seconds (S), and the low 32bit is a value in microseconds (uS). The low 16-bit value of the second value and the 16-bit value of the 23rd to 8th bits in the middle of the microsecond value are combined to form a 32-bit media time stamp, which simplifies the calculation of the media time stamp. Before sending data to the client, if the client requires the embedded server to encrypt the streaming media data sent to it, the server will perform encryption before streaming the data and then send the data.

The streaming media data receiving buffer of RTP data receiving thread 19 receives streaming media data packets and processes: 1. for the streaming media data packets arriving before the playing time limit, the packet header of each streaming media data packet will contain the sequence number of the packet, and receive a After grouping, first look for the reference packet in the streaming media data receiving buffer, that is, the current oldest packet, and then determine the storage location of the current packet in the streaming media data receiving buffer according to the sequence number of the reference packet. Due to the limited sequence number of the packet number, the streaming media data receiving buffer has to deal with the case where the packet sequence number jumps from the largest number to 0 number, and rearrange the streaming media data packets that arrive out of order; ② For the stream that arrives after the playback time limit Media data is grouped, discarded. In addition, the streaming media data receiving buffer needs to regularly count the network transmission status, including the peak value, average value and loss rate of streaming media data packets.

The estimator and decision maker thread 18 regularly invokes the network communication status estimator to read the network status statistics parameters from the streaming media data receiving buffer, and then sends the loopback time test packet of the RTCP APP to obtain the loopback time parameters. The network communication status estimator estimates the remaining bandwidth of the network, the mean value of the delay jitter, the standard deviation of the delay jitter, and the effective bandwidth of the streaming media data. Afterwards, the estimator and decider thread 18 invokes the decider for adjusting the code rate. According to the remaining network bandwidth calculated by the network communication status estimator, the decision maker adaptively sends APP application packets to control the compression and encoding quality of the embedded server, so as to achieve the acceptability of user satisfaction. When the network communication condition is poor, consider the acceptability of media playback and the bandwidth occupied by transmission from the following aspects: ① reduce the image coding quality; ② reduce the frame rate of image compression coding; ③ change the image format from D1 switches to CIF or QCIF; ④ If the network supports it, start the resource reservation of the RSVP protocol. If the network situation gradually improves, the above-mentioned measures will perform counter-operations. In addition, if the decoding error report message sent by the media playing thread 21 is received, the estimator and decision maker thread 18 will immediately perform a new round of operations.

As shown in FIG. 4 , the schematic diagram of the client media synchronization method of the present invention shows the mechanism of audio and video synchronization using the audio playing time as the driving time of media playback. To play audio and video data from the same synchronization source at the same time, the synchronization between audio and video must also be guaranteed. In addition, since the time stamp of the media source may deviate, the client must also have the function of synchronous fine-tuning to ensure the client's look and feel. Because multimedia systems have higher requirements for sound QoS, people are more sensitive to audio continuity than video continuity, and the playback rate of audio is constant, so the client uses the audio decoding time as the benchmark, and other media playback is based on the audio The playback speed of is used as a reference. The media playing thread 21 calls the audio decoder and the video decoder to perform media playback, and the audio decoder can calculate the global media time, and publish the global media time by means of global variables. The video decoder uses the global media time and the private media fine-tuning time to calculate the private media time stamp, and the video decoder reads the data in the streaming media data buffer with the private media time stamp to realize the synchronous playback of video and audio; Error concealment is performed on successfully decoded frames to improve the acceptability of media stream playback; and a decoding error message is reported to the estimator and decision maker thread 18 to inform it to perform a new round of network adaptive actions.

The device is simple, easy to deploy, and has low power consumption, and can provide various control interfaces to realize interactive operation.

Claims (6)

1. A transmission method for an embedded Linux multimedia signal acquisition and processing device, said device comprising an embedded server, several clients and a cloud platform, said client comprising an audio decoder, a video decoder, a streaming A media data receiving buffer, a network communication status estimator and a decision maker for adjusting code rate are characterized in that the steps of the method are as follows: Step 1: When logging in, the client software sends the source description packet of the real-time transmission control protocol to the embedded server; the embedded server verifies the user name and IP address according to the access security level, and if the verification fails, the embedded server records the user name and IP address If the same user name or IP address fails to verify multiple times in a short period of time, the user name or IP address will be recorded in the blacklist and the service will be stopped for the user name or IP address recorded in the blacklist. If the verification is successful, it will be registered. The user name and IP address are added to the user list and assigned access rights; Step 2: After the user logs in to the embedded server, the embedded server starts to send streaming media data to the clients in the user list. If the user requests the server to encrypt the streaming media data sent to it, the server is performing streaming media data When sending, the encryption operation will be performed; the streaming media data receiving buffer of the client receives and processes the streaming media data packets: for the streaming media data packets arriving before the playback time limit, determine its position in the streaming media data receiving buffer, rearrange Streaming media data packets arriving out of order; discarding streaming media data packets arriving after the playback time limit; streaming media data receiving buffer regularly counting network transmission status; Step 3: When transmitting streaming media data, the embedded server generates streaming media data packets in real-time transport protocol format, and generates media timestamps, which are written into each streaming media data packet; when the client plays back, The audio decoder calculates the global media time and the private media fine-tuning time set by the user. The video decoder uses these two times to calculate the private media timestamp. The video decoder reads the data in the streaming media data buffer with the private media timestamp. Realize synchronous playback of video and audio; In the fourth step, the client regularly calls the network communication status estimator, reads the network status statistics parameters from the streaming media data receiving buffer, and then sends the loopback time test packet of the real-time transmission control protocol application packet to obtain the loopback time parameter; the network communication status The estimator estimates the remaining bandwidth of the network, the average value of delay jitter, the standard deviation of delay jitter, and the effective bandwidth of streaming media data; the client regularly calls the decision maker for adjusting the code rate, and adaptively sends the application packet of the real-time transmission control protocol to Controls the compression encoding quality of the embedded server. 2. The transmission method according to claim 1, characterized in that in the first step, the source description packet contains the user name, password, encrypted public key and resource reservation information of the resource reservation protocol for client login. 3. The transmission method according to claim 1, wherein the access security level of the embedded server in the first step is divided into four levels, namely, no access control, restricted user name, restricted IP address, restricted Username and IP address. 4. The transmission method according to claim 1, characterized in that in the first step, the access authority is divided into two levels, namely monitoring level and control level. 5. transmission method according to claim 1, it is characterized in that the method for producing media time stamp in the described 3rd step is: read the time of 64bit embedded system, the high 32bit of this embedded system time is in seconds The value of the unit, the lower 32bit is the value in microseconds, take the lower 16bit value of the value in seconds and the 16bit value in the middle of the value in microseconds from the 23rd to the 8th, and combine them , which becomes a 32bit media timestamp. 6. The transmission method according to claim 1, characterized in that the network transmission status is represented by the arrival peak value, the arrival average value and the loss rate of the streaming media data packets.

Images