CN101848236A - Real-time data distribution system with distributed network architecture and working method thereof - Google Patents

Abstract

A real-time data distribution system with a distributed network architecture and its working method. The system uses a publish/subscribe communication mechanism to transmit information in the Internet and mobile networks. The system includes: a server subsystem that completes operations such as topic storage and topic matching, and completes from The terminal subsystem receives topics, forwards matching events to subscribers, submits publish/subscribe topics and other operations, the terminal subsystem completes user publish/subscribe information, and the distributed database stores publish/subscribe messages and system information. The system can automatically, quickly and safely provide real-time data transmission services for user subscription information, and can provide publishing/subscribing services for various types of data such as text information, streaming media, and geographic location information, and can provide up to 20 kinds of QoS control parameters , to realize the QoS hierarchical configuration of the application layer, and provide convenience for the function expansion of the data distribution system.

Description

Real-time data distribution system with distributed network architecture and its working method

technical field

The present invention relates to a network system and its working method based on the data distribution service standard DDS to realize the message publishing/subscribing mechanism, to be precise, to a real-time data distribution system with a distributed network architecture based on DDS and its working method , belongs to the technical field of computer application software.

Background technique

The wide application of Internet technology and the rapid development of mobile computing, grid computing and ubiquitous computing platforms pose new challenges to distributed systems: they must meet the requirements of large-scale decentralized control and dynamic changes. This requires a flexible communication and interaction mechanism with dynamic and loose coupling characteristics among the various participants of the system.

The Publish/Subscribe (Publish/Subscribe) system composed of three entities, the information publisher, the information subscriber and the information agent that transmits information among them, can adapt to this mechanism.

See Figure 1 to illustrate the working principle of the publish/subscribe mechanism: the publisher sends information or events to the information agent, that is, the publish/subscribe middleware, and the subscriber sends registration information and subscription conditions to the publish/subscribe middleware, indicating You can also unsubscribe at any time if you are interested in which information or events in the system. The publish/subscribe middleware screens all the input information according to the subscription condition, and transmits the interested information to all interested subscribers in a timely and reliable manner. The matching algorithm is responsible for efficiently finding all the subscription conditions that match the given information or event; while the routing algorithm is responsible for selecting the appropriate path to transmit the information or event from the publisher to the subscriber. The transmission of information enables loose coupling between the publisher and the subscriber, and the publish/subscribe system can adapt to the intermittent connection between data distribution applications and the network. Compared with the traditional message passing communication model, the publish/subscribe communication mechanism has the characteristics of asynchronous and multi-point communication, which completely decouples the communication participants in space, time and control flow, and can well meet the needs of large-scale distributed systems. The need for loose communication.

The object management organization OMG is responsible for formulating computer software standards, and has already formulated many standards including CORBA, UML and MDA. In June 2004, OMG formulated the DDS (Data Distributed Service) standard for information distribution services. This standard uses UML language to describe services, clearly defines the general application layer port of information distribution services, and uses network middleware technology to meet the communication requirements of real-time communication systems.

DDS (Data Distributed Service) information distribution service is a portable middleware technology that uses publish/subscribe mechanism as the main technical means to provide real-time information transmission. Real-time transmission of inter-control information, application API and underlying resource scheduling. The advantages of DDS are: the communication mechanism based on publish/subscribe is simple, reliable and flexible, it can support the automatic discovery mechanism of endpoint applications, it has the function of self-adaptation and dynamic expansion, and it can also set redundant nodes to improve system stability and data transmission. Stable, high bandwidth utilization, support one-to-one, one-to-many, many-to-one and many-to-many communications; in addition, there are many QoS configuration parameters, so that managers can control the transmission of each piece of data in the system. Therefore, publish/subscribe system technology has a good application prospect.

At present, the research on information distribution technology and system has become a hot spot at home and abroad. The early topic-based publish/subscribe system has already had many mature products widely used in many fields such as banking, securities, and informationization of manufacturing enterprises. However, the publish/subscribe system for large-scale distributed computing has not yet found a more influential application case, mainly because the key technology is not yet mature.

For example, there are already several mature publish/subscribe models based on the DDS standard of the OMG organization in the world: CORBA event service, TIB/Rendezvous, JMS, Gryphon, SIENA, JEDI, etc., but their common disadvantage is that they are not applied Effective support for QoS.

Among them, CORBA event service is based on remote method call or "object-centric" design, without application-level QoS support, and is only suitable for point-to-point communication, not suitable for multiple distributed real-time applications. The key to the collaboration model of the TIB/Rendezvous based collaboration system is through subject addressing. JMS has two message models, point-to-point and publish/subscribe, which can provide reliable message transmission, transaction and message filtering mechanisms, suitable for large-scale, data-centric networks; but lack of application-level QoS policies, not suitable for real-time Demanding emergency mission system. JEDI is a prototype system developed by the University of Technology in Milan, Italy. It is a publish/subscribe system based on Map. The customer's subscription conditions are generally composed of "AND" operations of various atomic constraints. Each atomic constraint involves only one attribute. The system proposes the structure and algorithm to support the subscription/publishing of mobility. SIENA is a prototype system developed by the University of Colorado in the United States. It is relatively complete. Based on this platform, many components have been developed, including mobile and routing algorithms. But it also lacks the application-level QoS strategy, and is still not suitable for fields such as medical treatment, ship control, and telemetry systems that have high real-time requirements and handle emergency tasks. Therefore, there are still some key issues unresolved in the core technology of the publish/subscribe system based on the DDS standard. How to build a real-time data distribution platform on a distributed Realized and applied to many clients, these technical problems have become the focus of research and attention of technical personnel in the industry.

Contents of the invention

The purpose of the present invention is to provide a real-time data distribution system with a distributed network architecture and its working method. The system and method of the present invention are based on the overall structure of DDS, combined with the data storage, data transmission, and data storage required in practical applications. Data security, distributed network routing, QoS mechanism and other functions, it can provide three types of publishing/subscribing services for text information, file transmission and real-time video communication, and geographical location information, and in future applications, it can Expand the system structure, functions and data types to meet the application requirements of large-scale nodes.

In order to achieve the above object, the present invention provides a real-time data distribution system with a distributed network architecture, which uses a publish/subscribe communication mechanism to transmit information in the Internet and mobile networks. It is characterized in that the system includes: , The Linux operating system is used as the server subsystem, agent subsystem, terminal subsystem, and distributed database for storing system information and publishing/subscribing messages in the software operating environment, among which:

The server subsystem is used to complete operations such as interaction with the database, other servers in this layer, lower-layer publish/subscribe agents, load balancing, topic storage, and matching. It is equipped with six modules: a database that interacts with the database that stores the server list Interaction module, the inter-server interaction module that interacts with other servers in this layer, connects the publish/subscribe agent, and obtains the publish/subscribe interface module that participates in matching information from the publish/subscribe module of the agent subsystem, adopts a load balancing algorithm, The load balancing module for each server in the subsystem performs load balancing, and stores the topic information of publishing/subscribing into the message storage module of the publishing list and subscription list respectively, and adopts different matching algorithms according to user requirements to accurately implement the publishing/subscribing topic and QoS or fuzzy matching message matching module;

The agent subsystem is used to complete operations such as receiving topics from terminals, receiving matching events and forwarding them to subscribers, encapsulating messages, submitting publish/subscribe topics to upper-level servers, interacting with databases, and communicating between agents. It is equipped with seven modules: The C/S client interface module used for data interaction with the lower client under the client/server mode (Client/Server, C/S), used for the browser/server mode (Browser/Server, B/S ) under the B/S client interface module for data interaction with the lower client, control the corresponding QoS level according to the information in the database, and use the set communication protocol and routing to ensure safe and accurate data transmission module, send the user The data content processing module that converts the text, streaming media, geographic location and other information into the format specified by the system for publishing/subscribing, and the publishing/subscribing module that completes the publishing/subscribing information function, manages user information, configures network and QoS parameters, and submits data to the database The database processing module that stores the published/subscribed topics, reads the list of servers and agents in the database, and the inter-agent interaction module that interacts with other agents in this layer;

Terminal subsystem, the terminal subsystem is divided into two types of clients: C/S client and B/S client, which are mainly responsible for information interaction with users. Among them, the B/S client does not need to install the client on the terminal, and only needs the B/S server of the WWW browser to access the proxy to complete the publishing/subscribing process. The C/S client needs to install the client on the terminal, equipped with two software modules: used to execute topics including login/registration, setting QoS parameters, publishing/subscribing three types of information (text, geographic location, streaming media) The user interaction module interacts with the upper-layer publish/subscribe agent, so as to pass information or instructions to the agent interaction module of the publish/subscribe agent according to user operations;

Distributed database, this project adopts distributed database in the data storage part. Since the data is stored by the distributed database, no matter which agent the user passes through, he can obtain his own data, which realizes the basis of terminal roaming switching; in the database The information stored in mainly includes three publishing/subscribing topics of text, streaming media and geographic location information and their QoS settings, as well as related data such as users, agents, and servers; horizontal sharding based on publishing agent distinction is adopted, that is, the same agent publishes or subscribed messages are processed as the same fragment, and the distinguishing mark is the agent number.

In order to achieve the above object, the present invention also provides a working method of a real-time data distribution system with a distributed network architecture, which is characterized in that it includes the following steps:

(1) The user registers and logs in at the client, and after passing identity authentication, configure relevant parameters: the user starts the client interface, registers and logs in; the agent receives the user login information passed by the client, stores it in the database processing module, and performs identity verification. Verification; and then return corresponding responses according to different users, for users to set various network transmission parameters and QoS parameters of the system, and then the client will pass the user's operation and configuration parameters to the agent, and the agent will store and process these parameters;

(2) Publishing information/subscribing information: users can publish/subscribe various types of information such as text information, files, streaming media, and geographical information through the client interface. The user not only needs to fill in the specific data of various types of information, but also sets the transmission attribute parameters and QoS parameters of the information required to be published or subscribed. The client transmits the publish or subscribe information input by the user to the connected agent, and the agent receives the information through the client interface, encapsulates it and stores it in the distributed database, and performs corresponding publish/subscribe operations to the server subsystem;

(3) The matching process of the topic: the matching process of the three types of topics: text, file and real-time video, and geographic location information is the same. All attributes of the topic can be matched accurately and fuzzy, and range matching can also be performed on the topic of geographical type. . After the agent receives the publish/subscribe information sent by the client, it stores it in the distributed database and passes it to the server. The server matches the publishing and subscribing topics according to each matching item in the message, and returns the result to the publishing agent;

(4) The transmission and processing process of the matching topic: the publishing agent sends the matching event to the subscribing agent where the subscribing topic is based on the matching result, and the subscribing agent receives the matched message instead of the subscribing terminal. According to the QoS required by the subscriber, the topic matching the subscription is sent to the subscriber. When the transmission is completed, the interactive interface of the subscriber client pops up a message indicating that the message has been successfully received, that is, an interaction between publishing and subscribing is completed. Then the user performs different processing operations according to the received information.

The present invention is a real-time data distribution system with a distributed network architecture. Its technical innovation is: the system is built on the basis of OMG's DDS standard to provide users with real-time online data transmission: as long as the release If subscribers and subscribers are online at the same time, publishers can forward data directly to subscribers in a timely and efficient manner, and can support data distribution of text information, file transmission, real-time video communication, and geographic location information.

The system of the present invention can provide QoS control parameter setting to realize the QoS hierarchical configuration of the application layer, wherein there are as many as 20 kinds of QoS parameters available for configuration. The system provides QoS setting interface with two optional functions of self-adaptation and customization. The invention realizes the QoS routing strategy, controls the realization of the QoS parameters of the application layer by detecting the transmission characteristics such as the throughput, delay, and response time of the network, and effectively optimizes the network.

For the data form, content and size of different information, the present invention adopts corresponding different storage strategies respectively: the data subject that needs to be matched and distributed and the text information with small amount of data are stored in the proxy cache (and in the connected distributed database) ), and the specific information content (such as large-capacity files, real-time video, etc.) of the large amount of data at the publishing end is stored in the client. In the data storage part, the distributed database is used to realize. Distributed database systems can provide reliable information anytime and anywhere, and users can flexibly access information at any time without caring about the location and method of data storage. This separation of data storage from data processing can enhance the scalability of the system.

Description of drawings

Figure 1 is a schematic diagram of the principle of the publish/subscribe mechanism.

Fig. 2 is a system structure diagram of the present invention.

Fig. 3 is a structural diagram of the server subsystem of the present invention.

Fig. 4 is a structural diagram of the proxy subsystem of the present invention.

Fig. 5 is a structural diagram of the terminal subsystem of the present invention.

Fig. 6 is a sequence diagram of overall operation steps of the present invention.

Fig. 7 is an operation sequence diagram of user registration and login in the present invention.

Fig. 8 is an operation sequence diagram of publishing/subscribing, matching and transmission of text information topics in the present invention.

FIG. 9 is an operation sequence diagram of publishing/subscribing, matching, and transmission of a file transmission topic in the present invention.

Fig. 10 is an operation sequence diagram of publishing/subscribing, matching and transmission of real-time video communication topics in the present invention.

Fig. 11 is an operation sequence diagram of publishing/subscribing, matching and transmission of geographic information topics in the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings.

Referring to Fig. 2, introduce that the present invention has the real-time data distributing system of distributed network architecture and the structural composition of main equipment wherein, this system is to adopt the communication mechanism of publishing/subscribing to transmit information in Internet and mobile network, and system is by Windows, Linux The operating system is composed of four parts: the server subsystem 1, the agent subsystem 2, the terminal subsystem 3 and the distributed database 4 as the software operating environment, of which:

Server subsystem 1 is used to complete operations such as interaction with the database, other servers in this layer, lower-layer publish/subscribe agents, load balancing, topic storage, and matching. It is equipped with six modules: the one that interacts with the database that stores the server list The database interaction module, the inter-server interaction module that interacts with other servers in this layer, the publish/subscribe interface module that connects to the publish/subscribe agent and obtains the matching information from the publish/subscribe module of the agent subsystem, adopts a load balancing algorithm, The load balancing module of each server of the server subsystem performs load balancing, stores the topic information of publishing/subscribing into the message storage module of the publishing list and the subscription list respectively, adopts different matching algorithms according to user requirements, and performs the publishing/subscribing topic and QoS Message matching module for exact or fuzzy matching;

Agent subsystem 2 is used to complete operations such as receiving topics from terminals, receiving matching events and forwarding them to subscribers, encapsulating messages, submitting publish/subscribe topics to upper-level servers, interacting with databases, and communicating between agents. It is equipped with seven modules : The C/S client interface module used for data interaction with lower-level clients under the client/server mode (C/S), and used for communicating with lower-level clients under the browser/server mode (B/S) The B/S client interface module for data interaction at the end, controls the corresponding QoS level according to the information in the database, uses the set communication protocol and routing to ensure safe and accurate data transmission module, and transmits the text, streaming media, geographic data sent by the user The data content processing module that converts information such as location into the format specified by the system for publishing/subscribing, and the publishing/subscribing module that completes the publishing/subscribing information function, manages user information, configures network and QoS parameters, and stores publishing/subscribing topics in the database , The database processing module that reads the list of servers and agents in the database, and the inter-agent interaction module that interacts with other agents in this layer;

Terminal subsystem 3. The terminal subsystem is divided into two types of clients: C/S client and B/S client, which are mainly responsible for information interaction with users. Among them, the B/S client does not need to install the client on the terminal, and only needs the B/S server of the WWW browser to access the proxy to complete the publishing/subscribing process. The C/S client needs to install the client on the terminal, equipped with two software modules: used to execute topics including login/registration, setting QoS parameters, publishing/subscribing three types of information (text, geographic location, streaming media) The user interaction module interacts with the upper-layer publish/subscribe agent, so as to pass information or instructions to the agent interaction module of the publish/subscribe agent according to user operations;

Distributed database 4, the present invention adopts distributed database to realize in data storage part, because data is stored by distributed database, so no matter which agent the user is by which agent all can obtain own data; The information stored in database mainly includes text Three publish/subscribe topics, streaming media and geographic location information and their QoS, as well as related data such as users, agents, and servers; adopt horizontal sharding based on publishing agent distinctions, and process messages published or subscribed by the same agent as the same fragment , the distinguishing mark is the agent number.

Referring to Fig. 3, the structural composition and the functions thereof of six software modules of the server subsystem 1 are specifically illustrated:

Database interaction module, each server of the server subsystem stores its own information into the server list of the database through this module, including a series of data such as the address, name, and load of each server;

Inter-server interaction module, data interaction with other servers in this layer, including synchronous sharing of published or subscribed topics, load balancing between servers, etc.;

The publish/subscribe interface module is used to connect to the lower-layer publish/subscribe agent, obtain the publish/subscribe message sent by the publish/subscribe module of the agent subsystem to participate in the matching, and send the information upward to the message storage module for caching to be matched ;

The load balancing module periodically transmits workload information among the servers of the server subsystem, and uses a load balancing algorithm to distribute the workload of each server in order to solve network congestion problems, provide users with better service quality, and improve Server response speed and utilization efficiency of other resources avoid single point of failure in key parts of the network;

The message storage module receives the information delivered by the publish/subscribe interface module, and puts the publish/subscribe topic information from the publisher/subscriber into the publish list and the subscribe list respectively for matching; when the publisher and the subscriber cancel the publication or subscribe, delete the corresponding publish or subscribe topics from the publish list and subscribe list respectively;

The message matching module extracts information from the publishing list and subscription list of the message storage module, and uses corresponding matching rules and algorithms to perform exact matching, fuzzy matching or Range matching, and supports the logical relationship of "and" and "or" of matching items; if the two information matches, the matching result including the name and address of the subscription agent will be sent to the matching corresponding publishing agent, and the published content will be sent to the matching subscribers.

Referring to Fig. 4, the structural composition and the function thereof of seven software modules of agent subsystem 2 are specifically illustrated:

C/S client interface module, in the client/server mode, is used to exchange information with the publish/subscribe agent interaction module of the client, and convert the messages obtained from the client into the format specified by the system for publishing/subscribing;

The B/S client interface module, in the browser/server mode, is used to exchange information with the client’s publish/subscribe agent interaction module, including the server function of the B/S mode, and converts the messages obtained from the client into the system Specify the format for publishing/subscribing, which has the same function as the C/S client interface module;

The data content processing module encapsulates different types of information such as text, streaming media, and geographic location sent by the client into a fixed format according to certain rules, and sends them to the database processing module for storage;

The data transmission module includes four parts: QoS processing, data security, cross-layer interaction and network management: responsible for controlling the corresponding transmission QoS level according to the information provided by the database processing module, and selecting data with the network attributes and parameters set by the database processing module The protocol and routing of the transmission control the implementation of QoS parameters at the application layer by detecting the transmission characteristics of the network such as throughput, delay, and response time, realize asynchronous transmission and streaming media functions, and use the set data format to ensure transmission security; The functions of these four parts are described in detail below:

The QoS processing part controls the corresponding transmission QoS level according to the QoS parameter configuration information provided by the database processing module; the data distribution service of this system depends on the use of QoS policies, and QoS is a combination of multiple characteristics for controlling and realizing data distribution QoS. In order to ensure efficient and reliable communication, the QoS policies of the publisher and the receiver must be consistent with each other, that is, try to ensure that the two parties are decoupled: if the subscriber requests reliable information data, the corresponding publisher only provides best-effort QoS policy, no communication will be established between entities. This part has not less than 20 kinds of QoS parameters for accurately expressing the various QoS attributes of the system, and controls the realization of the QoS parameters of the application layer by detecting various network transmission characteristics of the network throughput, delay, and response time. Optimize the network and enhance transmission security; the settings of QoS parameters are shown in the following table:

name Function LIVELINESS Controls when and how a service determines that a participant is reachable and active RELIABILITY Control how data readers and writers treat processed samples HISTORY Decide how data readers and writers save samples for the instance DURABILITY Controls whether data writers should save samples after they have been sent to known subscribers

name Function DURABILITY_SERVICE Controls deletion of samples in TRANSIENT or PERSISTENT caches RESOURCE_LIMITS Determine the total amount of resources that can be consumed in order to meet the required QoS policy PARTITION Allows users to create logical partitions within the domain DEADLINE Detect when no data is read or written for a specified period of time LIFESPAN Applications can specify when samples expire USER_DATA Information can be attached to the created entity, the value of the USER_DATA policy is available in the respective built-in topic data topIC_DATA Used to attach additional information to the theme, the application can obtain information through the built-in theme and use it in a custom way GROUP_DATA Set the value to attach information to the created entity, and propagate the value of the GROUP_DATA policy through the built-in topic TRANSPORT_PRIORITY This strategy can be regarded as a prompt to the transport layer, indicating which priority to send messages, the larger the value, the higher the priority LATENCY_BUDGET Use this value to specify a delay interval to report unacceptable delays in transferring samples from publication to subscription ENTITY_FACTORY Controls whether they are automatically enabled when entities are created PRESENTATION Control how changes to the instance by the publisher are presented to the data reader DESTINATION_ORDER Controls the order in which samples are available to a data reader in a given instance

name Function WRITER_DATA_LIFECYCLE Control the lifetime of data instances managed by data writers READER_DATA_LIFECYCLE Control the lifecycle of data instances managed by data readers TIME_BASED_FILTER Control the interval at which data readers pay attention to changes in data instance values

The data security part is used to ensure the security of data transmission in the system. It adopts the user authentication strategy of the user management part, the authority management strategy of the domain, XML function setting, encryption and other measures for the transmitted data to realize this function;

The cross-layer interaction part is used to realize the cross-layer processing when the system is applied in the network, so that the system has a pluggable transport layer, supports UDP, TCP and network layer multicast routing communication protocols, and supports IPV6 and IPV4 network addresses configuration;

The network management part is used to manage and configure various network forms and communication protocols supported by the cross-layer interaction part, and adopts a distributed control method for distributed networks;

The publish/subscribe module, supported by the publish/subscribe mechanism, realizes the data distribution function based on the DDS specification; it includes the following two parts: the publish part and the subscribe part;

In the publishing part, the published topic information is extracted from the database, and the topic name, QoS parameters and data types are registered with the upper server, and when a matching subscriber is found, the published specific information is read from the database and sent to the matching Subscriber's agent;

In the subscription part, read the subscribed topic information from the database, register the topic name, QoS parameters and data type to the upper server, and when a matching publisher is found, receive the matching message from the corresponding publishing agent, and send the The topic name, the content of the published message, and various information of the publisher are extracted, put into the database, and wait for the subscriber client to connect and transmit it to the subscriber;

Database processing module, including five parts: user management, network configuration, QoS parameter configuration, publish/subscribe topic storage, server and proxy list, responsible for storing, managing and auditing user identity and related information and configuration parameters, network transmission parameters and QoS Configure parameters, store publish/subscribe topics in the database or obtain publish/subscribe topics from the database, support various database interfaces including MySql, Oracle, SQLServer, so as to adopt different storage methods according to different forms, contents and sizes of data The strategy constructs the database, and queries the list of servers and agents in the database to obtain the status information of each server in the upper layer and the status information of each agent in this layer; the following is the specific description of these five parts:

The user management part is responsible for the management of user registration review and login authentication, divides users into administrators and ordinary users, and assigns different permissions and implements corresponding management strategies: new users after registration must be authenticated by the administrator , to use the system;

The network configuration part is used to configure the network transmission parameters and cooperate with the data transmission module;

The QoS parameter configuration part is used to store the QoS parameters in the publish/subscribe topic sent by the terminal, and cooperate with the QoS processing part of the data transmission module;

The publish/subscribe topic storage part stores the publish/subscribe topic in the database to be matched, or reads the publish/subscribe topic to be matched from the database and passes it to the publish/subscribe module;

In the server and proxy list part, read the server and proxy list in the database, obtain the status information of the upper server and each proxy information of this layer, and connect to the appropriate server according to the overall status of the system;

The inter-agent interaction module is used to transfer data with other agents in the agent subsystem, including the transmission of matching events, terminal mobility management and other functions.

Referring to Fig. 5, the structural composition and the function thereof of two software modules of the terminal subsystem 3 are specifically illustrated:

The user interaction module includes five parts: user registration/login, QoS parameter setting, text information topic publishing/subscription, streaming media topic publishing/subscription, and geographic location topic publishing/subscription. The interactive interface is used to perform operations including login/registration, setting QoS parameters, publishing/subscribing text information, collecting, displaying and publishing/subscribing geographical information, publishing/subscribing streaming media information and displaying and other operations, through which the module prompts the user to include Whether the login is successful, whether the sending/subscribing information is successful, and whether the subscribed information is received; the following are the specific descriptions of these five parts:

For the user registration/login part, new users who use the system for the first time need to enter their user name and password in the login box, and the information is sent up to the user management part in the database management module of the proxy subsystem, and the administrator performs identity authentication The system can only be used after the user has entered the login box; the existing user enters the user name and password in the login box, and sends the information up to the user management part of the database management module of the proxy subsystem. If the information is correct, the login is successful, otherwise a prompt message will pop up , you need to log in again;

In the QoS parameter setting section, set QoS in the QoS setting column of the interface. There are two optional functions in this part: four commonly used QoS overall configuration schemes are given: continuous data, status information, warning information, and failover , the user can choose one of the schemes according to the actual needs, the system automatically configures the QoS function of the scheme according to the setting, and configures the custom function of the QoS parameters including throughput, delay and other transmission attributes, after the setting is completed Upload the QoS information to the agent subsystem to configure the QoS of the system;

The text information topic publishing/subscribing part is used to publish/subscribe text information, input the subject, title, content, abstract, unit and other information of the text information to be published/subscribed for publication or subscription, and upload the user’s text information to the agent Subsystem for further processing;

The publishing/subscribing part of the streaming media topic is used to publish/subscribe audio and video type information, and can publish or receive file transmission or real-time video communication. The publisher publishes information such as the name and user of the file or real-time video, and the subscriber subscribes to the corresponding information. After the two are successfully matched, the file transmission or real-time video communication between the publisher and the subscriber can be established according to the received matching information;

Geographic information topic publish/subscribe part, used to publish/subscribe geographic information, can display maps, collect geographic information through external GPS module or directly input geographic information to the database, can be based on user name, unit type, affiliated unit, etc. One or several items query geographic information and display them on the map. You can add, delete, or modify geographic information stored in the database. When subscribing, you can choose exact matching, fuzzy matching, or range matching; geographic information can be published/subscribed Four functions can be completed: single-point coordinate subscription, single-point track subscription, multi-point coordinate subscription, and multi-point track subscription. Single-point coordinate subscription can subscribe to the location information of a specific person or place, and multi-point coordinate subscription can simultaneously subscribe to the location information of multiple people or places with the same attribute or within a certain range. Single-point track and multi-point track subscription can display a section Real-time motion trajectory of one or more units in time;

The agent interaction module is used to interact with the upper-layer publish/subscribe agent, so as to transmit data and operations such as publish/subscribe information to the publish/subscribe agent according to user operations.

The design and implementation of the distributed database 4 are described in detail below:

The data storage of this project is implemented by a distributed database, which is used to store three publish/subscribe topics of text, streaming media and geographic location information and their QoS, as well as related data such as users, agents, and servers.

Other parts of the project interact with the database as follows: the user logs in in the terminal, publishes or subscribes to the message, and sends the published or subscribed topic to the connected agent. The agent preprocesses the received topic and stores it in the distributed database through the allocation strategy. At the same time, the agent starts the publish/subscribe module, reads the data in the database, determines the data type, and passes the data to the server for matching. When the publish/subscribe module receives the matching message, it stores the message in the corresponding database, and at the same time notifies the terminal that the message is successfully matched, and the terminal reads the message in the database through the proxy to complete the final matching process.

In terms of data sharding, horizontal sharding based on publishing agent distinction is adopted, that is, messages published or subscribed by the same agent are processed as the same fragment, and the distinguishing mark is the agent number. Data allocation adopts a data allocation strategy with certain redundancy. The allocation algorithm of heuristic trial copy is adopted, and certain optimization is made for the actual application of the project, and the user information, server information, release information and subscription information that have a decisive impact on the operation of the entire project are fully copied; other database tables are horizontally divided The slices are divided into different slices, and are allocated to each database node with certain redundancy according to the result of the heuristic trial elimination replica allocation algorithm.

Referring to Fig. 6, the operating steps of the working method of the real-time data distribution system of the present invention are introduced:

(1) The user registers and logs in at the client, and after passing identity authentication, configure relevant parameters: the user starts the client interface, registers and logs in; the agent receives the user login information passed by the client, stores it in the database processing module, and performs identity verification. Verification; and then return corresponding responses according to different users, for users to set various network transmission parameters and QoS parameters of the system, and then the client will pass the user's operation and configuration parameters to the agent, and the agent will store and process these parameters.

Referring to Figure 7, this step (1) includes the following specific operations:

(11) The user starts the client interface, that is, the user interaction module of the client is called;

(12) User registration and login. Registered users log in directly. At this time, the client calls the agent interface module to pass the registration and login information input by the user to the agent, and the user management part of the agent database processing module writes the information of the new user into the distribution The user storage part of the database is audited or authenticated for registered users. After the audit and verification is passed, a confirmation message is returned to the client;

(13) The user management part of the agent judges whether the login user is an administrator user or an ordinary user, and grants different permissions respectively: the administrator user can enter the administrator operation interface to manage the permissions and information of other ordinary users, and the ordinary user Users can only open the interactive interface with common functions;

(2) Publishing information/subscribing information: users can publish/subscribe various types of information such as text information, files, streaming media, and geographical information through the client interface. The user not only needs to fill in the specific data of various types of information, but also sets the transmission attribute parameters and QoS parameters of the information required to be published or subscribed. The client transmits the publish or subscribe information input by the user to the connected agent, and the agent receives the information through the client interface, encapsulates it and stores it in the distributed database, and performs corresponding publish/subscribe to the server subsystem.

Referring to Figures 8 to 11, this step (2) includes the following specific operations according to the type of publish/subscribe information:

(21) See Figure 8. To publish/subscribe text-type messages, enter the text information publishing/subscribing interface, fill in the necessary information such as the topic name, publishing unit, QoS level, etc. required for publishing/subscribing, and also include the title of the article , publishing unit, keywords, abstract and other information belonging to the topic of the article. In addition, when publishing the topic, you can also attach attachments with a total size of no more than 5M, and then send the publish/subscribe topic to the affiliated agent through the agent interaction module; The publishing and subscribing topic operations are completely independent and do not affect each other;

(22) See Figure 9, when you need to publish/subscribe a file-based topic, enter the file publishing/subscribing interface, fill in the necessary information such as the topic name, publishing unit, etc., and also include the local file storage path, file size, File description and other information belonging to the file to be published/subscribed, and then sent to the agent;

(23) Referring to Figure 10, when it is necessary to publish/subscribe the topic of real-time video communication, enter the real-time video topic publishing/subscribing interface, and fill in the real-time video topic, initiator, video description, etc. required for publishing/subscribing, which belong to real-time video communication information, which is then sent to the agent;

(24) Referring to Figure 11, when the user needs to publish the subject of geographical location information, the user enters the geographical location information release interface, and the user can manually fill in or obtain the subject information of geographical information to be released from the corresponding GPS module, including user name, Longitude and latitude, user unit type, unit, QoS level and other information, and specify whether to publish the instantaneous position coordinates at a certain moment or the movement track coordinates within a certain period of time, then the geographic location information can be released to the agent. When subscribing, the user must first choose a subscription mode from the four modes of single-point coordinate subscription, single-point track subscription, multi-point coordinate subscription, and multi-point track subscription, and then enter the required subscription information to subscribe to the agent geographic location information.

(3) Topic matching process: The three types of topic matching process are the same for text, file and real-time video, and geographic location information. After the agent receives the publish/subscribe information sent by the client, it stores it in the distributed database and passes it to the server. The server matches the publishing and subscribing topics according to each matching item in the message, and returns the result to the publishing agent. According to the matching result, the publishing agent sends the matching event to the subscribing agent where the subscribing topic resides. Including the following specific operations:

(31) The agent's client interface (C/S client interface or B/S client interface) receives these publish/subscribe information from the terminal, stores information such as topics and QoS parameters in the distributed database, and After the format is reconstructed to conform to the publish/subscribe format, it is passed to the server by the publish/subscribe module;

(32) After receiving the publish/subscribe information from the publish/subscribe interface module, the server extracts the topic from the message, and puts it into the publish list and subscribe list of the message storage module according to different information types. The message matching module uses the matching algorithm to fuzzy the publishing/subscribing topics in the message list according to the topic, topic type, keyword, publishing unit and other fields as matching items and the logical relationship of "and" and "or" of each matching item or an exact match. If the matching result of the publishing information and the subscription information is a failure, wait to receive new publishing information and subscription information, so as to continue matching until the matching is successful; if the matching result of the publishing information and subscription information is successful, the matching result will be return the publisher proxy;

(33) After the publisher agent knows the agent to which the subscriber belongs in the matching result, the publishing module transmits the publishing information stored in the database to the subscriber agent according to the selected route according to the QoS policy required by the QoS processing part.

(4) Transmission and processing of matching topics: Subscriber agents receive matched messages instead of subscribing terminals. When subscribing terminals connect to the agent, the agent sends the topics matching the subscription to the subscribers according to the QoS required by the subscribers. When the transmission is completed, the interactive interface of the subscriber client pops up a message indicating that the message has been successfully received, that is, an interaction between publishing and subscribing is completed. Then the user performs different processing operations according to the received information. Due to the different characteristics of the three types of data, the following specific operations are included:

(41) Referring to Figure 8, when the terminal subscribes to a text-type topic, after the terminal is connected to the proxy, the proxy sends the received text information matching the subscription to the subscriber.

(42) Referring to Figure 9, if the published/subscribed message is a file transfer type topic, after the publisher is connected to the agent, the agent returns matching information to the publisher, and the publisher opens the file transfer listening port and waits for the subscriber's file Transmission request; when the subscriber is connected to the broker, the broker will send the topic-related information matching the subscription to the subscriber, and the subscriber will request file transfer from the publisher according to the publisher address contained therein. After the publisher and the subscriber successfully establish a connection, the file can be transmitted to the subscriber according to the agreed QoS policy.

(43) Referring to Figure 10, if the published/subscribed message is a subject of real-time video communication, after the publisher is connected to the agent, the agent returns matching information to the publisher, and the publisher opens the video communication monitoring port, waiting for the subscriber's Video communication request; when the subscriber is connected to the agent, the agent sends the relevant information about the topic matching the subscription to the subscriber, and the subscriber sends a video communication request to it according to the publisher address in the matching result. After the publisher and the subscriber establish a connection, the publisher and the subscriber can conduct real-time video communication through the established connection according to the agreed QoS. In addition, due to the synchronous loose coupling of the publish/subscribe mechanism, both publishers and subscribers can establish video connections with other clients while communicating with each other in real-time video communication, forming a video conference application mode;

(44) Referring to Figure 11, when the subscriber is connected to the proxy, the proxy sends the matching result to the subscriber according to the QoS required by the subscriber. If the subscription mode is single-point coordinate subscription and multi-point coordinate subscription, then the subscriber can directly obtain the geographic coordinates he needs from the matching result sent by the agent, and display them on the map of the terminal interface. If the subscription mode is single-point trajectory subscription and multi-point trajectory subscription, then the subscriber needs to send a request to the publishing terminal or agent according to the publisher's location contained in the matching result, establish a connection with the publisher, obtain the continuous location coordinates of the subscribing unit, and Continuously displayed on the map of the subscription terminal interface.

Claims (7)

1. the real-time data distribution system that has distributed network architecture, be in internet and mobile network, to adopt the communication mechanism of publish/subscribe to transmit information, it is characterized in that, this system comprises: the server subsystem of finishing operations such as theme storage and theme coupling, finish from terminal and receive theme, transmit the agent subsystem of operations such as match event, submission publish/subscribe theme to the subscriber, finish the terminal subsystem of user's publish/subscribe information, and the distributed data base of storage publish/subscribe message and system information; The publish/subscribe service of polytype data such as text message, Streaming Media, geographical location information can be provided, nearly 20 kinds of QoS Control Parameter can be provided, to realize the QoS hierarchical arrangement of application layer.

2. real-time data distribution system according to claim 1, it is characterized in that: server subsystem, be used to finish operations such as the storage of, load balancing mutual, theme and coupling, be equipped with six modules with database, other servers, the publish/subscribe agency of lower floor:

The database interactive module, each server of server subsystem deposits oneself information the server list of database in by this module, comprising data such as the address of each server, title, loads;

Interactive module between server is carried out data interaction with other servers of this layer, comprise issue or topic of subscription share the functions such as load balancing between each server synchronously;

The publish/subscribe interface module is used to connect publish/subscribe agency, obtains the parameter that the agent subsystem transmission comes and the publish/subscribe message of coupling, and mails to message storage module and carry out buffer memory with to be matched;

Load balancing module is periodically transmitted workload information between each server of server subsystem, adopt load balancing algorithm, and each server is carried out load balancing;

Message storage module receives the publish/subscribe subject information that the publish/subscribe interface module is transmitted, and puts into issue tabulation and subscription list respectively and carries out buffer memory with to be matched; When publisher and subscriber cancel issue or subscribe to, then from issue tabulation and subscription list respectively deletion issue accordingly or topic of subscription;

The match messages module, information extraction from the issue tabulation of message storage module and subscription list, and according to the requirement of theme coupling, to occurrences such as the title of publish/subscribe theme, type, QoS accurately mate, fuzzy matching or commensurate in scope, support occurrence " with " and " or " logical relation mate; If both information matches, the matching result that just will comprise subscription agent title and address sends to the correspondence issue agency of coupling, is acted on behalf of according to type of theme the content distributed subscriber who sends to coupling by this issue.

3. real-time data distribution system according to claim 1, it is characterized in that: agent subsystem, be used to finish from terminal and receive theme, receive match event and transmit to the subscriber, encapsulation messages, the stratum server that makes progress are submitted the publish/subscribe theme to, operations such as, inter-agent communication mutual with database have seven modules:

C/S client-side interface module under C/S (Client/Server) pattern, is acted on behalf of the interactive module interactive information with the publish/subscribe of C/S client, obtains the publish/subscribe theme;

B/S client-side interface module under B/S (Browser/Server) pattern, is carried out information interaction with the B/S client, comprises the server of B/S pattern, for the B/S client provides the data distribution services, obtains the publish/subscribe theme;

The data content processing module, different kinds of information such as the text that client is sent, Streaming Media, geographical position are packaged into set form according to certain rule, are sent to the database processing module to store;

Data transmission module, comprise QoS processing, data security, stride mutual and four part of network management of layer: be responsible for controlling corresponding transmission QoS grade according to the information that the database processing module provides, agreement and the route of selecting transfer of data with the network attribute and the parameter of the setting of database processing module, transmission characteristics such as the throughput by detecting network, time delay, response time, control the realization of application layer qos parameter, realize asynchronous transmission and flow medium function, and adopt specific data format to guarantee transmission security;

The publish/subscribe module is support with publish/subscribe mechanism, realizes based on DDS specification data distribution function; Comprise issue part and subscribe to part: the issue part, extract issuing subject from database, to server registration subject name, qos parameter and data type, and act on behalf of the transmission subject content to the subscriber of coupling; Subscribe to part, from database, read topic of subscription,, and receive the coupling subject content that sends from coupling issue agency, put into database to server registration subject name, qos parameter and data type;

The database processing module, comprise user management, network configuration, qos parameter configuration, the storage of publish/subscribe theme, server and five parts of list of proxies, be responsible for storage, management and audit user identity and relevant information and configuration parameter, network transmission parameter and qos parameter are configured, the state information of each server and each agency's of this layer state information stored in access publish/subscribe theme from database;

Interactive module between the agency, be used for and other agencies of agent subsystem between Data transmission, comprise the functions such as transmission, terminal mobility management of match event.

4. real-time data distribution system according to claim 1 is characterized in that: terminal subsystem is divided into two kinds of clients: C/S client and B/S client, mainly be responsible for and the user between information interaction; Wherein the B/S client need not installed client on terminal, only needs WWW browser access agency's B/S client-side interface module just can finish the publish/subscribe process; The C/S client then need be installed the C/S client in terminal, is equipped with two software modules:

User interactive module, comprise that the user registers/lands, qos parameter setting, text message theme publish/subscribe, Streaming Media theme publish/subscribe, five parts of geographical position theme publish/subscribe, be used for carrying out and comprise login/registration, qos parameter is set, the publish/subscribe text message, collection, demonstration and publish/subscribe geography information, the publish/subscribe stream media information also shows and other operation;

Act on behalf of interactive module, be used for carrying out mutual so that data such as publish/subscribe information and operation are passed to the publish/subscribe agency according to user's operation with upper strata publish/subscribe agency.

5. real-time data distribution system according to claim 1 is characterized in that: distributed data base, be responsible for the storage part; Be used for data such as publish/subscribe theme, qos parameter and the agency of storing subscriber information, three types (text message, geographical location information, stream media informations) and server state; Employing is based on the horizontal fragmentation that the issue agency distinguishes, and the message that is about to the same generation haircut cloth or subscription is handled as same fragment, and diacritics is agency's numbering, and data allocations adopts the data allocations strategy that has certain redundancy.

6. real-time data distribution system according to claim 1, it is characterized in that: system is that frame foundation makes up with the DDS standard of OMG, for the user provides the transmission of real-time online data: the publisher can be in time, efficiently data directly are forwarded to the subscriber, can support the data distribution of three types of text messages, Streaming Media, geographical location information; The method of work of native system comprises following operating procedure:

(1) user logins in the client registration: the user is in client registration login, and the agency passes to the database processing module with its logon information, carries out authentication; By client user's operation and configuration parameter passed to the agency then;

(2) release news/subscription information: the user can carry out publish/subscribe to the theme of three types of text messages, stream media information, geographical location information by client end interface, and sends it to the agency;

(3) matching process of theme: the agency stores the publish/subscribe theme of receiving into database and sends to server simultaneously, is required theme is mated according to coupling by server, and the result is returned to the agency;

(4) transmission and the processing procedure of coupling theme: the subscriber acts on behalf of and replaces subscribing to the theme that the terminal reception has been mated, and will mate theme according to the desired QoS of subscriber when terminal connects and send to the subscriber subscribing to, terminal is carried out different processing according to the type of theme that receives and is operated then.

7. real-time data distribution system according to claim 6 is characterized in that: described step (2) further comprises following concrete operations content according to the kind of publish/subscribe information:

(21) theme of publish/subscribe text type, fill in subject, issuing unit, QoS grade and article title, the unit of delivering, keyword, summary etc. and belong to the information of article theme, can attach the annex that is no more than 5M, send the publish/subscribe theme to the agency then;

(22) publish/subscribe is based on the theme of file transfer, and fill up a document name, file storage path, file size, file specification etc. belong to the information of file to be released, sends the publish/subscribe theme to the agency then;

(23) information that real-time video theme, promoter, video presentation etc. belong to real-time video communication filled in the theme of publish/subscribe real-time video communication, sends the publish/subscribe theme to the agency then;

(24) theme of issue geographical location information, the user can manually insert or obtain the geography information subject information that will issue from corresponding GPS module, comprise information such as user's name, longitude and latitude, user unit one belongs to type, affiliated unit, QoS grade, and to specify what will issue be the instantaneous position coordinate in a certain moment or the movement locus coordinate in a certain period, can be to agency's issue corresponding geographical position information;

Native system has been realized the geography information subscription of four kinds of patterns, be respectively that the single-point coordinate is subscribed to, the single-point track is subscribed to, the multiple spot coordinate is subscribed to, the multiple spot track is subscribed to: the single-point coordinate is subscribed to the positional information that can subscribe to specific people or place, the multiple spot coordinate is subscribed to subscribe to simultaneously has same alike result or a plurality of personnel within the specific limits or the positional information in place, and single-point track and multiple spot track are subscribed to the real time kinematics track that can show one or more units in a period of time; Subscribe to the theme of geographical location information type, the user at first will select a kind of subscribing mode from four kinds of patterns, import corresponding subscription information then and subscribe to.

Images