CN114547129B - Distributed satellite data receiving system and method based on big data storage platform - Google Patents

Abstract

The invention discloses a distributed satellite data receiving system and a distributed satellite data receiving method based on a big data storage platform, wherein the system comprises a plurality of data receiving computers, the big data storage platform and a monitoring computer, wherein the plurality of data receiving computers comprise a public data receiving computer and a plurality of observation data receiving computers, the public data receiving computer is used for receiving public data, and each data receiving computer in the plurality of observation data receiving computers is respectively used for establishing connection with a corresponding observation station according to a set observation station list and receiving observation data of the observation station; the big data storage platform is used for storing public data and observation data; the monitoring computer is used for monitoring the running state of each data receiving computer. The invention adopts a distributed architecture, connects different observation stations on a plurality of general computers respectively and receives different types of data, can reduce the requirement of a large amount of received data on the performance of the computers and has good expansibility.

Description

Distributed satellite data receiving system and method based on big data storage platform

technical field

The invention belongs to the technical field of satellite data reception, and in particular relates to a distributed satellite data reception system and method based on a large data storage platform.

Background technique

In the process of satellite data processing, it is necessary to receive satellite data from all satellite observation sites, and process and store them in real time. With the development of the satellite business, there are more and more satellites, and there are more and more satellite observation sites on the ground, which puts forward higher requirements for the data processing capabilities of the responsible units to receive data computers. The general idea is to solve this problem based on software performance optimization and computer performance improvement. However, due to the increasing amount of data to be received, the software performance optimization improvement is limited, and the higher the performance of the computer, the higher the cost. There came great financial pressure. At the same time, due to the increase in the amount of data, the real-time storage of data also brings high challenges when the receiving and processing capabilities barely meet the demand.

Contents of the invention

In order to solve the above problems in the prior art, the present invention provides a distributed satellite data receiving system and method based on a big data storage platform. The technical problem to be solved in the present invention is realized through the following technical solutions:

One aspect of the present invention provides a distributed satellite data receiving system based on a big data storage platform, including a plurality of data receiving computers, a big data storage platform and a monitoring computer, wherein,

The multiple data receiving computers include a public data receiving computer and a plurality of observation data receiving computers, wherein the public data receiving computer is used to receive public data, and the public data is data common to all observation sites, so Each data receiving computer in the plurality of observation data receiving computers is respectively used to establish a connection with the corresponding observation site according to the set observation site list and receive observation data from the corresponding observation site;

The big data storage platform is used to receive and store the public data and observation data from each observation data receiving computer;

The monitoring computer is connected to the plurality of data receiving computers, and is used for monitoring the running status of each data receiving computer.

In one embodiment of the present invention, each observation data receiving computer is provided with a configuration file setting module, a data receiving module and a data transmission module, wherein,

Communication configuration file and observation site configuration file are set in the described configuration file setting module, wherein, include Ntrip service end IP, port, username and password in the described communication configuration file, include to be connected in the described observation site configuration file A list of observation sites and the type of observation data that each observation site needs to receive;

The data receiving module is used to establish a connection with the corresponding observation site through the Ntrip protocol according to the set observation site list, and receive corresponding types of observation data;

The data transmission module is used to upload the received observation data to the big data storage platform.

In one embodiment of the present invention, each data receiving computer is also provided with a data processing module, and the data processing module is used to perform data processing on the observation data according to the preset storage specification to form the required file format.

In one embodiment of the present invention, the data receiving module is specifically used for:

Read the communication configuration file and the observation site configuration file, and obtain the first observation site to receive data; establish a connection with the first observation site through the Ntrip protocol and perform data reception and data processing; determine whether the current observation site is The last observation site in the observation site configuration file, if not, establishes a connection with the next observation site and performs data reception and processing, and if yes, the data reception is completed.

In one embodiment of the present invention, the monitoring computer and the multiple data processing computers are all in the same local area network.

In one embodiment of the present invention, the monitoring computer uses the Udp protocol to communicate with the data processing computer.

Another aspect of the present invention provides a method for receiving distributed satellite data based on a large data storage platform, which is executed by the distributed satellite data receiving system described in any one of the above-mentioned embodiments, and the method includes:

Use different observation data receiving computers to receive observation data from different observation sites;

Utilize public data receiving computers for receiving public data;

performing data processing on said public data and said observation data to form a specified file format;

Upload the data in the file format to the big data storage platform for storage;

During the data receiving process, the running status of each data receiving computer is monitored.

In one embodiment of the present invention, using different observation data receiving computers to receive observation data of different observation sites, including:

Set the communication configuration file and the observation site configuration file, wherein the communication configuration file includes the Ntrip server IP, port, user name and password, and the observation site configuration file includes the list of observation sites to be connected and each observation site The type of observational data to be received;

According to the set observation site list, establish a connection with the corresponding observation site through the Ntrip protocol, and receive the corresponding type of observation data;

According to the preset storage specification, the observation data is processed to form the required file format;

Upload the data in the file format to the big data storage platform.

In one embodiment of the present invention, according to the set observation site list, a connection is established with the corresponding observation site through the Ntrip protocol, and corresponding types of observation data are received, including:

Read the communication configuration file and the observation site configuration file, and obtain the first observation site to receive data;

Establish a connection with the first observation site through the Ntrip protocol and receive data;

Judging whether the current observation site is the last observation site in the observation site configuration file, if not, establishes a connection with the next observation site and performs data reception, and if so, completes the data reception.

In one embodiment of the present invention, the running status of each data receiving computer is monitored, including:

Receive the log information of each data receiving computer for data analysis and display;

Judging the running status of each data receiving computer, and automatically reconnecting the data receiving computer when the data receiving computer is abnormally connected;

Generate log files with a unified format and upload them to the big data storage platform for storage.

Compared with prior art, the beneficial effect of the present invention is:

The distributed satellite data receiving system based on the big data storage platform of the present invention uses a distributed architecture to connect different observation sites and receive different types of data on multiple general-purpose computers, and completes the reception of all data as a whole. The system of the present invention has good expansibility, theoretically, without considering the amount of hardware, it can realize the reception and storage of enough satellite site data, so that the number of sites no longer constitutes the performance bottleneck of receiving data, and can be quickly It can well solve the performance problems of many transmission sites, large data volume, high real-time requirements and long duration in the process of satellite data reception.

The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments.

Description of drawings

Fig. 1 is a block diagram of a distributed satellite data receiving system based on a big data storage platform provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of a specific framework of a distributed satellite data receiving system based on a big data storage platform provided by an embodiment of the present invention;

Fig. 3 is a schematic module diagram of an observation data receiving computer provided by an embodiment of the present invention;

Fig. 4 is a communication schematic diagram of a monitoring computer provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of a processing flow of a monitoring computer provided by an embodiment of the present invention;

6 is a flow chart of a satellite data distributed processing method based on a big data platform provided by an embodiment of the present invention;

Fig. 7 is a schematic diagram of a data receiving process of an observation data receiving computer provided by an embodiment of the present invention.

Detailed ways

In order to further explain the technical means and effects of the present invention to achieve the intended purpose of the invention, a distributed satellite data receiving system and method based on a large data storage platform proposed according to the present invention will be described below in conjunction with the accompanying drawings and specific implementation methods. Detailed description.

The aforementioned and other technical contents, features and effects of the present invention can be clearly presented in the following detailed description of specific implementations with accompanying drawings. Through the description of specific embodiments, the technical means and effects of the present invention to achieve the intended purpose can be understood more deeply and specifically, but the accompanying drawings are only for reference and description, and are not used to explain the technical aspects of the present invention. program is limited.

It should be noted that in this article, relational terms such as first and second etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the terms "comprises", "comprises" or any other variation are intended to cover a non-exclusive inclusion such that an article or device comprising a set of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the article or device comprising said element.

Embodiment one

Please refer to FIG. 1 and FIG. 2 , the distributed satellite data receiving system of this embodiment includes a plurality of data receiving computers, a big data storage platform 1 and a monitoring computer 2 . In the multiple data receiving computers, a public data receiving computer 3 and a plurality of observation data receiving computers 4 are included, wherein the public data receiving computer 3 is used to receive public data, and the public data is the common data of all observation sites, Each data receiving computer in the plurality of observation data receiving computers 4 is respectively used to establish a connection with a corresponding observation site according to the set observation site list and receive observation data from the corresponding observation site.

Specifically, for the public data, each observation site is the same, therefore, when receiving data, only the public data receiving computer 3 needs to receive the public data from the real-time data network or any observation site.

Further, please refer to FIG. 2. FIG. 2 is a block diagram of an observation data receiving computer provided by an embodiment of the present invention. Each observation data receiving computer in this embodiment is provided with a configuration file setting module 41, a data receiving Module 42 and data transmission module 43, wherein, configuration file setting module 41 is set with communication configuration file and observation site configuration file, includes Ntrip service end IP, port, user name and password in the described communication configuration file, described observation The site configuration file includes a list of observation sites to be connected and the type of observation data that each observation site needs to receive; the data receiving module 42 is used to establish a connection with the corresponding observation site through the Ntrip protocol according to the set observation site list, and receive corresponding type of observation data; the data transmission module 43 is used to upload the received observation data to the big data storage platform.

Further, the data receiving module 42 is specifically used to: read the communication configuration file and the observation site configuration file, and obtain the first observation site to receive data; establish a connection with the first observation site through the Ntrip protocol and perform data Receiving and data processing; judging whether the current observation site is the last observation site in the observation site configuration file, if not, establish a connection with the next observation site and perform data reception and processing, if so, the data reception is completed.

In this embodiment, each observation data receiving computer is also provided with a data processing module 44 for performing data processing on the observation data according to a preset storage specification to form a required file format.

The application modules set on each observation data receiving computer are the same, but the list of receiving observation sites is different. According to the amount of observation site data to be received and the performance of the data receiving computer, appropriate sites can be assigned to each receiving computer. Theoretically, the number of observation sites that can be received is unlimited.

It should be noted that, when the data receiving computer stores the file after receiving the data, in order to avoid the file being too large, the file needs to be stored in different periods. The length of the period depends on the frequency of subsequent processing. If it is processed once an hour, one file can be stored in one hour; if it is processed once every 30 minutes, it will take 30 minutes to store one file. The following is a 1-hour period to form the specification for storing files:

Corresponding to the private data of each observation site, a file is formed every hour, and the time is determined according to the time in the received content (not according to the computer time); the directory and file name should be formed according to the time and site name to provide convenience for subsequent data acquisition ;File storage format: the stored file format, it is necessary to be able to find the corresponding file according to the date, observation site name and hour.

The monitoring computer 2 is connected to the plurality of data receiving computers for monitoring the running status of each data receiving computer, and the monitoring computer 2 and the plurality of data processing computers are all in the same local area network.

In this embodiment, as shown in FIG. 4 , the monitoring computer 2 uses the Udp protocol to communicate with the data processing computer. Please refer to FIG. 5 . FIG. 5 is a schematic diagram of a processing flow of a monitoring module provided by an embodiment of the present invention. The running process of the monitoring computer 2 of the present embodiment is as follows: first, start the monitoring program, read the communication configuration, set up the Udp monitoring thread with the data receiving computer, receive the log (log) information from each data receiving computer, and the log information includes Information that needs to be monitored, such as heartbeat, sending and receiving data, algorithm execution, execution time, etc. Subsequently, the log information is analyzed, the data information is interpreted, and the data information is classified and displayed on the computer where the monitoring computer 2 is located, so as to realize the monitoring of the operation status of each data receiving computer. Furthermore, a unified log file can also be formed and sent to a big data storage platform for storage.

Further, the big data storage platform 1 is used to receive and store the public data and observation data from each observation data receiving computer and monitoring files generated by the monitoring computer 2 .

In this embodiment, the plurality of data receiving computers and the monitoring computer 2 can all access the big data platform to read and write data. Because it is in a high-speed local area network, the packet loss rate does not need to be considered. The entire communication protocol adopts the Udp protocol. In this way, the startup sequence of all application modules does not need to be deliberately arranged, and a data processing computer client can be started and stopped at will, and data can also be added at any time. Process the computer to meet the needs of running applications continuously and connecting and sending applications at regular intervals.

In addition, the distributed satellite data receiving system of this embodiment has good scalability, and can flexibly configure stations according to the actual needs of receiving satellite observation site data, and use general-purpose computers to complete the reception of satellite data. Its expansion method is: according to the number of satellite observation stations to be received and the amount of data of each observation station, by adjusting the list of receiving stations of each data receiving computer, and increasing or reducing the number of data receiving computers, any number of satellite observations can be satisfied Site data reception.

In this embodiment, the monitoring computer 2 needs to receive and display the log information of all data receiving computers, and a computer with higher performance can be selected for processing, or the types of log information can be reduced to reduce the amount of monitored information.

This embodiment is based on the distributed satellite data receiving system of the big data storage platform, through the research on the communication protocol of the satellite observation site, and according to the structure and function of the big data storage platform, the data to be received is classified: for the observation site data, according to The ability of each data receiving computer divides the receiving sites, and multiple receiving computers complete the data receiving for all sites; for public data, other separate receiving computers can be arranged to receive; at the same time, provide the monitoring capability of the entire received information, each The receiving computer sends the receiving status information to the monitoring computer to complete the monitoring of the receiving status of all data and the status of the receiving computer.

This embodiment is based on a distributed satellite data receiving system based on a big data storage platform. By using a distributed architecture, multiple general-purpose computers are respectively connected to different observation sites and receive different types of data, so as to complete the reception of all data as a whole. The system of the present invention has good expansibility, theoretically, without considering the amount of hardware, it can realize the reception and storage of enough satellite site data, so that the number of sites no longer constitutes the performance bottleneck of receiving data.

Embodiment two

On the basis of the foregoing embodiments, this embodiment provides a method for receiving distributed satellite data based on a large data storage platform, as shown in FIG. 6 , the method for receiving distributed satellite data includes:

S1: Use different observation data receiving computers to receive observation data from different observation sites.

Specifically, a communication configuration file and an observation site configuration file are set, wherein, the communication configuration file includes the Ntrip server IP, port, user name and password, and the observation site configuration file includes a list of observation sites to be connected and each The type of observation data that each observation site needs to receive; according to the set observation site list, establish a connection with the corresponding observation site through the Ntrip protocol, and receive the corresponding type of observation data; according to the preset storage specification, store the observation data into data Processing to form the required file format; upload the data in the file format to the big data storage platform.

Further, as shown in Figure 7, according to the set observation site list, establish a connection with the corresponding observation site through the Ntrip protocol and receive corresponding types of observation data, specifically including:

Read the communication configuration file and the observation site configuration file, and obtain the first observation site to receive data; establish a connection with the first observation site through the Ntrip protocol and perform data reception; determine whether the current observation site Configure the last observation site in the observation site configuration file, if not, establish a connection with the next observation site and perform data reception, if yes, complete the data reception.

In this embodiment, the application modules set on each observation data receiving computer are the same, but the list of receiving observation sites is different. According to the amount of observation site data to be received and the performance of the data receiving computer, it can be assigned to each Receive computer appropriate site. Theoretically, the number of observation sites that can be received is unlimited.

S2: Utilize the public data receiving computer to receive the public data.

The public data is data common to all observation sites, and for the public data, each observation site is the same, so when receiving data, it is only necessary to receive the public data from the real-time data network or any observation site That's it.

S3: Perform data processing on the public data and the observation data to form a specified file format.

Specifically, when the data receiving computer stores the file after receiving the data, in order to avoid the file being too large, the file needs to be stored in time periods. The length of the period depends on the frequency of subsequent processing. If it is processed once an hour, one file can be stored in one hour; if it is processed once every 30 minutes, it will take 30 minutes to store one file.

S4: Upload the data in the file format to the big data storage platform for storage.

S5: During the data receiving process, monitor the running status of each data receiving computer.

Further, said S5 includes:

S51: receiving log information of each data receiving computer for data analysis and display;

S52: Determine the running status of each data receiving computer, and automatically reconnect the data receiving computer when the data receiving computer is abnormally connected;

S53: Generate a log file with a unified format and upload it to a big data storage platform for storage.

This embodiment is based on the distributed satellite data receiving method of the big data storage platform, through the research on the communication protocol of the satellite observation site, and according to the structure and function of the big data storage platform, the data to be received is classified: for the observation site data, according to The ability of each data receiving computer divides the receiving sites, and multiple receiving computers complete the data receiving for all sites; for public data, other separate receiving computers can be arranged to receive; at the same time, provide the monitoring capability of the entire received information, each The receiving computer sends the receiving status information to the monitoring computer to complete the monitoring of the receiving status of all data and the status of the receiving computer, which can well solve the problem of many transmission sites, large data volume, high real-time requirements, duration Long performance issues.

The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be assumed that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field of the present invention, without departing from the concept of the present invention, some simple deduction or replacement can be made, which should be regarded as belonging to the protection scope of the present invention.

Claims (8)

1. A distributed satellite data receiving system based on big data storage platform, is characterized in that, comprises a plurality of data receiving computers, big data storage platform and monitoring computer, wherein, The multiple data receiving computers include a public data receiving computer and a plurality of observation data receiving computers, wherein the public data receiving computer is used to receive public data, and the public data is data common to all observation sites, so Each data receiving computer in the plurality of observation data receiving computers is respectively used to establish a connection with the corresponding observation site according to the set observation site list and receive observation data from the corresponding observation site; The big data storage platform is used to receive and store the public data and observation data from each observation data receiving computer; The monitoring computer is connected to the plurality of data receiving computers, and is used to monitor the operating status of each data receiving computer; Each observation data receiving computer is provided with a configuration file setting module, a data receiving module and a data transmission module, wherein, Communication configuration file and observation site configuration file are set in the described configuration file setting module, wherein, include Ntrip service end IP, port, username and password in the described communication configuration file, include to be connected in the described observation site configuration file A list of observation sites and the type of observation data that each observation site needs to receive; The data receiving module is used to establish a connection with the corresponding observation site through the Ntrip protocol according to the set observation site list, and receive corresponding types of observation data; The data transmission module is used to upload the received observation data to the big data storage platform, The monitoring computer and the plurality of data receiving computers are all in the same local area network, and the operation process of the monitoring computer includes: Start the monitoring program, read the communication configuration, establish a Udp monitoring thread with the data receiving computer, receive log information from each data receiving computer, then analyze the log information, interpret the data information and analyze the data on the monitoring computer The information is classified and displayed to realize the monitoring of the operation status of each data receiving computer. 2. the distributed satellite data receiving system based on big data storage platform according to claim 1, is characterized in that, each data receiving computer is also provided with data processing module, and described data processing module is used for according to preset Store the specification, and process the observation data to form the required file format. 3. the distributed satellite data receiving system based on big data storage platform according to claim 2, is characterized in that, described data receiving module is specifically used for: Read the communication configuration file and the observation site configuration file, and obtain the first observation site to receive data; establish a connection with the first observation site through the Ntrip protocol and perform data reception and data processing; determine whether the current observation site is The last observation site in the observation site configuration file, if not, establishes a connection with the next observation site and performs data reception and processing, and if yes, the data reception is completed. 4. the distributed satellite data receiving system based on big data storage platform according to claim 1, is characterized in that, described monitoring computer adopts Udp protocol to carry out data communication with described data receiving computer. 5. A method for receiving distributed satellite data based on a large data storage platform, characterized in that, utilizing the distributed satellite data receiving system described in any one of claims 1 to 4 to perform, said method comprising: Use different observation data receiving computers to receive observation data from different observation sites; Utilize public data receiving computers for receiving public data; performing data processing on said public data and said observation data to form a specified file format; Upload the data in the file format to the big data storage platform for storage; During the data receiving process, the running status of each data receiving computer is monitored. 6. the distributed satellite data receiving method based on big data storage platform according to claim 5, is characterized in that, utilizes different observation data receiving computers to receive the observation data of different observation sites, comprising: Set the communication configuration file and the observation site configuration file, wherein the communication configuration file includes the Ntrip server IP, port, user name and password, and the observation site configuration file includes the list of observation sites to be connected and each observation site The type of observational data to be received; According to the set observation site list, establish a connection with the corresponding observation site through the Ntrip protocol, and receive the corresponding type of observation data; According to the preset storage specification, the observation data is processed to form the required file format; Upload the data in the file format to the big data storage platform. 7. The distributed satellite data receiving method based on the big data storage platform according to claim 6, characterized in that, according to the set observation site list, establish a connection with the corresponding observation site through the Ntrip protocol, and receive corresponding types of observations data, including: Read the communication configuration file and the observation site configuration file, and obtain the first observation site to receive data; Establish a connection with the first observation site through the Ntrip protocol and receive data; Judging whether the current observation site is the last observation site in the observation site configuration file, if not, establishes a connection with the next observation site and performs data reception, and if so, completes the data reception. 8. the distributed satellite data receiving method based on big data storage platform according to claim 7, is characterized in that, the running status of each data receiving computer is monitored, comprising: Receive the log information of each data receiving computer for data analysis and display; Judging the running status of each data receiving computer, and automatically reconnecting the data receiving computer when the data receiving computer is abnormally connected; Generate log files with a unified format and upload them to the big data storage platform for storage.

Images