CN115833913B - Multi-source heterogeneous space-based data acquisition method and system based on message-driven mode - Google Patents
Multi-source heterogeneous space-based data acquisition method and system based on message-driven mode Download PDFInfo
- Publication number
- CN115833913B CN115833913B CN202211469130.0A CN202211469130A CN115833913B CN 115833913 B CN115833913 B CN 115833913B CN 202211469130 A CN202211469130 A CN 202211469130A CN 115833913 B CN115833913 B CN 115833913B
- Authority
- CN
- China
- Prior art keywords
- data
- day
- module
- processing module
- acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000012545 processing Methods 0.000 claims abstract description 107
- 238000012544 monitoring process Methods 0.000 claims abstract description 68
- 230000004044 response Effects 0.000 claims abstract description 27
- 230000007246 mechanism Effects 0.000 claims abstract description 12
- 230000008569 process Effects 0.000 claims description 25
- 238000012216 screening Methods 0.000 claims description 25
- 238000004140 cleaning Methods 0.000 claims description 22
- 230000005540 biological transmission Effects 0.000 claims description 19
- 238000006243 chemical reaction Methods 0.000 claims description 11
- 238000013480 data collection Methods 0.000 claims description 9
- 238000007689 inspection Methods 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 2
- 230000001934 delay Effects 0.000 abstract description 3
- 230000010365 information processing Effects 0.000 abstract description 2
- 230000008859 change Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 206010033799 Paralysis Diseases 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012938 design process Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000035515 penetration Effects 0.000 description 1
- 239000002344 surface layer Substances 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
- Radio Relay Systems (AREA)
Abstract
The invention relates to a multi-source heterogeneous day-base data acquisition method and system based on a message driving mode, wherein the method comprises the steps of utilizing a data acquisition message processing module to plan an acquisition plan according to the requirements on the day-base data and sending out data requirement planning; the data monitoring and sending module is used for receiving the data demand plan, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, monitoring newly added day base data in a multithreading mode according to a monitoring mechanism which is found to be accessed, sending monitored day base data information to the data scheduling processing module in a message driving mode, carrying out unified scheduling processing on all acquired day base data by the data scheduling processing module, and feeding back a processed normalized multisource data response to the data acquisition information processing module. The invention can realize the automatic access of the multi-type satellite with different data types, formats, precision and time delays and the space-based data of each network segment of various loads.
Description
Technical Field
The invention relates to the technical field of day-based data acquisition, in particular to a multi-source heterogeneous day-based data acquisition method and system based on a message driving mode.
Background
The day-based data is used as an important component of the day-to-ground integrated information network, and has the characteristics of high updating speed and large data volume. The data sources are complex and various, and the network systems are different, so that great difficulty is brought to data collection and aggregation. Different day data sources are complex and various, timeliness and stability are different, and elastic collection cannot be carried out according to the characteristics of each type of data.
The data-driven acquisition mode is often realized by adopting shared storage, the mode is greatly influenced by network fluctuation, and the phenomenon of slow storage and mounting response can occur after the mode operates for a certain time, so that data can not be acquired and accessed in time, or the situation of instantaneous concurrent access of backlog data occurs, and huge impact is caused for a back-end data processing platform.
In engineering practice, there are several prominent problems with the construction of day-based data acquisition capabilities:
(1) In the prior art, the physical isolation of data acquisition among different network systems is not high, so that data mixed penetration is easy to cause, and the safety of a data source is threatened;
(2) The stability of the data monitoring driving mechanism is poor, the response to a large amount of data acquired instantaneously is slow, and even the response is jammed and paralyzed, so that the data is lost;
(3) The robustness of information interaction in the data acquisition process is low, and information can not be presented in a closed loop mode;
(4) The data acquisition system has weak elastic upgrading capability along with the increase of the types of the accessible antenna-based information due to the lack of expansibility and compatibility.
Disclosure of Invention
In order to solve the technical problems, the invention provides a multi-source heterogeneous day-base data acquisition method and system based on a message driving mode, so as to improve the acquisition capability of the day-base data with complex and various sources and different types in the same network segment or different network segments.
In order to achieve the above purpose, the technical scheme of the invention is as follows:
in a first aspect, the present invention provides a method for collecting multisource heterogeneous day base data based on a message driven mode, including:
a data acquisition message processing module is utilized to plan an acquisition plan according to the requirements on the space-based data, and data requirement planning is sent out;
The data monitoring and sending module is used for receiving the data demand plan, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, monitoring newly added day base data in a multithreading mode according to a monitoring mechanism which is found to be accessed, and sending monitored day base data information to the data scheduling and processing module in a message driving mode;
And carrying out unified scheduling processing on all acquired day-base data by utilizing the data scheduling processing module, and feeding back the processed normalized multi-source data response to the data acquisition message processing module.
According to one aspect of the invention, the data requirements include data sources of the same network segment and their data requirements and data sources of different network segments and their data requirements.
According to one aspect of the present invention, in a process of planning a data acquisition plan according to a requirement for day-based data by using a data acquisition message processing module and issuing the data requirement plan, the method includes:
Calling a first network switching device to transfer data sources and data requirements of different network segments sent by the data acquisition message processing module to the data monitoring and sending module;
and directly transmitting the data sources and the data demands of the same network segment to the data monitoring and transmitting module by utilizing the data acquisition message processing module.
According to one aspect of the present invention, in the process of receiving the data demand plan by using the data monitoring and sending module, acquiring a corresponding data source according to a collection type, executing a data collection task, and multi-thread monitoring newly added day base data according to a monitoring mechanism which is found to be accessed, sending the monitored day base data message to the data scheduling processing module in a message driven mode, the method comprises the following steps:
Analyzing the message protocol and the file name of the monitored day-based data;
comparing the monitored file information with the monitored file information by utilizing historical collected data of a system cache and a database, if the file information is the same, not collecting the monitored file to the data scheduling processing module, otherwise, monitoring the file and collecting the monitored file to the data scheduling processing module;
and automatically identifying whether transmission across network segments is needed or not according to the requirements of the planning and distinguishing data service.
According to one aspect of the invention, the data scheduling processing module is used for carrying out unified scheduling processing on all kinds of acquired day-based data, and the method comprises the following steps:
According to the day-based data acquisition service requirement, setting a data acquisition strategy by using a mode selection module;
receiving a data driving message by using a data configuration module, and performing hierarchical monitoring according to multi-source day-base data of different levels;
the breakpoint continuous transmission module is utilized to pause or continue receiving the day-based data at any time in the transmission process of the day-based data;
The redundant data generated in the data acquisition process is identified and de-duplicated by utilizing a data de-duplication module;
Setting a cleaning strategy by using a timing cleaning module according to different acquisition service requirements, and cleaning redundant data at regular time;
Checking the quality and standardability of the acquired data by using a quality checking module, and eliminating invalid data;
the space-time screening module is utilized to carry out time screening and space screening on the effective data, and the data meeting the preset space-time conditions are selectively accessed;
Setting and selecting the type of the access data by using a type screening module;
Automatically matching the data types by using a format conversion module, and uniformly converting the format and the specification of the data;
And (3) utilizing an archiving storage module to archive and store the converted standard data according to a template corresponding to the data type matching, recording the stored data information, and cleaning the redundant data at fixed time or historical redundant data at one time to obtain the standardized multi-source day-to-day data.
According to one aspect of the invention, in the process of uniformly scheduling the collected various day-based data by using the data scheduling processing module and feeding back the processed normalized multi-source data response to the data collecting message processing module, the method comprises the following steps:
After the data monitoring and sending module receives the data demand planning, a second network switching device is called, and normalized multi-source data responses of different network segments processed by the data scheduling processing module are fed back to the data acquisition message processing module through a channel of the second network switching device;
And after the data monitoring and sending module receives the data demand planning, the data scheduling processing module directly feeds back the normalized multi-source data response of the same network segment to the data acquisition message processing module.
According to one aspect of the invention, the normalized multisource data response includes normalized multisource day-based data and scheduling process messages.
In a second aspect, the present invention further provides a multi-source heterogeneous day base data acquisition system for performing the multi-source heterogeneous day base data acquisition method based on a message driving mode, including:
The data acquisition message processing module is used for planning an acquisition plan according to the requirements on the day-based data and sending out data requirement planning;
a data monitoring and sending module for receiving the data demand plan, acquiring corresponding data sources according to the acquisition types, executing data acquisition tasks, monitoring newly added day base data in a multithread mode according to a monitoring mechanism which is found to be accessed, sending the monitored day base data information to a data scheduling and processing module in a message driven mode, and
The data scheduling processing module is used for carrying out unified scheduling processing on all acquired day data and feeding back the processed normalized multi-source data response to the data acquisition message processing module.
According to another aspect of the present invention, the data scheduling processing module includes:
the mode selection module is used for setting a data acquisition strategy according to the day-based data acquisition service requirement;
The data configuration module is used for receiving the data driving message and carrying out hierarchical monitoring according to the multi-source day-base data of different levels;
the breakpoint continuous transmission module is used for suspending, stopping or continuing to receive the day-based data at any time in the transmission process of the day-based data;
The data deduplication module is used for identifying and deduplicating redundant data generated in the data acquisition process;
The timing cleaning module is used for setting a cleaning strategy according to different acquisition business requirements and cleaning redundant data at fixed time;
The quality inspection module is used for inspecting the quality and normalization of the acquired data and eliminating invalid data;
the space-time screening module is used for carrying out time screening and space screening on the effective data and selectively accessing the data meeting the preset space-time conditions;
The type screening module is used for setting and selecting the type of the access data;
A format conversion module for automatically matching data types and uniformly converting the format and specification of the data, and
And the archiving and storing module is used for archiving and storing the converted standard data according to the corresponding templates matched with the data types, recording the stored data information, and cleaning the redundant data at fixed time or historical redundant data at one time to obtain the standardized multi-source day-based data.
According to another aspect of the invention, the system further comprises a first network switching device for transmitting the data sources and the data demands of the different network segments sent by the data acquisition message processing module to the data monitoring and sending module, and
And the second network switching equipment is used for feeding back the normalized multi-source data responses of different network segments processed by the data scheduling processing module to the data acquisition message processing module through a channel of the second network switching equipment after the data monitoring and sending module receives the data demand planning.
Compared with the prior art, the invention has the following beneficial effects:
According to the scheme of the invention, the multi-source heterogeneous day-base data acquisition method and system in a message driving mode are provided, and the day-base data of different types and sources are stably acquired by a message driving and closed loop processing method, so that the automatic access of the day-base data of multi-type satellites and various loads with different data types, formats, precision and time delays is realized.
According to the scheme of the invention, the automatic cross-network access can be realized for the day-based data of each network segment, and the receiving, processing, storing and displaying of the data are realized in a message-driven mode, so that the influence of network fluctuation on the stability of shared storage is solved.
According to one scheme of the invention, the system can realize continuous and uninterrupted stable operation of the system by performing multi-thread monitoring, data synchronous transmission and processing on the day-based data according to the data source and the category, including recognition, screening, conversion, deduplication and the like of the data format.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments will be briefly described below. It is apparent that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained from these drawings without inventive effort for a person of ordinary skill in the art.
FIG. 1 schematically illustrates a flow chart of a multi-source heterogeneous day-base data collection method based on a message-driven mode provided by an embodiment of the invention;
FIG. 2 schematically illustrates a network architecture diagram of a multi-source heterogeneous day-based data acquisition system based on a message-driven mode according to an embodiment of the present invention;
Fig. 3 schematically shows a process flow diagram of a data scheduling processing module according to an embodiment of the present invention.
Detailed Description
The description of the embodiments of this specification should be taken in conjunction with the accompanying drawings, which are a complete description of the embodiments. In the drawings, the shape or thickness of the embodiments may be enlarged and indicated simply or conveniently. Furthermore, portions of the structures in the drawings will be described in terms of separate descriptions, and it should be noted that elements not shown or described in the drawings are in a form known to those of ordinary skill in the art.
Any references to directions and orientations in the description of the embodiments herein are for convenience only and should not be construed as limiting the scope of the invention in any way. The following description of the preferred embodiments will refer to combinations of features, which may be present alone or in combination, and the invention is not particularly limited to the preferred embodiments. The scope of the invention is defined by the claims.
As shown in fig. 1, the method for collecting multi-source heterogeneous day base data based on a message driving mode disclosed by the embodiment of the invention comprises the following steps:
And 100, planning an acquisition plan according to the requirements of the space-based data by utilizing a data acquisition message processing module, and sending out data requirement planning.
Further, in this embodiment, the data source is divided into the data source of the same network segment and the data source of different network segments according to the location of the data source. Thus, the data requirements include data sources and data requirements for the same network segment and data sources and data requirements for different network segments. Aiming at multi-source space-based data of different network segments, the data sources and the data application service system are not in the same network segment, and the data types are various and the timeliness requirement is high. Compared with a data source network, the data application service system network has different access right requirements, realizes automatic data circulation by adding unidirectional network switching equipment, and maintains the singleness of data flow and the completeness of message interaction. Aiming at multisource day-based data of the same network segment, the data types of the monitoring access are various, naming rules and data formats are inconsistent, and the requirements on real-time performance and safety of data acquisition are high. Then, for different data source file paths under the same network segment, the embodiment of the invention adopts a multithreading task mode, ensures the peak processing capability, and is accessed point to point when data is found, and the acquisition response is driven by a message and the acquisition state is fed back. In the embodiment of the invention, a configurable design is adopted, the addition and the change of data types are supported, the data types are automatically identified independent of surface layer information such as file names, file suffixes and the like, historical redundant data are cleaned regularly, and the overmuch long-term operation data is prevented.
Then, in the step 100, a data acquisition plan is planned according to the requirement for the space-based data by using a data acquisition message processing module, and in a specific implementation process of data requirement planning, a first network switching device is called to transfer the data source and the data requirement thereof sent by the data acquisition message processing module to the data monitoring and sending module according to the data source conditions of different network segments. And directly transmitting the data sources and the data demands of the same network segment to the data monitoring and sending module by utilizing the data acquisition message processing module aiming at the condition of the data sources of the same network segment.
And 200, receiving the data demand plan by using a data monitoring and sending module, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, monitoring newly-added day-based data in a multithread mode according to a monitoring mechanism which is found to be accessed, and sending the monitored day-based data message to a data scheduling and processing module in a message-driven mode.
In this embodiment, the data monitoring and sending module receives the data demand plan, responds to events that change the original state, such as new addition of data and information files, file content or size change, and the like, and monitors newly added day-based data according to a monitoring mechanism that is found to be accessed in a multithreading mode, and sends the monitored day-based data information to the data scheduling processing module in a message-driven mode, so that the emergency of network fluctuation and data surge can be solved.
In some embodiments, in the step 200, the data monitoring and sending module is used to receive the data demand plan, obtain a corresponding data source according to a collection type, perform a data collection task, monitor newly added day-based data in a multithread manner according to a monitoring mechanism that is found to be accessed, send the monitored day-based data message to a specific implementation process of the data scheduling processing module in a message-driven mode, firstly analyze a message protocol and a file name of the monitored day-based data, issue the message protocol and the file name to the corresponding data source according to the analyzed collection type and information, and perform the data collection task. And comparing the monitored file information with the monitored file information by utilizing historical collected data of the system cache and the database, if the file information is the same, not collecting the monitored file to the data scheduling processing module, otherwise, monitoring the file and collecting the monitored file to the data scheduling processing module. And automatically identifying whether transmission across network segments is needed or not according to the requirements of the planning and distinguishing data service.
And 300, carrying out unified scheduling processing on all acquired day-base data by utilizing the data scheduling processing module, and feeding back the processed normalized multi-source data response to the data acquisition message processing module.
In some embodiments, as shown in fig. 3, the specific implementation process of performing unified scheduling processing on the collected various day base data by using the data scheduling processing module in step 300 includes:
The scheduling logic is mainly designed according to data requirements, receives all types of day-based data in network segments sent by a data monitoring and sending module, firstly sets a data acquisition strategy according to the requirements of day-based data acquisition services, utilizes a mode selection module to set data acquisition strategies, utilizes a data configuration module to receive data driving information and conduct hierarchical monitoring according to different levels of multi-source day-based data, utilizes a breakpoint continuous transmission module to pause or continue receiving the day-based data at any time in the transmission process of the day-based data, utilizes a type module to set and automatically select the types of access data, utilizes a type conversion module to automatically convert the types of access data into a format, and utilizes a data de-duplication module to identify and de-duplicate the redundant data according to a certain strategy when the repeated data is generated in the acquisition process, utilizes a timing clearing module to clear the redundant data according to different acquisition service requirements, then utilizes a quality check module to check the quality and standardability of the acquired data, eliminates invalid data, utilizes a screening module to conduct time screening and space screening on the effective data, utilizes the type module to set and automatically select the types of access data meeting preset space-time conditions, utilizes the type conversion module to automatically convert the types of access data into the data, and obtain the corresponding types of the redundant data after the data is subjected to the specification, and the data is subjected to the specification conversion according to the specification, and the data is stored after the data is subjected to the specification is subjected to the data conversion specification.
In this embodiment, in step 300, the data scheduling processing module performs unified scheduling processing on the collected various day-based data, and feeds back the normalized multi-source data response processed by the data collecting message processing module to the data collecting message processing module, and the data monitoring sending module calls the second network switching device and feeds back the normalized multi-source data response processed by the data scheduling processing module to the data collecting message processing module through the channel of the second network switching device after receiving the data demand planning according to the situation of the data sources of different network segments. And aiming at the condition of the data sources of the same network segment, the data monitoring and sending module directly feeds back normalized multi-source data response to the data acquisition message processing module through the data scheduling processing module after receiving the data demand planning.
Further, the normalized multi-source data response includes normalized multi-source day-base data and scheduling process messages. After the collected day base data is cleaned according to the scheduling processing logic, the embodiment sends normalized multisource day base data and scheduling processing process information to the data collection information processing module for closed loop processing.
The design process of the multi-source heterogeneous day-base data acquisition method of the embodiment implements the general, systematic and modularized ideas, builds a general platform, ensures the consistency and standardization of interfaces, and improves the acquisition capacity of multi-source day-base data to the greatest extent. Based on a high-automation principle, through a task message driving mode of centralized monitoring, all the acquisition modules are uniformly scheduled to work cooperatively, and even among different networks with physical isolation, closed-loop processing of message interaction can be realized, so that the effect of stable and efficient acquisition of multi-source day-based data is achieved.
The embodiment of the invention also discloses a multi-source heterogeneous day-base data acquisition system for executing the multi-source heterogeneous day-base data acquisition method based on the message driving mode, which is used for sending an acquisition process from 'to' normalized multi-source data response 'around' data demand planning, and adopts a source control and distributed acquisition deployment method as shown in figure 2.
The system comprises a data acquisition message processing module, a data monitoring and sending module, a data scheduling processing module and a data processing module, wherein the data acquisition message processing module is used for planning an acquisition plan according to the requirement on the day-based data and sending out the data requirement plan, the data monitoring and sending module is used for receiving the data requirement plan, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, monitoring newly-added day-based data in a multithread mode according to a monitoring mechanism which is found to be accessed, sending the monitored day-based data to the data scheduling processing module in a message driving mode, and the data scheduling processing module is used for carrying out unified scheduling processing on all acquired day-based data and feeding back normalized multi-source data responses processed by the data acquisition message processing module. Meanwhile, the data monitoring and sending module is also used for task management, real-time monitoring and tracking of task execution states, closed-loop processing, and collecting and monitoring of system software and hardware equipment states and data source states.
Corresponding to the specific process of the data scheduling processing module for scheduling the acquired multi-source day-based data of each network segment and each type shown in fig. 3, the data scheduling processing module comprises 10 functional modules including a mode selection module, a data configuration module, a breakpoint continuous transmission module, a data deduplication module, a timing cleaning module, a quality inspection module, a type screening module, a format conversion module and an archiving storage module. Wherein,
The mode selection module is used for setting a data acquisition strategy according to different day-based data acquisition service requirements, and can solve the coordination problems of factors such as data multisource, timeliness, source control, cooperative scheduling and the like.
The data configuration module is used for receiving the data driving message, carrying out hierarchical monitoring according to the multi-source day-based data of different levels, and particularly monitoring in different folders according to different day-based data types. According to the added types of the space-based information, the elastic expansion of the data acquisition capacity can be realized.
The breakpoint continuous transmission module is used for suspending, stopping or continuing to receive the day-based data at any time in the transmission process of the day-based data, and can flexibly control the data transmission archiving process.
The data deduplication module is used for identifying and deduplicating redundant data generated in the data acquisition process, and specifically, the module judges deduplication according to file names, file sizes, file MD5 values and the like, automatically identifies monitored repeated data and judges the monitored repeated data as redundant data. After the file name and the content are analyzed, comparing with the system cache and the historical database table, and selectively accessing the monitored file and the monitored file.
The timing cleaning module is used for setting a cleaning strategy according to different acquisition business requirements and cleaning redundant data at fixed time, wherein elements of the cleaning strategy relate to time, data category, flow type, mode type and the like.
The quality inspection module is used for inspecting the quality and standardability of the collected data and removing invalid data, and particularly, the module is used for checking naming rules and formats of the standardized space-based data, inspecting the integrity of the space-based data, recursively decompressing compressed format files, distinguishing compression types by reading byte headers when the actual formats of some compressed data are inconsistent with suffix names, and archiving according to types.
The space-time screening module is used for carrying out time screening and space screening on the effective data, selectively accessing the data meeting the preset space-time conditions, and specifically, selectively accessing the data meeting the preset space-time conditions by setting a time window and defining a region range.
And the type screening module is used for setting and selecting the type of the access data, and further, other types of data which are not in the set range are temporarily not accessed, so that the stable link resource can be maintained. And after the type setting is canceled, accessing.
The format conversion module is used for automatically matching the data types and uniformly converting the formats and specifications of the data, and particularly, in order to facilitate the organization and management of the data by a subsequent application system, the module automatically matches the data types and uniformly converts the naming formats, the data header files, the index files and the like of the multi-source space-based data so as to improve the operation efficiency of the data application system.
And the archiving and storing module is used for archiving and storing the converted standard data according to the corresponding templates matched with the data types, recording the stored data information, and cleaning the redundant data at fixed time or historical redundant data at one time to obtain the standardized multi-source day-based data. Specifically, the module matches corresponding templates for archiving according to different types of IDs of the multi-source data.
In the process of executing tasks by the data scheduling processing module, task information of each node is recorded in each log file, and timely alarming can be conducted aiming at abnormal conditions.
As shown in fig. 2, the network architecture of the multi-source heterogeneous day base data acquisition system based on the message driving mode disclosed in this embodiment further includes a first network device and a second network device. The data monitoring and sending module is used for receiving the data request, and the second network switching device is used for feeding back normalized multi-source data responses of different network segments processed by the data scheduling processing module to the data acquisition message processing module through a channel of the second network switching device after the data request is raised.
In this embodiment, the system may collect data from a multi-source day-based data information platform, and collect various day-based data information such as image data, signal data, measurement and feature data. According to the data types, the method can be divided into electronic data and imaging data, wherein the electronic data is large in receiving types and quantity, small in single file and wide in source, the imaging data is different in receiving format, large in single data quantity and different in real-time requirements. The method is characterized in that the method is used for stably collecting the day-based data of different types and sources by a message-driven and closed-loop processing method so as to realize the automatic access of the day-based data of multiple types of satellites and various loads with different data types, formats, precision and time delays. Meanwhile, according to the source and the category of the data, multi-thread monitoring, data synchronous transmission and processing are carried out, and continuous and uninterrupted stable operation of the system can be realized.
The sequence numbers of the steps related to the method of the present invention do not mean the sequence of the execution sequence of the method, and the execution sequence of the steps should be determined by the functions and the internal logic, and should not limit the implementation process of the embodiment of the present invention in any way.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather to enable any modification, equivalent replacement, improvement or the like to be made within the spirit and principles of the invention.
Claims (9)
1. A multi-source heterogeneous day-base data acquisition method based on a message driving mode comprises the following steps:
a data acquisition message processing module is utilized to plan an acquisition plan according to the requirements on the space-based data, and data requirement planning is sent out;
The data monitoring and sending module is used for receiving the data demand plan, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, monitoring newly added day base data in a multithreading mode according to a monitoring mechanism which is found to be accessed, and sending monitored day base data information to the data scheduling and processing module in a message driving mode;
The data scheduling processing module is utilized to perform unified scheduling processing on all types of acquired day-based data, and the processed normalized multi-source data response is fed back to the data acquisition message processing module;
The method for monitoring the newly added day base data by utilizing the data monitoring and sending module to receive the data demand plan, acquiring a corresponding data source according to the acquisition type, executing a data acquisition task, and monitoring the newly added day base data in a multithread mode according to a monitoring mechanism which is discovered to be accessed, and sending the monitored day base data message to the data scheduling and processing module in a message driving mode comprises the following steps:
Analyzing the message protocol and the file name of the monitored day-based data;
comparing the monitored file information with the monitored file information by utilizing historical collected data of a system cache and a database, if the file information is the same, not collecting the monitored file to the data scheduling processing module, otherwise, monitoring the file and collecting the monitored file to the data scheduling processing module;
and according to the data demand planning and distinguishing data service demand, automatically identifying whether cross-network-segment transmission is needed.
2. The method of claim 1, wherein the data requirements include data sources and data requirements for the same network segment and data sources and data requirements for different network segments.
3. The method according to claim 2, wherein in the process of planning data collection according to the requirement for the day-based data by using the data collection message processing module and issuing the data requirement plan, the method comprises:
Calling a first network switching device to transfer data sources and data requirements of different network segments sent by the data acquisition message processing module to the data monitoring and sending module;
and directly transmitting the data sources and the data demands of the same network segment to the data monitoring and transmitting module by utilizing the data acquisition message processing module.
4. The method of claim 2, wherein the data scheduling processing module performs unified scheduling processing on the collected various types of day-based data, and the method comprises the following steps:
According to the day-based data acquisition service requirement, setting a data acquisition strategy by using a mode selection module;
receiving a data driving message by using a data configuration module, and performing hierarchical monitoring according to multi-source day-base data of different levels;
the breakpoint continuous transmission module is utilized to pause or continue receiving the day-based data at any time in the transmission process of the day-based data;
The redundant data generated in the data acquisition process is identified and de-duplicated by utilizing a data de-duplication module;
Setting a cleaning strategy by using a timing cleaning module according to different acquisition service requirements, and cleaning redundant data at regular time;
Checking the quality and standardability of the acquired data by using a quality checking module, and eliminating invalid data;
the space-time screening module is utilized to carry out time screening and space screening on the effective data, and the data meeting the preset space-time conditions are selectively accessed;
Setting and selecting the type of the access data by using a type screening module;
Automatically matching the data types by using a format conversion module, and uniformly converting the format and the specification of the data;
And (3) utilizing an archiving storage module to archive and store the converted standard data according to a template corresponding to the data type matching, recording the stored data information, and cleaning the redundant data at fixed time or historical redundant data at one time to obtain the standardized multi-source day-to-day data.
5. The method according to claim 2, wherein the process of performing unified scheduling processing on the collected various day-based data by using the data scheduling processing module and feeding back the processed normalized multi-source data response to the data collection message processing module includes:
After the data monitoring and sending module receives the data demand planning, a second network switching device is called, and normalized multi-source data responses of different network segments processed by the data scheduling processing module are fed back to the data acquisition message processing module through a channel of the second network switching device;
And after the data monitoring and sending module receives the data demand planning, the data scheduling processing module directly feeds back the normalized multi-source data response of the same network segment to the data acquisition message processing module.
6. The method of any of claims 1-5, wherein normalizing the multi-source data response comprises normalizing the multi-source day-based data and scheduling a process message.
7. A multi-source heterogeneous day base data acquisition system for performing the multi-source heterogeneous day base data acquisition method based on a message driven mode as claimed in any one of claims 1-6, comprising:
The data acquisition message processing module is used for planning an acquisition plan according to the requirements on the day-based data and sending out data requirement planning;
a data monitoring and sending module for receiving the data demand plan, acquiring corresponding data sources according to the acquisition types, executing data acquisition tasks, monitoring newly added day base data in a multithread mode according to a monitoring mechanism which is found to be accessed, sending the monitored day base data information to a data scheduling and processing module in a message driven mode, and
The data scheduling processing module is used for carrying out unified scheduling processing on all acquired day data and feeding back the processed normalized multi-source data response to the data acquisition message processing module.
8. The system of claim 7, wherein the data scheduling processing module comprises:
the mode selection module is used for setting a data acquisition strategy according to the day-based data acquisition service requirement;
The data configuration module is used for receiving the data driving message and carrying out hierarchical monitoring according to the multi-source day-base data of different levels;
the breakpoint continuous transmission module is used for suspending, stopping or continuing to receive the day-based data at any time in the transmission process of the day-based data;
The data deduplication module is used for identifying and deduplicating redundant data generated in the data acquisition process;
The timing cleaning module is used for setting a cleaning strategy according to different acquisition business requirements and cleaning redundant data at fixed time;
The quality inspection module is used for inspecting the quality and normalization of the acquired data and eliminating invalid data;
the space-time screening module is used for carrying out time screening and space screening on the effective data and selectively accessing the data meeting the preset space-time conditions;
The type screening module is used for setting and selecting the type of the access data;
A format conversion module for automatically matching data types and uniformly converting the format and specification of the data, and
And the archiving and storing module is used for archiving and storing the converted standard data according to the corresponding templates matched with the data types, recording the stored data information, and cleaning the redundant data at fixed time or historical redundant data at one time to obtain the standardized multi-source day-based data.
9. The system of claim 7, further comprising a first network switching device for transferring data sources of different network segments and their data requirements from said data acquisition message processing module to said data listening and sending module, and
And the second network switching equipment is used for feeding back the normalized multi-source data responses of different network segments processed by the data scheduling processing module to the data acquisition message processing module through a channel of the second network switching equipment after the data monitoring and sending module receives the data demand planning.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211469130.0A CN115833913B (en) | 2022-11-22 | 2022-11-22 | Multi-source heterogeneous space-based data acquisition method and system based on message-driven mode |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211469130.0A CN115833913B (en) | 2022-11-22 | 2022-11-22 | Multi-source heterogeneous space-based data acquisition method and system based on message-driven mode |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115833913A CN115833913A (en) | 2023-03-21 |
| CN115833913B true CN115833913B (en) | 2025-02-11 |
Family
ID=85530366
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211469130.0A Active CN115833913B (en) | 2022-11-22 | 2022-11-22 | Multi-source heterogeneous space-based data acquisition method and system based on message-driven mode |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115833913B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106685516A (en) * | 2017-02-06 | 2017-05-17 | 北京军盛科技发展有限公司 | Data transmission system based on the space-based communication |
| CN110943772A (en) * | 2019-10-31 | 2020-03-31 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | A Time Division Duplex Time Slot Scheduling Method for Large Span Space-Based Data Links |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6643635B2 (en) * | 2001-03-15 | 2003-11-04 | Sagemetrics Corporation | Methods for dynamically accessing, processing, and presenting data acquired from disparate data sources |
| CN105099738B (en) * | 2014-05-13 | 2019-08-02 | 中兴通讯股份有限公司 | Data capture method, apparatus and system |
| CN111124679B (en) * | 2019-12-19 | 2023-11-21 | 南京莱斯信息技术股份有限公司 | Multi-source heterogeneous mass data-oriented time-limited automatic processing method |
-
2022
- 2022-11-22 CN CN202211469130.0A patent/CN115833913B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106685516A (en) * | 2017-02-06 | 2017-05-17 | 北京军盛科技发展有限公司 | Data transmission system based on the space-based communication |
| CN110943772A (en) * | 2019-10-31 | 2020-03-31 | 西南电子技术研究所（中国电子科技集团公司第十研究所） | A Time Division Duplex Time Slot Scheduling Method for Large Span Space-Based Data Links |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115833913A (en) | 2023-03-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN1959717B (en) | System and method for preprocessing mass remote sensing data collection driven by order form | |
| CN110245078A (en) | A kind of method for testing pressure of software, device, storage medium and server | |
| CN111858251B (en) | Data security audit method and system based on big data computing technology | |
| CN105488610A (en) | Fault real-time analysis and diagnosis system and method for power application system | |
| CN106649071B (en) | Method and device for reporting log in process | |
| CN111127250B (en) | Power data monitoring event analysis system and method | |
| EP3384391B1 (en) | Real-time change data from disparate sources | |
| CN116777182B (en) | Task dispatch method for semiconductor wafer manufacturing | |
| CN112417050A (en) | Data synchronization method and device, system, storage medium and electronic device | |
| CN119316380B (en) | A concentrator efficient data collection optimization method and system | |
| CN110955551B (en) | Intelligent fault diagnosis device based on tomcat middleware | |
| CN115833913B (en) | Multi-source heterogeneous space-based data acquisition method and system based on message-driven mode | |
| CN120216194A (en) | Real-time data stream processing method, device, electronic device and storage medium | |
| CN113704203B (en) | A log file processing method and device | |
| CN112260902B (en) | Network equipment monitoring method, device, equipment and storage medium | |
| CN108595120A (en) | A kind of scada near-realtime datas storage method and system | |
| KR100992417B1 (en) | Monitoring method for unified management system, system for the same, and recording medium storing a program thereof | |
| CN115718690B (en) | Data accuracy monitoring system and method | |
| CN112448840A (en) | Communication data quality monitoring method, device, server and storage medium | |
| CN116011972A (en) | Data processing system, method, electronic equipment and computer storage medium | |
| CN115460072A (en) | Log processing system integrating log collection, analysis, storage and service | |
| CN116126619A (en) | Method and device for processing database node information | |
| CN117609315B (en) | Data processing method, device, equipment and readable storage medium | |
| CN117827442A (en) | ELK-based GPU cluster data acquisition and analysis platform | |
| CN119719063A (en) | Linux user and program-level log automatic rotation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |