Disclosure of Invention
In view of the shortcomings of the prior art, the present invention is directed to a quality control system for GNSS/MET water vapor observations.
In order to achieve the above purpose, the present invention provides the following technical solutions:
The quality control system of the GNSS/MET water vapor observation data comprises an observation data acquisition module, a data quality prediction module, a data quality control module and a data observation optimization module;
The observation data acquisition module controls the GNSS/MET station to periodically acquire water vapor observation data, and marks the acquired water vapor observation data as actual acquisition observation data;
The data quality estimation module is used for acquiring predicted observation data of the current period node, acquiring an observation gap index based on actual acquisition observation data and the predicted observation data, and marking the actual acquisition observation data as quality control data or quality reliable data according to a comparison result of the observation gap index and an observation gap threshold index;
the data quality control module acquires quality control sequence values of various quality control means after the actual acquisition observation data are marked as quality control data, sequentially sorts all the quality control means according to the sequence of the quality control sequence values, and sequentially performs quality analysis on the quality control data according to the sorting sequence of the quality control means;
The data observation optimization module is used for periodically acquiring the water vapor observation measurement value of the GNSS/MET station and judging whether to optimize the observation quality of the water vapor observation data of the GNSS/MET station based on the comparison result of the water vapor observation measurement value and the water vapor observation measurement threshold value.
Further, the prediction observation data of the current period node is obtained, specifically, a water vapor observation prediction model of the current season is obtained, the current period node is used as input data of the water vapor observation prediction model, and the water vapor observation prediction model is output to obtain the prediction observation data of the current period node.
Further, an observation gap index is obtained based on actual acquisition observation data and prediction observation data, specifically, the actual acquisition observation data and the prediction observation data are preprocessed, the preprocessed actual acquisition observation data and the preprocessed prediction observation data are packed into an observation comparison group, a water vapor observation gap model is obtained, the observation comparison group is used as input data of the water vapor observation gap model, and the water vapor observation gap model is output to obtain the observation gap index.
Further, according to the comparison result of the observation gap index and the observation gap threshold index, marking the actual acquisition observation data as quality control data or reliable quality data, specifically, setting the observation gap threshold index, marking the actual acquisition observation data as quality control data when the observation gap index is larger than the observation gap threshold index, and marking the actual acquisition observation data as reliable quality data when the observation gap index is smaller than or equal to the observation gap threshold index.
Further, the quality control sequence value of the quality control mode is obtained by obtaining all quality analysis records generated by the same quality control means before the current time of the system, sequentially sequencing all quality analysis records according to the sequence of the quality analysis time, performing difference calculation on quality analysis indexes of two adjacent quality analysis records after sequencing, taking absolute values to obtain quality analysis oscillation values, setting a quality analysis oscillation threshold, increasing one quality analysis oscillation quantity when the quality analysis oscillation value is greater than or equal to the quality analysis oscillation threshold, not performing processing when the quality analysis oscillation value is smaller than the quality analysis oscillation threshold, marking the quality analysis oscillation quantity as Cjbs, summing the quality analysis indexes of two adjacent quality analysis records after sequencing to obtain a quality analysis continuous value, setting the quality analysis continuous threshold, increasing one quality analysis continuous quantity when the quality analysis continuous value is greater than or equal to the quality analysis continuous threshold, not performing processing when the quality analysis continuous value is smaller than the quality analysis continuous threshold, marking the quality analysis continuous quantity as Pytk by utilizing a quality analysis continuous formulaAnd obtaining a quality control sequence value GTm of the quality control mode, wherein h1 is a quality analysis oscillation quantity coefficient, and h2 is a quality analysis continuous quantity coefficient.
Further, quality control data are sequentially subjected to quality analysis according to the sorting sequence of the quality control means, specifically, after each pair of quality control data is subjected to quality analysis of one quality control means, a quality analysis record of the quality control means is generated, the quality analysis record comprises a quality analysis index and quality analysis time, and when each quality analysis record is generated, the quality analysis index of the quality analysis record is obtained, and a quality analysis limit index is set;
When the quality analysis index is larger than or equal to the quality analysis limit index, the quality control means is used for processing the quality control data, after the processing is completed, an observation gap index is obtained based on the processed quality control data and the predicted observation data, when the observation gap index is larger than or equal to the observation gap threshold index and the quality control data does not finish the processing of all the quality control means, the quality analysis of the next quality control means is carried out on the processed quality control data according to the sorting order of the quality control means, when the observation gap index is larger than or equal to the observation gap threshold index and the quality control data is processed by all the quality control means, the quality control data is marked as quality reliable data, and when the observation gap index is smaller than the observation gap threshold index, the processed quality control data is marked as quality reliable data;
When the quality analysis index is smaller than the quality analysis limit index, the quality control means is subjected to quality analysis of the next quality control means according to the sorting order of the quality control means.
Further, the quality analysis index of the quality analysis record is obtained by obtaining a means analysis model of the quality control means, taking the quality control data as input data of the means analysis model, and outputting the means analysis model to obtain the quality analysis index.
Further, every T time length, acquiring a water vapor observation measurement value of the GNSS/MET station, setting a water vapor observation measurement threshold, optimizing the observation quality of water vapor observation data of the GNSS/MET station when the water vapor observation measurement value is more than or equal to the water vapor observation measurement threshold, and not processing when the water vapor observation measurement value is less than the water vapor observation measurement threshold.
Further, the water vapor observation measurement value of the GNSS/MET station is obtained by collecting all quality analysis records generated in the T time period, further obtaining the intervention value of each quality control means, setting the intervention threshold value of the means, marking the quality control means as an salient control means when the intervention value of the means is larger than or equal to the intervention threshold value of the means, marking the total quantity of the salient control means as Hte when the intervention value of the means is smaller than the intervention threshold value of the means, grouping all salient control means in pairs, summing the intervention values of the two salient control means in the same group to obtain the intervention sum value, setting the intervention sum value of the means, increasing the intervention superposition times once when the intervention sum value of the means is larger than or equal to the intervention sum threshold value of the means, marking the intervention superposition times as LGy when the intervention sum value of the means is smaller than the intervention sum threshold value of the means, and marking the intervention superposition times as LGy by using a formulaAnd obtaining a water vapor observation measurement value XKm of the GNSS/MET station, wherein u1 is a quantity coefficient of the salient control means, and u2 is an intervention superposition frequency coefficient.
Further, the intervention value of the quality control means is obtained by obtaining all the quality analysis records of the same quality control means, summing the quality analysis indexes of all the quality analysis records and taking the average value to obtain a quality analysis average index Fcz, setting a quality analysis boundary index, marking the quality analysis record as a protruding quality record when the quality analysis index of the quality analysis record is greater than or equal to the quality analysis boundary index, marking the total number of protruding quality records as Etp when the quality analysis index of the quality analysis record is less than the quality analysis boundary index without processing, sequentially sorting all the protruding quality records according to the sequence of the quality analysis time, calculating the time difference of the quality analysis time of two adjacent protruding quality records after sorting to obtain a protruding quality difference, summing the time differences of all the protruding quality differences and taking the average value to obtain an average protruding quality difference Mhg, and using the formulaObtaining a measure intervention value Sdj of the quality control measure, wherein r1 is a quality analysis average index coefficient, r2 is a salient quality record quantity coefficient, and r3 is an average salient quality time difference coefficient.
Compared with the prior art, the invention has the following beneficial effects:
The method comprises the steps of setting an observation data acquisition module, a data quality prediction module and a data quality control module, predicting water vapor observation data through a water vapor observation prediction model, intelligently analyzing the difference between actual acquisition observation data and predicted observation data through a water vapor observation difference model, sequentially carrying out quality analysis on the actual acquisition observation data according to the sequence of quality control means when the difference exists, and carrying out corresponding quality control means after analyzing quality problems, so as to ensure that the actual acquisition observation data is analyzed and controlled by the quality control means with high quality hidden danger preferentially, and judging whether the predicted observation data has prediction deviation in time after the analysis and control of all the quality control means, thereby facilitating the prediction optimization of subsequent water vapor observation data;
The data observation optimization module is arranged, so that the water vapor observation measurement quality of the GNSS/MET station can be comprehensively and three-dimensionally analyzed based on the quality analysis record, and whether the observation quality of the water vapor observation data of the GNSS/MET station is required to be optimized or not can be timely judged.
Detailed Description
Referring to fig. 1 to 3, a quality control system of gnss/MET water vapor observation data includes an observation data acquisition module, a data quality estimation module, a data quality control module, and a data observation optimization module.
The observation data acquisition module is used for setting a water vapor observation period (the water vapor observation period is provided with a plurality of period nodes with equal time intervals, the length of the time intervals can be adjusted according to requirements), acquiring water vapor observation data (the acquisition range of the water vapor observation data is limited between the current period node and the previous period node) by the GNSS/MET station when the period nodes of the water vapor observation period are reached, and marking the acquired water vapor observation data as actual acquisition observation data.
The data quality prediction module is used for acquiring predicted observation data of the current period node, acquiring an observation gap index based on actual observation data and the predicted observation data, setting an observation gap threshold index, presetting the observation gap threshold index, and adjusting the size according to actual requirements, marking the actual observation data as quality control data when the observation gap index is larger than the observation gap threshold index, and marking the actual observation data as quality reliable data when the observation gap index is smaller than or equal to the observation gap threshold index (the quality reliable data can be used as reference data for training of a subsequent water vapor observation prediction model).
The method comprises the steps of obtaining predicted observation data of a current period node, specifically obtaining a water vapor observation prediction model of a current season, taking the current period node as input data of the water vapor observation prediction model, and outputting the water vapor observation prediction model to obtain the predicted observation data of the current period node.
The method comprises the steps of obtaining periodic nodes of k current seasons, constructing a neural network model, taking the periodic nodes of the k current seasons as training data of the neural network model, giving predictive observation data to each periodic node, dividing the training data into a training set and a verification set according to the proportion of 1:3, and carrying out neural network iterative training on the training set and the verification set to obtain the water vapor observation prediction model of the current seasons. The water vapor observation prediction model is used for predicting water vapor observation data of the corresponding period node.
The method comprises the steps of preprocessing actual acquisition observation data and predicted observation data (preprocessing modes include but are not limited to format conversion), packaging the preprocessed actual acquisition observation data and the preprocessed predicted observation data into an observation comparison group, obtaining a water vapor observation gap model, taking the observation comparison group as input data of the water vapor observation gap model, and outputting the water vapor observation gap model to obtain the observation gap index.
The water vapor observation gap model is obtained by obtaining i observation comparison groups, wherein actual acquisition observation data and prediction observation data in each observation comparison group can be real data or virtual set data, constructing a neural network model, taking the i observation comparison groups as training data of the neural network model, endowing the observation gap index to each observation comparison group, dividing the training data into a training set and a verification set according to the proportion of 2:3, and performing neural network iterative training on the training set and the verification set to obtain the water vapor observation gap model. The larger the value of the observation gap index is, the larger the difference between the actual observation data and the predicted observation data is.
The data quality control module is used for acquiring quality control sequence values of various quality control means after the actual observation data are marked as quality control data (the quality control means of the water vapor observation data are various and comprise but not limited to a data integrity control means, an internal consistency control means and a time consistency control means, wherein the data integrity control means are used for continuously and completely processing the water vapor observation data, the internal consistency control means are used for processing the logical rationality of the water vapor observation data, the time consistency control means are used for processing the data smoothness of the water vapor observation data), all the quality control means are sequentially sequenced according to the sequence of the quality control sequence values, the quality control data are sequentially subjected to quality analysis according to the sequencing sequence of the quality control means, the quality analysis record of the quality control means is generated after the quality analysis of one quality control means is performed on each pair of quality control data, the quality analysis record comprises a quality analysis index and a quality analysis time, the quality analysis limit index of the quality analysis record is acquired when one quality analysis record is generated, and the quality analysis limit index is set as a preset index, and the quality control data can be adjusted according to the actual requirements.
When the quality analysis index is greater than or equal to the quality analysis limit index, the quality control means is used for processing the quality control data (for example, the quality control data is processed by the data integrity control means, the processing modes of the corresponding means are conventional processing modes and do not belong to the application focus of the application, so that details are not needed), after the processing is finished, the observation gap index is obtained based on the processed quality control data and the predicted observation data (a steam observation gap model is used), when the observation gap index is greater than or equal to the observation gap threshold index and the quality control data is not processed by all the quality control means, the quality analysis of the next quality control means is performed on the processed quality control data according to the sorting order of the quality control means, when the observation gap index is greater than or equal to the observation gap threshold index and the quality control data is processed by all the quality control means, the quality control data is marked as reliable data (indicating that the predicted observation gap index has prediction deviation), and when the observation gap index is less than the observation gap threshold index, the processed quality control data is marked as reliable data.
When the quality analysis index is smaller than the quality analysis limit index, the quality control means is subjected to the quality analysis of the next quality control means according to the sorting order of the quality control means (if the quality control means has finished processing the quality control data, the quality analysis of the quality control means is not required, and the quality control data is marked as reliable quality data).
The quality analysis index of the quality analysis record is obtained by obtaining a means analysis model of the quality control means, taking the quality control data as input data of the means analysis model, and outputting the means analysis model to obtain the quality analysis index.
In order to facilitate analysis of different quality control means, the application constructs the different means analysis models in the same way, only needs to correspondingly replace training data of the different means analysis models, so the embodiment only provides a construction mode of the means analysis models of the data integrity control means, namely the construction mode of the means analysis models of the data integrity control means can be obtained in the same way, the construction mode of the means analysis models of the data integrity control means is as follows, L quality control data are obtained, a neural network model is constructed, L quality control data are used as training data of the neural network model, the quality analysis index is given to each quality control data, the training data are divided into a training set and a verification set according to the proportion of 1:1, neural network iterative training is carried out on the training set and the verification set, and after the iterative completion, the means analysis model of the data integrity control means is obtained. The value ranges of the quality analysis indexes of the different means analysis models are all the same, and are (0-1), the larger the value of the quality analysis index is, the more abnormal the data integrity of the quality control data is (the larger the quality analysis indexes of the rest quality control means are, the more abnormal the internal consistency of the quality control data is represented), the smaller the value of the quality analysis index is, the more normal the data integrity of the quality control data is represented (the rest quality control means are, the smaller the quality analysis index of the time consistency control means is, and the more normal the time consistency of the quality control data is represented).
The quality control sequence value of the quality control mode is obtained by acquiring all quality analysis records generated by the same quality control means before the current time of the system, sequentially sequencing all quality analysis records according to the sequence of the quality analysis time, carrying out difference calculation on quality analysis indexes of two adjacent sequenced quality analysis records and taking absolute values to obtain a quality analysis oscillation value, setting a quality analysis oscillation threshold value, wherein the quality analysis oscillation threshold value is a preset index, and can be adjusted according to actual requirements, when the quality analysis oscillation value is greater than or equal to the quality analysis oscillation threshold value, the quality analysis oscillation quantity is increased by one, when the quality analysis oscillation value is less than the quality analysis oscillation threshold value, no processing is carried out, the quality analysis oscillation quantity is marked as Cjbs, the quality analysis indexes of two adjacent sequenced quality analysis records are summed to obtain a quality analysis continuous value, the quality analysis continuous threshold value is set, the quality analysis continuous threshold value is the preset index, and can be adjusted according to actual requirements, when the quality analysis continuous value is greater than or equal to the quality analysis continuous threshold value, the quality analysis continuous value is increased by one, and when the quality analysis continuous value is less than the quality analysis continuous value, the quality analysis continuous value is not processed by the quality analysis continuous value is marked as Pytk by using a formulaThe quality control sequence value GTm of the quality control mode is obtained, wherein h1 is a quality analysis oscillation quantity coefficient, h2 is a quality analysis continuous quantity coefficient, the value of h1 is 0.95, and the value of h2 is 0.88.
The method comprises the steps of setting an observation data acquisition module, a data quality prediction module and a data quality control module, predicting water vapor observation data through a water vapor observation prediction model, intelligently analyzing the difference between actual acquisition observation data and predicted observation data through a water vapor observation difference model, sequentially carrying out quality analysis on the actual acquisition observation data according to the sequence of quality control means when the difference exists, and carrying out corresponding quality control means after analyzing quality problems, so as to ensure that the actual acquisition observation data is analyzed and controlled by the quality control means with high quality hidden danger preferentially, and timely judging whether the predicted observation data has prediction deviation after the analysis and control of all the quality control means, thereby being convenient for the prediction optimization of subsequent water vapor observation data.
The data observation optimization module is used for acquiring a water vapor observation measurement value of the GNSS/MET station every T time length, setting a water vapor observation measurement threshold, wherein the water vapor observation measurement threshold is a preset index, and can be adjusted according to actual requirements, when the water vapor observation measurement value is larger than or equal to the water vapor observation measurement threshold, the observation quality of the water vapor observation data of the GNSS/MET station is optimized (various optimization modes include, but not limited to, maintenance and replacement of an observation instrument, and the optimization mode is not a research and development focus of the application and is not repeated), and when the water vapor observation measurement value is smaller than the water vapor observation measurement threshold, the water vapor observation measurement value is not processed.
Acquiring all quality analysis records generated in a time length of T, further acquiring a tool intervention value of each quality control tool, setting a tool intervention threshold value, wherein the tool intervention threshold value is a preset index, adjusting the size according to actual requirements, marking the quality control tool as an outstanding control tool when the tool intervention value is more than or equal to the tool intervention threshold value, marking the total number of outstanding control tools as Hte when the tool intervention value is less than the tool intervention threshold value, grouping all outstanding control tools in pairs, summing the tool intervention values of two outstanding control tools in the same group to obtain tool intervention sum values, setting the tool intervention sum values, increasing the intervention superposition times once when the tool intervention sum values are more than or equal to the tool intervention sum values, marking the intervention superposition times as LGy when the tool intervention sum values are less than the tool intervention sum values, and utilizing a formulaAnd obtaining a water vapor observation measurement value XKm of the GNSS/MET station, wherein u1 is a quantity coefficient of the salient control means, u2 is an intervention superposition frequency coefficient, the value of u1 is 1.38, and the value of u2 is 1.19.
The intervention value of the quality control means is obtained by obtaining all quality analysis records of the same quality control means, summing the quality analysis indexes of all quality analysis records and taking the average value to obtain quality analysis average index Fcz, setting quality analysis boundary index (different from quality analysis limit index), which is a preset index, adjusting the size according to actual requirement, marking the quality analysis record as a protruding quality record when the quality analysis index of the quality analysis record is more than or equal to the quality analysis boundary index, marking the total number of protruding quality records as Etp when the quality analysis index of the quality analysis record is less than the quality analysis boundary index without processing, sequentially ordering all protruding quality records according to the sequence of quality analysis time, calculating the time difference of the quality analysis time of two adjacent protruding quality records after ordering (subtracting the quality analysis time of the adjacent front quality analysis time) to obtain the protruding quality difference, summing the average value of all protruding quality records and taking the average value to obtain the protruding quality difference Mhg by utilizing a formulaThe intervention value Sdj of the quality control means is obtained, wherein r1 is a quality analysis average index coefficient, r2 is a salient quality record quantity coefficient, r3 is an average salient quality time difference coefficient, the value of r1 is 0.62, the value of r2 is 0.55, and the value of r3 is 0.94.
The data observation optimization module is arranged, so that the water vapor observation measurement quality of the GNSS/MET station can be comprehensively and three-dimensionally analyzed based on the quality analysis record, and whether the observation quality of the water vapor observation data of the GNSS/MET station is required to be optimized or not can be timely judged.
The above formulas are all dimensionality removed and numerical calculation is carried out, and preset parameters in the formulas are set by those skilled in the art according to actual conditions.
The above embodiments may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with embodiments of the present application are produced in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center by wired or wireless means (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more sets of available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium may be a solid state disk.
It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. The storage medium includes a U disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.