CN115151900A - Method and apparatus for filling in missing industrial longitudinal data - Google Patents

Abstract

A method, apparatus, system, and computer-readable medium for populating missing industrial longitudinal data is presented. Considering the slices as a whole, in contrast to current linear regression or interpolation, also takes into account the trend of the slices over time, in such a way that missing data can be filled in a more meaningful way and the true physical state is reflected.

Description

Method and apparatus for filling in missing industrial longitudinal data

technical field

The present invention relates to the technology of industrial data processing, and more particularly, to a method, apparatus and computer-readable storage medium for filling in missing industrial longitudinal data.

Background technique

Industrial data are widely used in industrial fields such as condition monitoring of systems and devices, predictive maintenance, and the like. Some industrial data is time series data. For example, data on the load rate of a grid may be collected at separate points in time and vary over time of day.

Furthermore, we can find that data from grids can share similar patterns over different days, which can indicate different working patterns of electricity consumers. In such cases, the time series data collected from the grid can be presented as longitudinal data, where each slice is an observation for the corresponding day, representing the loading rate of the grid, as shown in FIG. 1 . Due to the periodic nature of gridded data, we can represent the periodicity in the form of longitudinal data. For example, each slice (or instance) is a daily run of data, and all slices are arranged in chronological order. Here, missing data are defined as missing slices, as shown in Figure 1, where several slices from May 2015 to July 2015 are missing.

Missing data occurs when data values are not stored or collected and can have a significant impact on conclusions drawn from the data. Missing data are common and certainly common in longitudinal data.

Various approaches have been proposed to fill in missing data for longitudinal data. There are also general missing data imputation methods that can be used for time series data (non-longitudinal data), eg, interpolation. However, to the best of our knowledge, there is no method for dealing with longitudinal data where each slice is a piece of time series data rather than a general multidimensional feature.

SUMMARY OF THE INVENTION

In this disclosure, we propose a solution for filling missing data in longitudinal data in the industrial domain, where each slice is time series data. Compared to current linear regression or interpolation, treating the slice as a whole and also considering the trend of the slice over time can fill in missing data in a more meaningful way and reflect the true physical state.

Embodiments of the present disclosure include methods and apparatus for filling in missing industrial longitudinal data.

According to a first aspect of the present disclosure, a method for filling in missing industrial longitudinal data is presented. The method comprises the steps of: collecting industrial longitudinal data, wherein the industrial longitudinal data includes missing slices, each slice corresponding to a collection time point; estimating an overall trend over time for all slices of the industrial longitudinal data; calculating a trend for each missing slice based on the overall trend value; for each missing slice, find at least one similar slice based on the trend value; fill each missing slice based on at least one similar slice.

According to a second aspect of the present disclosure, an apparatus for filling in missing industrial longitudinal data is presented. The apparatus comprises: a data collection module configured to collect industrial longitudinal data, wherein the industrial longitudinal data includes missing slices, each slice corresponding to a collection time point; a data processing module configured to: estimate all slices of the industrial longitudinal data over time The overall trend of time; the trend value of each missing slice is calculated based on the overall trend; for each missing slice, at least one similar slice is found based on the trend value; each missing slice is filled based on at least one similar slice.

According to a third aspect of the present disclosure, an apparatus for filling in missing industrial longitudinal data is presented. The apparatus comprises: at least one processor; at least one memory coupled to the at least one processor configured to perform the method according to the first aspect.

According to a fourth aspect of the present disclosure, there is presented a computer readable medium for filling in missing industrial longitudinal data. The computer-readable medium stores computer-executable instructions, wherein the computer-executable instructions, when executed, cause at least one processor to perform the method according to the first aspect.

In the case of the solution provided in this disclosure, each missing slice is populated as a whole based on the overall trend and existing data with similar trend values, compared to the currently used solution, estimating the overall trend of the slice of longitudinal data, Presents a more reasonable padding solution that can be widely used for periodic time series data.

Optionally, each slice of the industrial longitudinal data can be normalized before estimating the overall trend over time for all the slices of the industrial longitudinal data; then, it can be determined whether all the normalized slices of the industrial longitudinal data have the same shape, And if not, all slices of the industrial longitudinal data can be divided into parts, where the slices in each part have the same shape; and for each part with missing slices, the overall trend of the slices in the part can be estimated and can be based on The overall trend of the portion calculates a trend value for each missing slice, at least one similar slice for each missing slice can be found in the portion based on the trend value, and each missing slice can be filled based on the at least one similar slice. To ensure that the fill result is closer to the true state, you should first look for slices with the same shape for reference. However, slices with the same shape and significantly different amplitudes can be regarded as different shapes under the influence of the amplitude difference. To introduce more slices for reference, first, the effects of different amplitudes should be removed by normalization. If there are truly different shapes over time after normalization, slices with the same shape should be treated as separate parts in order to ensure the closest slice to be referenced. Existing slices can be selected from this section to fill in missing slices. With normalization and division by the shape of the slice, the fill result can be more accurate and closer to the true state. The normalization method can be customized based on the requirements of the consumer.

Optionally, the slices during the time period are determined to have the same shape if the difference in shape between all normalized slices during the time period is less than a predefined threshold.

Optionally, the trend value is the mean of the slice.

Description of drawings

The above-mentioned attributes and other features and advantages of the present technology, as well as the manner in which it is achieved, will become more apparent, and the present technology itself will be further enhanced by reference to the following description of embodiments of the present technology taken in conjunction with the accompanying drawings. Easy to understand, in the attached picture:

Figure 1 depicts industry longitudinal data and missing data.

2 depicts a block diagram of an apparatus for filling in missing industrial longitudinal data according to one embodiment of the present disclosure.

3A depicts a flowchart of a method for filling in missing industrial longitudinal data according to one embodiment of the present disclosure.

FIG. 3B depicts a flowchart of step S302.

FIG. 4 depicts normalized slices of the data in FIG. 1 .

5A and 5B depict the extracted trends from the slices in FIG. 1 .

FIG. 6 depicts the fill results using the solutions provided in this disclosure.

Reference number:

10. Equipment for filling in missing industrial longitudinal data

101. Data collection module

102, data processing module

103, at least one processor

104, at least one memory

105, communication module

30. Data processing program

31. Longitudinal data collected

300, Methods for Filling Missing Industrial Longitudinal Data

S301-S305, the steps of the method 300

S3021～S3023, sub-steps of S302

Detailed ways

In the following, the above-described features and other features of the present technology are described in detail. Various embodiments are described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It may be noted that the illustrated embodiments are intended to illustrate, but not to limit, the invention. It may be evident that such embodiments may be practiced without these specific details.

When introducing elements of various embodiments of the present disclosure, the articles "a/an" and "the" are intended to mean that there are one or more of the elements. The terms "comprising", "comprising" and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements.

Now, the present disclosure will hereinafter be described in detail by referring to FIGS. 2 to 6 .

2 depicts a block diagram of a device according to one embodiment of the present disclosure. The apparatus 10 for filling in missing industrial longitudinal data presented in this disclosure may be implemented as a network of computer processors to perform the following method 300 for filling in missing industrial longitudinal data presented in this disclosure. The device 10 may also be a single computer, as shown in FIG. 2, which includes at least one memory 104 including a computer-readable medium, such as random access memory (RAM). Device 10 also includes at least one processor 103 coupled with at least one memory 104 . Computer-executable instructions are stored in at least one memory 104 and, when executed by at least one processor 103, can cause at least one processor 103 to perform the steps described herein. The at least one processor 103 may include a microprocessor, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a state machine, and the like. Examples of computer readable media include, but are not limited to, floppy disks, CD-ROMs, magnetic disks, memory chips, ROM, RAM, ASICs, configured processors, all optical media, all magnetic tapes or other magnetic media, or the computer processor can Any other medium from which instructions are read. Also, various other forms of computer-readable media can transmit or carry the instructions to a computer, including routers, private or public networks, or other transmission devices or channels, both wired and wireless. Instructions may contain code from any computer programming language, including, for example, C, C++, C#, Visual Basic, Java, and JavaScript.

The at least one memory 104 shown in FIG. 2 may contain a data processing program 30 that, when executed by the at least one processor 103, causes the at least one processor 103 to perform the method 300 presented in this disclosure for filling in missing industrial longitudinal data . Longitudinal data 31 may also be stored in at least one memory 104 . These data may be received via the communication module 105 of the device 10 .

Data processing program 30 may include:

- a data collection module 101 configured to collect industrial longitudinal data 31;

- A data processing module 102 configured to process the collected industrial longitudinal data 31 .

Here, the industrial longitudinal data 31 includes missing slices, and each slice corresponds to a collection time point.

In detail, the data processing module 102 is configured to

- Estimate the general trend over time for all slices of industrial longitudinal data 31;

- Calculate the trend value for each missing slice based on the overall trend;

- for each missing slice, find at least one similar slice based on the trend value;

- Filling each missing slice based on at least one similar slice.

Optionally, the data processing module 102 is further configured to: normalize each slice of the industrial longitudinal data before estimating the overall trend of all the slices of the industrial longitudinal data over time; when estimating the overall trend of all the slices of the industrial longitudinal data over time; Determines whether all normalized slices of industrial longitudinal data have the same shape when the overall trend of For each part of , estimate the general trend of the slices in the part; when calculating the trend value for each missing slice based on the general trend, for each part with missing slices, calculate the trend value for each missing slice based on the general trend of the part; when For each missing slice, when looking for at least one similar slice based on the trend value, for each section with missing slices, look for at least one similar slice for each missing slice in the section based on the trend value; when filling each section based on at least one similar slice When a slice is missing, for each section with missing slices, each missing slice is filled based on at least one similar slice.

Optionally, the data processing module 102 is further configured to, when determining whether all normalized slices of the industrial longitudinal data have the same shape, if the difference in shape between all normalized slices during the time period is less than a predefined threshold, The slices during the time period are then determined to have the same shape.

Optionally, the trend value is the mean of the slice.

Details of data processing by the data processing module 102 will be described later with reference to FIGS. 3A and 3B .

Although the data collection module 101 , the data processing module 102 are described above as software modules of the data processing program 30 . Also, the data collection module and the data processing module may be implemented via hardware, such as an ASIC chip. The data collection module and the data processing module can be integrated into one chip, or implemented separately and electrically connected.

It should be mentioned that the present disclosure may encompass devices having different architectures than that shown in FIG. 2 . The above architecture is exemplary only and serves to explain the exemplary method 300 shown in FIGS. 3A and 3B .

Various methods in accordance with the present disclosure can be performed. An exemplary method 300 according to the present disclosure includes the following steps:

S301: Collect industrial longitudinal data, wherein the industrial longitudinal data includes missing slices, and each slice corresponds to a collection time point;

S302: Estimate the overall trend over time for all slices of industrial longitudinal data;

S303: Calculate the trend value of each missing slice based on the overall trend;

S304: For each missing slice, find at least one similar slice based on the trend value;

S305: Fill each missing slice based on at least one similar slice.

Optionally, prior to step S302, estimating an overall trend over time for all slices of the industrial longitudinal data, the method 300 may further comprise:

S301': Normalize each slice of industrial longitudinal data. 3B, step S302 may include the following sub-steps:

S3021: Determine whether all normalized slices of the industrial longitudinal data have the same shape, and if not, the procedure proceeds through sub-step S3022.

S3022: Divide all slices of industrial longitudinal data into sections, wherein the slices in each section have the same shape; and

S3023: For each section with missing slices, estimate the overall trend of the slices in the section.

Then step S303 may include: for each section with missing slices, calculating a trend value for each missing slice based on the overall trend of the section; step S304 may include: for each section with missing slices, searching for each section in the section based on the trend value At least one similar slice of a missing slice; and step S305 may include: for each portion having the missing slice, filling each missing slice based on the at least one similar slice.

Optionally, in sub-step S3021, if the shape difference between all normalized slices during the time period is less than a predefined threshold, it may be determined that the slices during the time period have the same shape.

Next, an exemplary embodiment is described taking data from the grid shown in FIG. 1 as an example.

As shown in Figure 1, grid power data in the form of longitudinal data was collected from January 2015 to June 2017. Several slices from May 2015 to June 2015 are missing. With the solution provided in this disclosure, missing slices can be filled with reference to data trends over time axis and existing slices.

The basic idea is to fill in missing data with the help of general trends and other slices. Optionally, in order to eliminate the influence of the magnitude difference on the shape judgment, in step S301 ′, normalization may be performed on the collected data, otherwise, slices that differ only in latitude may be regarded as having different shapes. Referring to Figure 1, the magnitude of the loading rate in winter (about January and February) is significantly lower than that in summer (about July, August, and September in the same year), while the shape of the slices is consistent ( From 0 o'clock to 24 o'clock, first keep low, then gradually rise, keep flat and fall).

Then in sub-step S3021, a comparison may be made between the normalized slices with respect to the shape of the normalized slices. Euclidean distance can be used to measure the difference in shape of slices. If the shape difference of the normalized slices over time is greater than a predefined threshold, then in sub-step S3022, all slices may be split into sections, in each of which all slices share a similar shape. Then in sub-step S3023, for each section (or for all slices, in the case where there is no significant shape difference in all slices), the overall trend of the slices over time may be calculated. Optionally, the mean for each slice can be the calculated trend value for the slice.

Next in step S303, for the missing slices in each section, several techniques can be applied to estimate the missing slices, such as polynomial curve fitting and a gaussian process. In steps S304 and S305, with the estimated trend values of the missing slices, the missing slices can be filled based on other slices with similar trend values.

In step S301', for each slice, we can use feature scaling as the normalization method:

where x is the original slice and x _normalized is the normalized slice, and all normalized slices of the data are depicted in FIG. 4 . Each curve in Figure 4 can be viewed as the shape of the corresponding slice. We can see from Figure 4 that almost all slices share the same shape. Therefore, the data does not have to be sliced into different parts. Otherwise, some techniques (eg, clustering) can be used to separate all slices so that slices in the same section have the same shape.

5A and 5B illustrate the extracted trends for each slice. Here, the mean (thick dots) of each slice is used as a trend. Referring to FIG. 5B, in steps S302 and S303, a Gaussian process may be applied to estimate the trend (dotted line) of missing slices. In cases where there are multiple sections because the shape of the slice is not constant, this step will be performed for each section separately.

In step S304, at least one existing slice with most similar trend values may be found. Here, using 2 existing slices slice _a and slice _b , the missing slice can be represented as:

where slice _a and slice _b are the two slices with the closest trend value to slice _missing , and trend _a , trend _b , and trend _missing are the trend values of the slice, respectively. The number of existing slices to be used to calculate missing slices may be considered according to poor application requirements.

Fill results and other methods of the present disclosure. Referring to Figure 6, in the middle of the figure, linear regression is used to fill in missing data. The dots on the slice are the dimensions, here there are 24 dimensions in the slice, each dimension representing a specific hour of the day. The percentage on the right is the value of the point on the slice in the range 0 to 1. For each dimension of each slice, missing values are filled separately, and we can see that the filled slices no longer make sense. However, techniques other than linear regression can also be applied here, which have the same disadvantages. In the bottom of the filled missing slices according to the present disclosure, which achieves more reasonable results, a sharp increase can be found on May 1, 2015, which is in line with the trend for the same period in 2016 and 2017. However, in the middle of the graph, the missing slices increase bit by bit, and the importance change information will be omitted using such methods.

Below are 2 use cases where the solutions provided in this disclosure can be employed.

Use Case 1: Condition Evaluation Manager for Transformers

Data is collected at the transformer and transmitted into a data management system. Following data processing and analysis, a transformer health report and load-displacement recommendations can be provided to consumers. The collected data will be incomplete due to many reasons and missing data imputation methods are required in the data processing and analysis section.

Use Case 2: Distributed Energy Systems

Various applications exist under the topic of distributed energy systems, eg, load balancing, peak avoidance, anti-theft, etc. All of these applications are based on continuous monitoring of associated devices, which has a low tolerance for missing data, making imputation methods an essential part of data processing.

Also provided in the present disclosure is a computer-readable medium storing executable instructions that, when executed by a computer, cause the computer to perform any of the methods presented in the present disclosure.

A computer program is being executed by at least one processor and performs any of the methods presented in this disclosure.

Although the present technology has been described in detail with reference to certain embodiments, it is to be understood that the present technology is not limited to those precise embodiments. Indeed, given this disclosure, which describes exemplary modes for practicing the invention, many modifications and variations can be made by those skilled in the art without departing from the scope and spirit of the invention. Accordingly, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All changes, modifications and variations that come within the meaning and range of equivalency of the claims are to be deemed to be within the scope of the claims.

Claims (10)

1. A method (300) for populating missing industrial longitudinal data, comprising:

-collecting (S301) industrial longitudinal data, wherein the industrial longitudinal data comprises missing slices, each slice corresponding to a collection time point;

-estimating (S302) a general trend of all slices of the industrial longitudinal data over time;

-calculating (S303) a trend value for each missing slice based on the overall trend;

-for each missing slice, finding (S304) at least one similar slice based on the trend value;

-filling (S305) each missing slice based on the at least one similar slice.

2. The method (300) of claim 1,

-before estimating (S302) a general trend of all slices of the industrial longitudinal data over time, the method further comprises: normalizing (S301') each slice of the industrial longitudinal data;

-estimating (S302) a general trend over time of all slices of the industrial longitudinal data, comprising:

-determining (S3021) whether all normalized slices of the industrial longitudinal data have the same shape, and if not, then

-dividing (S3022) all slices of the industrial longitudinal data into portions, wherein the slices in each portion have the same shape;

-for each portion with missing slices, estimating (S3023) a general trend of the slices in said portion;

-calculating (S303) a trend value for each missing slice based on the overall trend comprises: for each portion having a missing slice, calculating (S303) a trend value for each missing slice based on the overall trend for the portion;

-for each missing slice, finding (S304) at least one similar slice based on the trend value comprises: for each portion having missing slices, finding (S304) at least one similar slice of each missing slice in the portion based on the trend values;

-filling (S305) each missing slice based on the at least one similar slice comprises: for each portion having a missing slice, filling (S305) each missing slice based on the at least one similar slice.

3. The method (300) as claimed in claim 2, wherein determining (S3021) whether all normalized slices of the industrial longitudinal data have the same shape comprises: determining (S3021) that the slices during the time period have the same shape if the shape differences between all normalized slices during the time period are smaller than a predefined threshold.

4. The method (300) of claim 1, wherein the trend value is a mean of the slices.

5. An apparatus (10) for filling missing industrial longitudinal data, comprising:

-a data collection module (101) configured to collect industrial longitudinal data, wherein the industrial longitudinal data comprises missing slices, each slice corresponding to a collection time point;

-a data processing module (102) configured to:

-estimating the general trend of all slices of the industrial longitudinal data over time;

-calculating a trend value for each missing slice based on the overall trend;

-for each missing slice, finding at least one similar slice based on the trend values;

-filling each missing slice based on the at least one similar slice.

6. The apparatus (10) of claim 5,

-the data processing module (102) is further configured to: normalizing each slice of the industrial longitudinal data prior to estimating a general trend of all slices of the industrial longitudinal data over time;

-the data processing module (102) is further configured to, when estimating the general trend of all slices of the industrial longitudinal data over time:

-determining whether all normalized slices of the industrial longitudinal data have the same shape and, if not, then

-dividing all slices of the industrial longitudinal data into portions, wherein the slices in each portion have the same shape;

-for each portion with missing slices, estimating the general trend of the slices in said portion;

-the data processing module (102) is further configured to, when calculating a trend value for each missing slice based on the overall trend: for each portion having a missing slice, calculating a trend value for each missing slice based on the overall trend for the portion;

-the data processing module (102) is further configured to, when for each missing slice, find at least one similar slice based on a trend value: for each portion having a missing slice, finding at least one similar slice for each missing slice in the portion based on the trend values;

-the data processing module (102) is further configured to, when filling each missing slice based on the at least one similar slice: for each portion having a missing slice, filling each missing slice based on the at least one similar slice.

7. The device (10) of claim 6, wherein the data processing module (102) is further configured to, when determining whether all normalized slices of the industrial longitudinal data have the same shape: determining that the slices during a time period have the same shape if the shape difference between all normalized slices during the time period is less than a predefined threshold.

8. The apparatus (10) of claim 5, wherein the trend value is a mean of the slices.

9. An apparatus (10) for filling missing industrial longitudinal data, comprising:

-at least one processor (103);

-at least one memory (104), coupled to the at least one processor (103), configured to perform the method according to any one of claims 1 to 4.

10. A computer-readable medium for populating missing industrial longitudinal data, storing computer-executable instructions, wherein the computer-executable instructions, when executed, cause at least one processor to perform the method of any of claims 1 to 4.

Images