WO2018126367A1

WO2018126367A1 - Data cleaning method and device

Info

Publication number: WO2018126367A1
Application number: PCT/CN2017/070190
Authority: WO
Inventors: 黄建华; 康宏
Original assignee: 上海温尔信息科技有限公司
Priority date: 2017-01-04
Filing date: 2017-01-04
Publication date: 2018-07-12

Abstract

A data cleaning method and device. The data cleaning method comprises: obtaining a time-domain sampled data sequence of a physical quantity (201); removing abnormal data in the data sequence according to the time domain characteristic of the physical quantity (202); and in time, uniformly processing continuous data segments in the data sequence, the abnormal data of which is removed(203). By means of the method, reliable and accurate time-domain sampled data can be obtained, and the accuracy of related analysis based on the obtained sampled data is improved.

Description

Data cleaning method and device

Technical field

The present application relates to the field of data processing technologies, and in particular, to a data cleaning method and apparatus.

Background technique

A wearable device is a portable device that can be worn directly on the user or integrated into the user's clothing or accessories. The wearable device can continuously collect relevant data of the user and upload the collected data to the server to implement data interaction.

Based on the large amount of data collected by the wearable device, the user's health index, behavioral habits, life preferences, and the like can be analyzed. Before using the data collected by the wearable device, the data can be denoised to remove noise in the data, improve the reliability of the data, and improve the accuracy of the analysis results.

Summary of the invention

A lot of data collected by the wearable device from the human body is limited by the complex scene of the human body and the problem of the device itself or the connection with the mobile device. Some wrong data may occur, such as data loss caused by device falling off, or data loss caused by device communication. Wrong data needs to be eliminated in time to avoid further analysis affecting analytical calculations. The inventors thought that before using the data collected by the wearable device, the data needs to be thoroughly cleaned to obtain clean and reliable data, and the accuracy of the correlation analysis based on the data is improved.

The embodiment of the present application provides a data cleaning method, including:

Obtaining a data sequence of a physical quantity time domain sample;

Removing abnormal data in the data sequence based on a time domain characteristic of the physical quantity;

In time, the continuous data segments in the data sequence after the abnormal data are removed are uniformly processed.

In an optional implementation manner, the step of removing the abnormal data includes: setting a time window based on a time domain characteristic of the physical quantity; and using the time window, dividing the data sequence into at least one data segment; respectively The abnormal data in the at least one data segment is removed.

In an optional implementation manner, the step of removing the abnormal data in the first data segment of the at least one data segment includes: removing data exceeding the data range in the first data segment; and/or removing The data in the first data segment whose volatility is greater than the fluctuation threshold.

In an optional implementation, the step of removing the data beyond the data range includes: calculating a mean and a variance of the data in the first data segment; setting the data range according to the mean and the variance a boundary and a lower boundary; removing data in the first data segment that is larger than the upper boundary and data smaller than the lower boundary.

In an optional implementation manner, the step of removing the data whose volatility is greater than the fluctuation threshold includes: calculating a differential of the data in the first data segment; and removing an absolute value of the differential in the first data segment to be greater than a differential Threshold data.

In an optional implementation manner, the step of uniformly processing the continuous data segment includes: identifying, according to the set time interval threshold, the continuous data segment from the data sequence after the abnormal data is removed; using data interpolation In a way, the continuous data segments are processed uniformly in time.

In an optional implementation, the step of identifying the continuous data segment includes: determining, according to the time interval threshold, a data interruption point in the data sequence after the abnormal data is removed; and identifying the data after removing the abnormal data A data segment in the data sequence that is disconnected by the data interruption point as the continuous data segment.

In an optional implementation, the step of uniformly processing the consecutive data segments includes: determining at least one uniformization time point corresponding to the continuous data segment; and interpolating the at least the data in the continuous data segment A data that equalizes the time point.

The embodiment of the present application further provides a data cleaning device, including:

An acquiring unit, configured to acquire a data sequence of a physical quantity time domain sample;

a removing unit, configured to remove abnormal data in the data sequence based on a time domain characteristic of the physical quantity;

A uniform processing unit for uniformly processing contiguous data segments in the data sequence after the abnormal data is removed in time.

In an optional implementation, the removing unit includes: a setting subunit, configured to set a time window based on a time domain characteristic of the physical quantity; and a dividing subunit, configured to use the time window to use the data sequence Dividing into at least one data segment; removing subunits for respectively removing abnormal data in the at least one data segment.

In an optional implementation, the removing subunit is specifically configured to: remove data exceeding the data range in the first data segment; and/or remove data in the first data segment whose volatility is greater than a fluctuation threshold.

In an optional implementation, the removing subunit is specifically configured to: calculate a mean and a variance of data in the first data segment; and set upper and lower boundaries of the data range according to the mean and variance And removing data larger than the upper boundary and data smaller than the lower boundary in the first data segment.

In an optional implementation, the removing subunit is specifically configured to: calculate a differential of data in the first data segment; and remove data in which the absolute value of the differential entropy in the first data segment is greater than a differential threshold.

In an optional implementation, the uniform processing unit includes: an identifying subunit, configured to identify the continuous data segment from the data sequence after the abnormal data is removed according to a set time interval threshold; The unit is configured to uniformly process the continuous data segment in time by using data interpolation.

In an optional implementation, the identifying subunit is specifically configured to: determine, according to the time interval threshold, a data interruption point in the data sequence after the abnormal data is removed; and identify the data sequence after the abnormal data is removed A piece of data that is disconnected by the data interruption point as the continuous data segment.

In an optional implementation, the interpolation subunit is specifically configured to: determine at least one homogenization time point corresponding to the continuous data segment; and interpolate the at least one homogenization by using data in the continuous data segment The data corresponding to the time point.

The embodiment of the present application further provides a computer storage medium, which stores the following program instructions:

a first program instruction for acquiring a data sequence of a physical quantity time domain sample;

a second program instruction, configured to remove abnormal data in the data sequence based on a time domain characteristic of the physical quantity;

The third program instruction is configured to uniformly process the continuous data segments in the data sequence after the abnormal data is removed in time.

The embodiment of the present application further provides an electronic device, including:

a memory configured to store a computer program;

a communication interface configured to implement communication between the electronic device and other devices;

A processor coupled to the memory and the communication interface is configured to execute the computer program for:

Obtaining, by the communication interface, a data sequence of a physical quantity time domain sampling;

In an optional implementation, when the processor removes the abnormal data, the method is specifically configured to: set a time window based on a time domain characteristic of the physical quantity; and use the time window to divide the data sequence into At least one data segment; respectively removing abnormal data in the at least one data segment.

In an optional implementation, when the processor removes the abnormal data, the method is specifically configured to: remove data exceeding the data range in the first data segment; and/or remove fluctuations in the first data segment Data with a rate greater than the fluctuation threshold.

In an optional implementation, when the processor removes the data that is out of the data range, the processor is specifically configured to: calculate a mean and a variance of the data in the first data segment;

Setting upper and lower boundaries of the data range according to the mean and variance;

Data in the first data segment that is larger than the upper boundary and data that is smaller than the lower boundary are removed.

In an optional implementation manner, the processor removes the number of the volatility greater than a fluctuation threshold According to the time, it is specifically used to: calculate a differential of the data in the first data segment; and remove data in which the absolute value of the differential entropy in the first data segment is greater than a differential threshold.

In an optional implementation manner, when the processor uniformly processes the consecutive data segments, the method is specifically configured to: identify the continuous data from the data sequence after the abnormal data is removed according to a set time interval threshold Data segment; using data interpolation, uniformly processing the continuous data segment in time.

In an optional implementation, when the processor identifies the continuous data segment, the method is specifically configured to: determine, according to the time interval threshold, a data interruption point in the data sequence after the abnormal data is removed; In the data sequence after the abnormal data is removed, the data segment disconnected by the data interruption point is used as the continuous data segment.

In an optional implementation, when the processor uniformly processes the consecutive data segments, the method is specifically configured to: determine at least one homogenization time point corresponding to the continuous data segment; and utilize data in the continuous data segment And interpolating data corresponding to the at least one homogenization time point.

In the embodiment of the present application, in combination with the data sampling scenario, considering the physical and mathematical characteristics of the data itself, the data sequence of the physical quantity time domain sampling, the abnormal data is removed based on the time domain characteristics of the physical quantity, and uniformly processed in time. The continuous data segment in the data sequence after the abnormal data is removed, the data sequence of the time domain sampling is cleaned, and finally the reliable and accurate sampling data is obtained, thereby improving the accuracy of the correlation analysis based on the sampled data.

DRAWINGS

The drawings described herein are intended to provide a further understanding of the present application, and are intended to be a part of this application. In the drawing:

1 is a schematic structural diagram of a data cleaning system according to an embodiment of the present application;

2 is a schematic flowchart of a data cleaning method according to another embodiment of the present application;

3 is a schematic flowchart of a data cleaning method according to another embodiment of the present application;

4 is a schematic flowchart of a data cleaning method according to another embodiment of the present application;

FIG. 5 is a schematic flowchart diagram of a data cleaning method according to another embodiment of the present application;

FIG. 6 is a schematic flowchart diagram of a data cleaning method according to another embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram of a data cleaning apparatus according to another embodiment of the present disclosure;

FIG. 8 is a schematic structural diagram of an electronic device according to another embodiment of the present disclosure.

detailed description

The technical solutions of the present application will be clearly and completely described in the following with reference to the specific embodiments of the present application and the corresponding drawings. It is apparent that the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

The data cleaning method provided by the embodiment of the present application can be implemented based on the data cleaning system shown in FIG. 1, but is not limited thereto. As shown in FIG. 1, the data cleaning system includes: a data collection device 10 and a data cleaning device 20; and the data collection device 10 is communicatively coupled to the data cleaning device 20.

The data collection device 10 is configured to perform time domain sampling on a physical quantity to obtain sampling data of the physical quantity.

Optionally, the data collection device 10 can directly report the physical quantity of the sampling data to the data cleaning device 20, so that the data cleaning device 20 obtains the data sequence of the physical quantity time domain sampling. or,

Optionally, the data collection device 10 may store the sampled data of the physical quantity into a database for the data cleaning device 20 to acquire the data sequence of the time-domain samples of the physical quantity from the database.

For the data cleaning device 20, it is mainly used to acquire a data sequence of physical quantity time domain samples, and the data sequence is cleaned to obtain reliable and accurate data, and provide basic data for subsequent application or analysis.

The data collection device 10 and the data cleaning device 20 may be connected by wireless or wired network. In this embodiment, if the data collection device 10 passes the mobile network and the data cleaning device 20 Communication connection, the network standard of the mobile network may be any of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), WiMax, etc. One. In addition, the data collection device 10 can also be connected to the data cleaning device 20 by wireless communication methods such as Bluetooth, Wi-Fi, and infrared.

The physical quantity in this embodiment may be any physical quantity that supports time domain acquisition, and may be, for example, temperature or humidity.

The data collection device 10 in this embodiment may be any device capable of performing time domain acquisition on physical quantities, for example, various sensors. Taking the physical quantity as the temperature, especially the body temperature of the human body as an example, the data acquisition device may be a wearable device with a temperature sensor.

The data cleaning device 20 in this embodiment may be any device having data storage and data processing functions such as a server, a computer, a tablet computer, a smart terminal, and the like.

The following is a detailed description of the flow of the data cleaning method provided by the embodiment of the present application from the perspective of the data cleaning device 20 in conjunction with the data cleaning system shown in FIG.

FIG. 2 is a schematic flowchart diagram of a data cleaning method according to another embodiment of the present application. As shown in FIG. 2, the method includes:

201. Obtain a data sequence of a physical quantity time domain sample.

202. Remove abnormal data in the data sequence based on a time domain characteristic of the physical quantity.

203. Process the continuous data segments in the data sequence after the abnormal data is removed in time.

In step 201, the data cleaning device 20 acquires a data sequence of a physical quantity of time domain samples. For example, the data cleaning device 20 can acquire a data sequence formed by the data acquisition device 10 performing time domain sampling on the physical quantity. The data sequence includes sampled data of the physical quantities at different points in time and corresponding timestamps.

In an application scenario, the data collection device 10 performs time domain sampling on the physical quantity. For each sampled data, the data collection device 10 automatically adds a time stamp to it. Optionally, the data collection device 10 reports the time-stamped sample data to the data cleaning device 20 for the data cleaning device 20 to obtain the time-domain sampled data sequence of the physical quantity. Alternatively, the data collection device 10 stores the time-stamped sample data to a database for the data cleaning device 20 to obtain the physical quantity from the database. Time-domain sampled data sequence.

In another application scenario, the data collection device 10 begins at a specified point in time, sampling the physical quantities at even time intervals, with no time stamps for the sampled data. Optionally, the data collection device 10 reports the sample data without time stamp to the data cleaning device 20, and the data cleaning device 20 adds a time stamp to the sample data to obtain a data sequence of the time domain sample of the physical quantity. Alternatively, the data collection device 10 stores the sample data without time stamps in a database. In the storage process, the sample data is time stamped for the data cleaning device 20 to acquire the time domain samples of the physical quantity from the database. Data sequence.

In step 202, the inventors of the present application have found that the physical quantity generally has a certain time domain characteristic from the physical and mathematical characteristics of the physical quantity itself. For example, some physical quantities are continuous over a certain time frame, and the changes are gentle and do not suddenly jump or change rapidly. Taking human body temperature as an example, under normal circumstances, the body temperature of the human body is continuous and does not suddenly jump; if the body temperature data of sudden jump in the actual collection is abnormal, it should belong to the abnormal situation in the collection process, not the body temperature of the sampling object. The jump has changed. Of course, the body temperature change is relatively flat. Generally, the body temperature change will not exceed 0.05 degrees per second. If the actual body temperature data changes exceed this range, it should be an abnormal situation in the collection process, not the measurement object. Body temperature really changed at this rate.

The above enumerated cases are unreasonable for the sampled data itself and cannot be identified using conventional noise filtering methods. Based on the above findings, the data cleaning device 20 removes the abnormal data in the data sequence acquired in step 201 based on the time domain characteristics of the physical quantity. The data sequence after the abnormal data is removed includes data conforming to the time domain characteristics of the physical quantity, and the data is reliable and accurate.

In step 203, considering step 202 to remove the abnormal data in the data sequence, the data sequence may not be continuous in time, no longer uniform, and overall is inconvenient to use, but the continuous data segments in the data sequence still have certain use. Value. In this regard, the data cleaning device 20 uniformly processes the continuous data segments in the data sequence after the removal of the abnormal data in time to provide a reliable, temporally continuous and uniform continuous data segment for subsequent use. The continuous data segment refers to a data sequence in which the abnormal data is removed, and the time interval corresponding to all adjacent data is less than a preset. The data segment of the time interval threshold.

The value of the time interval threshold may be different according to the application scenario and the physical quantity. This embodiment does not limit the value of the time interval threshold, and can be adaptively set.

In this embodiment, combined with the data sampling scenario, considering the physical and mathematical characteristics of the data itself, the data sequence of the physical quantity time domain sampling, the abnormal data is removed based on the time domain characteristics of the physical quantity, and is uniformly processed in time. The continuous data segment in the data sequence after the abnormal data realizes the cleaning of the data sequence of the time domain sampling, and finally obtains reliable and accurate sampling data, thereby improving the accuracy of correlation analysis based on the sampled data.

In the above embodiment or the following embodiments, as shown in FIG. 3, removing the abnormal data in the data sequence based on the time domain characteristics of the physical quantity may include the following steps:

2011. Set a time window based on the time domain characteristics of the physical quantity.

2022. Using a time window, divide the data sequence into at least one data segment.

2023. Remove abnormal data in at least one data segment.

In step 2021, a time window is set, which reflects the time domain characteristics of the physical quantity, which is simply a characteristic of the physical quantity changing with time.

For example, in order to continuously collect human body temperature, in general, the body temperature does not change more than 0.5 degrees within 3 minutes, and according to this characteristic, the time window can be set to 3 minutes. This means that in the body temperature data within 3 minutes, the body temperature data that changes by more than 0.5 degrees is abnormal data.

For another example, in order to continuously collect the human heart rate, in general, the human heart rate does not change more than 15 times within 10 seconds, and according to this characteristic, the time window can be set to 10 seconds. This means that in the heart rate data within 10 seconds, the heart rate data that has changed more than 15 times is abnormal data.

In step 2022, based on the time window set in step 2021, the data sequence can be divided into at least one data segment, the length of time of each data segment being the length of the time window.

Optionally, the data sequence is divided into at least one data segment by using a time window, and at least one data segment does not overlap. Further, if the length of the last data segment in the data sequence is less than the length of the time window, but the ratio of the length of time to the time window is greater than or equal to the specified The ratio, for example greater than 1/3, preserves the last data segment as a separate data segment. Conversely, if the length of the last data segment in the data sequence is less than the length of the time window, and the ratio of the length of time to the time window is less than a specified ratio, such as less than 1/3, meaning that the last data segment is less than the required time window 1/3, The last data segment is then merged into the most recent time period data. For example, if the data sequence corresponds to 12:00:00-13:35:00 and the time window is 30 minutes, the data in 12:00:00-12:30:00 can be divided into a data segment. The data in 12:30:00-13:00:00 is divided into a data segment, and the data in 13:00:00-13:30:00 is divided into a data segment, and the data of the last 5 minutes is merged into 13 :00:00-13:30:00 time period.

In step 2023, the data segment divided by step 2022 is removed from the abnormal data.

Optionally, in step 2022, each data segment is divided, that is, the process proceeds to step 2023, the abnormal data in the data segment is removed, and then the process returns to step 2022. or,

Optionally, after all the data segments are divided in step 2022, the process proceeds to step 2023 to remove the abnormal data in each data segment one by one.

For the first data segment in the at least one data segment, the abnormal data may be removed in the following manner:

Removing data out of the data range in the first data segment; and/or

Data in the first data segment whose volatility is greater than the fluctuation threshold is removed.

Optionally, the step of removing the number of data segments beyond the data range in the first data segment may be: calculating a mean and a variance of the data in the first data segment, respectively recorded as μ and σ; setting data according to the mean and variance The upper and lower boundaries of the range are denoted as μ+ρσ and μ-ρσ, respectively; the data larger than the upper boundary μ+ρσ in the first data segment and the data smaller than the lower boundary μ-ρσ are removed, that is, only the first data segment is retained The data between the upper boundary μ+ρσ and the lower boundary μ-ρσ. Where ρ is a coefficient, which can be determined according to the application scenario and physical quantity.

Optionally, the step of removing the data in the first data segment whose volatility is greater than the fluctuation threshold may be: calculating a differential of the data in the first data segment; and removing the data in the first data segment whose absolute value is greater than the differential threshold. In this alternative embodiment, the volatility of the data is represented by differentiation, and correspondingly, the volatility threshold is embodied by a differential threshold. Can diversify all data in the data sequence The absolute value of the score is compared to the differential threshold, and the differential that generally exceeds the differential threshold will appear in pieces. The data whose absolute value of the differential is greater than the differential threshold is the data of the abnormality of the change, for example, it may be the initial stage of the acquisition, or the end of the collection, or the acquisition object is lost for some reason (for example, the temperature measuring device falls off). These data are generally abnormal data.

There are many ways to calculate the above differentials. The following examples illustrate:

For example, a differential calculation formula is: dT(n)=(T(n)-T(n-1))/(t(n)-t(n-1)), dT(1)=dT(2 ). In the differential calculation formula, n is a positive integer; dT(n) represents the differentiation of the nth time point; T(n) and T(n-1) represent the data of the nth time point and the n-1th, respectively. Data at time points; t(n) and t(n-1) represent the nth time point and the n-1th time point, respectively.

For another example, another differential calculation formula is: dT(n)=(T(n+1)-T(n))/(t(n+1)-t(n)), dT(end)=dT (end-1). In the differential calculation formula, n is a non-negative integer; dT(n) represents the differentiation of the nth time point; T(n) and T(n+1) represent the data of the nth time point and the n+th, respectively. Data at one time point; t(n) and t(n+1) represent the nth time point and the n+1th time point, respectively; dT(end) and dT(end-1) respectively represent the last time The differentiation of the point and the differentiation of the penultimate time point.

For another example, another differential calculation formula is: dT(n)=(T(n+1)-T(n-1))/(t(n+1)-t(n-1)), dT( 1) = dT(2), T(end) = dT(end-1). The differential calculation formula belongs to the central differential. In the differential calculation formula, n is a positive integer; dT(n) represents the differentiation of the nth time point; T(n-1) and T(n+1) respectively represent the nth - data of one time point and data of the n+1th time point; t(n-1) and t(n+1) respectively represent the n-1th time point and the n+1th time point; dT (end) and dT(end-1) represent the differentiation of the last time point and the differentiation of the penultimate time point, respectively.

It should be noted that the abnormal data in the second and third data segments in the at least one data segment may be removed in the same manner as the first data segment, but is not limited thereto.

In the above embodiment or the following embodiments, the step of uniformly processing the continuous data segments in time may be: identifying the continuous data segment from the data sequence after the abnormal data is removed according to the set time interval threshold; The data interpolation method uniformly processes the continuous data segments in time.

Optionally, the step of identifying the continuous data segment may be: according to the time interval threshold, The data interruption point in the data sequence after the abnormal data is removed; the data segment broken by the data interruption point in the data sequence after the abnormal data is removed is identified as a continuous data segment. Specifically, the difference between the timestamp corresponding to the adjacent data in the data sequence after the abnormal data is removed is compared with the time interval threshold, and the adjacent data with the difference of the timestamp greater than the time interval threshold is used as the interrupt data point. The interrupt data points are used as a segmentation point to divide the data sequence into at least one continuous data segment. In each successive data segment, the difference between the timestamps corresponding to the adjacent data is less than or equal to the time interval threshold.

Optionally, the step of uniformly processing the continuous data segment may be: determining at least one uniformization time point corresponding to the continuous data segment; and using data in the continuous data segment, interpolating data corresponding to the at least one uniformization time point.

In a specific implementation, if the homogenization time point is the same as the time stamp of a certain data in the continuous data segment, the data may be directly used as the data corresponding to the homogenization time point; if the homogenization time point is not related to any data in the continuous data segment If the time stamps are the same, the data corresponding to the homogenization time points can be interpolated and the data corresponding to the homogenization time points can be taken. The interpolation method may be linear interpolation, spline interpolation, or the like.

In the application scenario of collecting human body temperature, a data cleaning method is shown in FIG. 4, including:

401. The wearable device continuously collects body temperature and stores the collected body temperature data into a database.

Optionally, the wearable device can add a time stamp to the collected body temperature data. Or, in the process of storing to the database, time stamp data is added.

402. The data cleaning device acquires a data sequence corresponding to the body temperature of the human body from the database, and the data sequence includes a series of body temperature data.

403. The data cleaning device sets a time window based on a time domain characteristic of the physical quantity.

404. The data cleaning device divides the data sequence into at least one data segment by using a time window.

405. The data cleaning device calculates a mean and a variance of body temperature data in each of the at least one data segment.

406. The data cleaning device respectively determines the mean and variance of the body temperature data in each data segment. Set the upper and lower boundaries of the data range corresponding to each data segment.

407. The data cleaning device separately removes body temperature data larger than the upper boundary and body temperature data smaller than the lower boundary in each data segment.

408. The data cleaning device identifies the continuous data segment from the data sequence after the abnormal data is removed according to the set time interval threshold.

409. The data cleaning device adopts a data interpolation manner to uniformly process the continuous data segment in time.

In the application scenario of collecting human body temperature, another data cleaning method is shown in FIG. 5, including:

501. The wearable device continuously collects human body temperature, and stores the collected body temperature data into a database.

502. The data cleaning device acquires a data sequence corresponding to the body temperature of the human body from the database, and the data sequence includes a series of body temperature data.

503. The data cleaning device sets a time window based on a time domain characteristic of the physical quantity.

504. The data cleaning device divides the data sequence into at least one data segment by using a time window.

505. The data cleaning device calculates a differential of body temperature data in each of the at least one data segment.

506. The data cleaning device respectively removes body temperature data whose absolute value in each data segment is greater than a differential threshold.

507. The data cleaning device identifies the continuous data segment from the data sequence after the abnormal data is removed according to the set time interval threshold.

508. The data cleaning device adopts a data interpolation manner to uniformly process the continuous data segment in time.

In the application scenario of collecting human body temperature, another data cleaning method is shown in FIG. 6, which includes:

601. The wearable device continuously collects body temperature and stores the collected body temperature data in a database.

602. The data cleaning device acquires a data sequence corresponding to the body temperature of the human body from the database, and the data sequence includes a series of body temperature data.

603. The data cleaning device sets a time window based on a time domain characteristic of the physical quantity.

604. The data cleaning device divides the data sequence into at least one data segment by using a time window.

605. The data cleaning device calculates a mean and a variance of body temperature data in each of the at least one data segment.

606. The data cleaning device sets upper and lower boundaries of the data range corresponding to each data segment according to the mean and variance of the body temperature data in each data segment.

607. The data cleaning device separately removes body temperature data larger than the upper boundary and body temperature data smaller than the lower boundary in each data segment.

608. The data cleaning device calculates a differential of body temperature data in each of the at least one data segment.

609. The data cleaning device respectively removes body temperature data whose absolute value in each data segment is greater than a differential threshold.

610. The data cleaning device identifies the continuous data segment from the data sequence after the abnormal data is removed according to the set time interval threshold.

611. The data cleaning device adopts a data interpolation manner to uniformly process the continuous data segment in time.

It is to be noted that the order of execution of the above steps 605-607 and steps 608-609 is not limited to the order described in the embodiment, and the operations described in steps 608-609 may be performed first, and the operations described in steps 605-607 may be performed. Wherein, the operations described in steps 605-607 are performed first, and the operations described in steps 608-609 are performed as a preferred embodiment.

In the above embodiment, in combination with the specific physical quantity of the human body temperature, considering the physical and mathematical characteristics of the body temperature data itself, the data sequence corresponding to the human body temperature first removes the abnormal data based on the time domain characteristics of the human body temperature, and identifies and removes the abnormal data. a contiguous segment of data in the subsequent data sequence, The continuous data segment is homogenized in time to obtain reliable and accurate body temperature data, which provides a good basic condition for subsequent analysis based on body temperature data, which is beneficial to improve the accuracy of subsequent analysis results.

It should be noted that the execution bodies of the steps of the method provided by the foregoing embodiments may all be the same device, or the method may also be performed by different devices. For example, the execution body of steps 201 to 203 may be device A; for example, the execution body of

steps

201 and 202 may be device A, the execution body of step 203 may be device B, and the like.

FIG. 7 is a schematic structural diagram of a data cleaning apparatus according to another embodiment of the present application. As shown in FIG. 7, the apparatus includes an acquisition unit 71, a removal unit 72, and a uniform processing unit 73.

The obtaining unit 71 is configured to acquire a data sequence of a physical quantity of time domain samples.

The removing unit 72 is configured to remove the abnormal data in the data sequence based on the time domain characteristic of the physical quantity.

The uniform processing unit 73 is configured to uniformly process the continuous data segments in the data sequence after the abnormal data is removed in time.

In an optional implementation, an implementation structure of the removing unit 72 includes:

Setting a subunit for setting a time window based on a time domain characteristic of the physical quantity;

Dividing a subunit for dividing the data sequence into at least one data segment by using the time window;

The subunit is removed for respectively removing abnormal data in the at least one data segment.

Further optionally, the removing subunit is specifically configured to: remove data exceeding the data range in the first data segment; and/or remove data in the first data segment whose volatility is greater than a fluctuation threshold.

Further, the removing subunit is specifically configured to: calculate a mean value and a variance of the data in the first data segment when removing data exceeding the data range in the first data segment; according to the mean value and the variance, Setting an upper boundary and a lower boundary of the data range; removing data larger than the upper boundary and data smaller than the lower boundary in the first data segment.

Further optionally, the removing subunit is specifically configured to: calculate a differential of the data in the first data segment when removing the data whose volatility is greater than the fluctuation threshold in the first data segment; The data in which the absolute value of the differential in the first data segment is greater than the differential threshold.

In an alternative embodiment, an implementation structure of the uniform processing unit includes:

a identifying subunit, configured to identify the continuous data segment from the data sequence after the abnormal data is removed according to the set time interval threshold;

The interpolation subunit is configured to uniformly process the continuous data segment in time by using a data interpolation method.

Further, the identifying subunit is specifically configured to: determine, according to the time interval threshold, a data interruption point in the data sequence after the abnormal data is removed; and identify the data sequence after the abnormal data is removed, A data segment in which the data break point is broken as the continuous data segment.

Further optionally, the interpolation subunit is specifically configured to: determine at least one homogenization time point corresponding to the continuous data segment; and interpolate data corresponding to the at least one homogenization time point by using data in the continuous data segment .

The data cleaning device provided in this embodiment may be used to perform the process provided by the foregoing method embodiments, and details are not described herein again.

The data cleaning device provided in this embodiment combines the data sampling scenario, and considers the physical and mathematical characteristics of the data itself, and the time-domain sampling data sequence of the physical quantity, the abnormal data is removed based on the time domain characteristics of the physical quantity, and in time, Uniformly processing the continuous data segments in the data sequence after the abnormal data is removed, thereby purifying the data sequence of the time domain sampling, and finally obtaining reliable and accurate sampling data, thereby improving the accuracy of correlation analysis based on the sampled data.

The internal function and structure of the data cleaning device are described above, as shown in FIG. 8. In practice, the data cleaning device can be implemented as an electronic device, including: a memory 81, a processor 82, and a communication interface 83.

The memory 81 is configured to store a computer program.

Additionally, the memory 81 can also be configured to store other various data to support operation on the electronic device. Examples of such data include instructions for any application or method operating on an electronic device, contact data, phone book data, messages, pictures, videos, and the like.

Memory 81 can be any type of volatile or non-volatile storage device or combination thereof Implementations such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory ( ROM), magnetic memory, flash memory, disk or optical disk.

The communication interface 83 is configured to implement communication between the electronic device and other devices, such as wired or wireless communication.

The electronic device can access a wireless network based on a communication standard such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication interface 83 receives broadcast signals or broadcast associated information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, communication interface 83 also includes a near field communication (NFC) module to facilitate short range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

A processor 82, coupled to the memory 81 and the communication interface 83, is configured to execute a computer program in the memory 81 for:

Obtaining a data sequence of a physical quantity of time domain samples through the communication interface 83;

In an optional implementation manner, the processor 82 is configured to: when the abnormal data is removed, to set a time window based on the time domain characteristic of the physical quantity; and use the time window to divide the data sequence into at least one data a segment; respectively removing abnormal data in the at least one data segment.

In an optional implementation, when the processor 82 removes the abnormal data in the first data segment of the at least one data segment, the processor 82 is specifically configured to: remove data in the first data segment that is out of the data range; and/or And removing data in the first data segment whose volatility is greater than a fluctuation threshold.

In an optional implementation, when the processor 82 removes the data beyond the data range, the processor 82 is specifically configured to: calculate a mean and a variance of the data in the first data segment; and set the data according to the average and the variance. An upper boundary and a lower boundary of the range; data larger than the upper boundary and smaller than the lower boundary in the first data segment are removed.

In an optional implementation, when the processor 82 removes data whose volatility is greater than the fluctuation threshold, Specifically, the method is: calculating a differential of the data in the first data segment; and removing data in which the absolute value of the differential in the first data segment is greater than a differential threshold.

In an optional implementation manner, when the processor 82 uniformly processes the consecutive data segments, the method is specifically configured to: identify, according to the set time interval threshold, the continuous data from the data sequence after the abnormal data is removed. Segment; using data interpolation, uniformly processing the continuous data segments in time.

In an optional implementation, when the processor 82 identifies the continuous data segment, the method is specifically configured to: determine, according to the time interval threshold, a data interruption point in the data sequence after the abnormal data is removed; In the data sequence after the abnormal data is removed, the data segment broken by the data interruption point is used as the continuous data segment.

In an optional implementation, when the processor 82 uniformly processes the consecutive data segments, the method is specifically configured to: determine at least one homogenization time point corresponding to the continuous data segment; and utilize data in the continuous data segment, Interpolating data corresponding to the at least one homogenization time point.

Further, as shown in FIG. 8, the electronic device further includes: a display 84, a power supply component 85, an audio component 86, and the like. Only some of the components are schematically illustrated in FIG. 8, and it is not meant that the client device includes only the components shown in FIG.

Display 84 includes a screen whose screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may sense not only the boundary of the touch or sliding action, but also the duration and pressure associated with the touch or slide operation.

A power supply assembly 85 provides power to various components of the electronic device. Power component 85 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for client devices.

The audio component 86 is configured to output and/or input an audio signal. For example, the audio component 86 includes a microphone (MIC) that is configured to receive an external audio signal when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in the memory 81 or transmitted via the communication interface 83. In some embodiments, the audio Component 86 also includes a speaker for outputting an audio signal.

The embodiment of the present application further provides a computer storage medium suitable for a computer program, where the computer storage medium stores the following program instructions:

When the above program instructions are executed, the process provided by the foregoing method embodiment can be implemented, and the data sequence of the time domain sampling is cleaned, and reliable and accurate data is obtained, which provides a good basic condition for subsequent analysis based on the sampled data. Improve the accuracy of subsequent analysis results.

Those skilled in the art will appreciate that embodiments of the present invention can be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware. Moreover, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the computer readable memory is stored in the computer readable memory. The instructions in the production result include an article of manufacture of the instruction device that implements the functions specified in one or more blocks of the flowchart or in a flow or block of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory, such as read only memory (ROM) or flash memory (flashRAM), in a computer readable medium. Memory is an example of a computer readable medium.

Computer readable media includes both permanent and non-persistent, removable and non-removable media. Information storage can be implemented by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transportable media can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary storage of computer readable media, such as modulated data signals and carrier waves.

It is also to be understood that the terms "comprises" or "comprising" or "comprising" or any other variations are intended to encompass a non-exclusive inclusion, such that a process, method, article, Other elements not explicitly listed, or elements that are inherent to such a process, method, commodity, or equipment. In the absence of more restrictions, elements defined by the phrase "including one..." are not excluded from the process, method, There are additional identical elements in the item or device.

Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment in combination of software and hardware. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

The above description is only an embodiment of the present application and is not intended to limit the application. Various changes and modifications can be made to the present application by those skilled in the art. Any modifications, equivalents, improvements, etc. made within the spirit and scope of the present application are intended to be included within the scope of the appended claims.

Claims

A data cleaning method, comprising:

Obtaining a data sequence of a physical quantity time domain sample;

Removing abnormal data in the data sequence based on a time domain characteristic of the physical quantity;

In time, the continuous data segments in the data sequence after the abnormal data are removed are uniformly processed.
The method according to claim 1, wherein the step of removing the abnormal data comprises:

Setting a time window based on a time domain characteristic of the physical quantity;

Using the time window, dividing the data sequence into at least one data segment;

The abnormal data in the at least one data segment is separately removed.
The method according to claim 2, wherein the step of removing the abnormal data for the first data segment of the at least one data segment comprises:

Removing data out of the data range of the first data segment; and/or

Data in the first data segment having a volatility greater than a fluctuation threshold is removed.
The method according to claim 3, wherein the step of removing the data beyond the data range comprises:

Calculating a mean and a variance of data in the first data segment;

Setting upper and lower boundaries of the data range according to the mean and variance;

Data in the first data segment that is larger than the upper boundary and data that is smaller than the lower boundary are removed.
The method according to claim 3, wherein the step of removing the data whose volatility is greater than the fluctuation threshold comprises:

Calculating a differential of data in the first data segment;

Data in which the absolute value of the differential in the first data segment is greater than a differential threshold is removed.
The method according to any one of claims 1 to 5, wherein the step of uniformly processing the continuous data segments comprises:

Identifying the continuous data segment from the data sequence after the abnormal data is removed according to the set time interval threshold;

The data segmentation method is used to uniformly process the continuous data segments in time.
The method according to claim 6, wherein the step of identifying the continuous data segment comprises:

Determining, according to the time interval threshold, a data interruption point in the data sequence after the abnormal data is removed;

A data segment that is disconnected by the data interruption point in the data sequence after the abnormal data is removed is identified as the continuous data segment.
The method according to claim 6, wherein the step of uniformly processing the continuous data segments comprises:

Determining at least one homogenization time point corresponding to the continuous data segment;

Using data in the continuous data segment, data corresponding to the at least one homogenization time point is interpolated.
A data cleaning device, comprising:

An acquiring unit, configured to acquire a data sequence of a physical quantity time domain sample;

a removing unit, configured to remove abnormal data in the data sequence based on a time domain characteristic of the physical quantity;

A uniform processing unit for uniformly processing contiguous data segments in the data sequence after the abnormal data is removed in time.
The apparatus according to claim 9, wherein said removing unit comprises:

Setting a subunit for setting a time window based on a time domain characteristic of the physical quantity;

Dividing a subunit for dividing the data sequence into at least one data segment by using the time window;

The subunit is removed for respectively removing abnormal data in the at least one data segment.
The apparatus according to claim 10, wherein the removing subunit is specifically configured to:

Removing data out of the data range of the first data segment; and/or

Data in the first data segment having a volatility greater than a fluctuation threshold is removed.
The apparatus according to any one of claims 9-11, wherein the uniform processing unit comprises:

a identifying subunit, configured to identify the continuous data segment from the data sequence after the abnormal data is removed according to the set time interval threshold;

The interpolation subunit is configured to uniformly process the continuous data segment in time by using a data interpolation method.
The device according to claim 12, wherein the identification subunit is specifically configured to:

Determining, according to the time interval threshold, a data interruption point in the data sequence after the abnormal data is removed;

A data segment that is disconnected by the data interruption point in the data sequence after the abnormal data is removed is identified as the continuous data segment.
The apparatus according to claim 12, wherein the interpolation subunit is specifically configured to:

Determining at least one homogenization time point corresponding to the continuous data segment;

Using data in the continuous data segment, data corresponding to the at least one homogenization time point is interpolated.
A computer storage medium, characterized in that the computer storage medium stores the following program instructions:

a first program instruction for acquiring a data sequence of a physical quantity time domain sample;

a second program instruction, configured to remove abnormal data in the data sequence based on a time domain characteristic of the physical quantity;

The third program instruction is configured to uniformly process the continuous data segments in the data sequence after the abnormal data is removed in time.
An electronic device, comprising:

a memory configured to store a computer program;

a communication interface configured to implement communication between the electronic device and other devices;

A processor coupled to the memory and the communication interface is configured to execute the computer program for:

Obtaining, by the communication interface, a data sequence of a physical quantity time domain sampling;

Removing abnormal data in the data sequence based on a time domain characteristic of the physical quantity;

In time, the continuous data segments in the data sequence after the abnormal data are removed are uniformly processed.
The electronic device according to claim 16, wherein the processor is specifically configured to: when the abnormal data is removed:

Setting a time window based on a time domain characteristic of the physical quantity;

Using the time window, dividing the data sequence into at least one data segment;

The abnormal data in the at least one data segment is separately removed.
The electronic device according to claim 16 or 17, wherein the processor is specifically configured to: when uniformly processing the continuous data segment:

Identifying the continuous data segment from the data sequence after the abnormal data is removed according to the set time interval threshold;

The data segmentation method is used to uniformly process the continuous data segments in time.
The electronic device according to claim 18, wherein the processor is configured to: when identifying the continuous data segment:

Determining, according to the time interval threshold, a data interruption point in the data sequence after the abnormal data is removed;

A data segment that is disconnected by the data interruption point in the data sequence after the abnormal data is removed is identified as the continuous data segment.
The electronic device according to claim 18, wherein the processor is specifically configured to: when uniformly processing the continuous data segment:

Determining at least one homogenization time point corresponding to the continuous data segment;

Using data in the continuous data segment, data corresponding to the at least one homogenization time point is interpolated.