CN118656273A - Hard disk failure prediction and data migration method for low-quality data sets - Google Patents

Abstract

The present invention provides a hard disk failure prediction and data migration method for low-quality data sets, including: obtaining the SMART information of the hard disk to obtain an information set, reconstructing the information set with positive and negative samples, and using the information set with lost data as the original data set; performing useless data cleaning operations on the original data set and performing undersampling processing; filling missing values on the original data set, and converting the original data set after missing value filling into time series data; constructing and training a prediction model, inputting the time series data and the corresponding ASFD features into the prediction model to obtain the prediction result of the hard disk failure; identifying the hard disk that is about to fail according to the prediction result, and completing the repair of the hard disk data that is about to fail based on the bipartite graph maximum matching strategy and the repair scheduling strategy. Beneficial effects of the present invention: In the case of low-quality hard disk SMART, a higher hard disk failure prediction accuracy is achieved, and the data of the hard disk that is about to fail can be actively migrated and repaired in advance.

Description

Hard disk failure prediction and data migration method for low-quality data sets

Technical Field

The present invention belongs to the field of data storage, and in particular relates to a hard disk failure prediction and data migration method for low-quality data sets.

Background Art

Today, hard disks are the most important storage devices in computers, and many data centers rely on a large number of hard disks to store important information. In some scenarios where large-scale storage systems are used, such as high-performance computing and Internet services, hard disk failures occur frequently. A survey of relevant literature shows that 78% of hardware replacements are caused by hard disk failures. The catastrophic consequences of hard disk failures are permanent and difficult to recover, thereby reducing the reliability of the data center. Therefore, predicting disk failures as early as possible can not only reduce the risk of data loss, but also reduce the cost of data recovery.

SMART (Self-Monitoring Analysis and Reporting Technology) was proposed in the 1990s. This technology can detect various working information inside the hard disk, such as the number of hard disk reads and writes, the number of head load and unload times, and the seek error rate. The traditional threshold-based hard disk failure technology compares the current SMART attribute of the hard disk with the set threshold. If the SMART attribute of the hard disk exceeds the corresponding threshold, the hard disk will send an alarm message to the operating system. However, this method has only a 3% to 10% TPR (True Positive Rate) when the FPR (False Positive Rate) is 0.1%. In the past few decades, many scholars have used machine learning and deep learning technologies to propose a variety of methods to improve the accuracy of fault prediction, including the use of Long short-term memory (LSTM), Generative Adversarial Network (GAN), regularized greedy forest (RGF), Temporal Convolutional Networks (TCNs), recurrent neural networks (RNNs), Convolutional neural networks (CNNs) and other algorithms to establish hard disk failure prediction models. Unfortunately, these related works generally train models based on high-quality datasets, such as the Backblaze open source dataset. Such prediction models have poor applicability in low-quality dataset scenarios.

In actual industrial applications, some data may be lost due to objective or subjective reasons during the collection, transmission, and storage of hard disk SMART data. For example, various errors may occur during data collection, such as sensor errors, data transmission errors, equipment failures, etc. These errors will cause inaccurate or incomplete data in the data set. For another example, a large-scale data center must turn off the SMART collection function during holidays. In addition, in order to prevent themselves from being replaced by intelligent operation and maintenance software, operation and maintenance engineers destroy sample data to reduce the accuracy of hard disk failure prediction technology. How to achieve good prediction results under low-quality data sets has become an important problem that needs to be solved in the actual engineering application of hard disk failure prediction technology.

When faced with low-quality data sets, existing technologies often use the method of filling the mode. Although it is simple to implement, the accuracy of hard disk failure prediction is insufficient. At the same time, existing technologies do not consider the timing of hard disk data. Hard disk failure does not occur suddenly at a certain moment, but is a gradual process.

Summary of the invention

In view of this, the present invention aims to propose a hard disk failure prediction and data migration method for low-quality data sets, in order to solve at least one of the above-mentioned technical problems.

To achieve the above object, the technical solution of the present invention is achieved as follows:

The hard disk failure prediction and data migration method for low-quality data sets includes the following steps:

Obtaining SMART information of the hard disk to obtain an information set, marking the information set with positive and negative samples, and using the information set with missing data as the original data set;

Performing a useless data cleaning operation and under-sampling processing on the original data set;

Filling missing values in the original data set, converting the original data set after missing value filling into time series data, and introducing ASFD features;

Build and train a prediction model, input time series data and corresponding ASFD features into the prediction model to obtain the prediction results of hard disk failure;

According to the prediction results, the hard disk that is about to fail is identified, and the migration and repair of the failed hard disk is completed based on the bipartite graph maximum matching strategy and repair scheduling strategy.

Furthermore, the process of labeling the information set as positive and negative samples includes:

Obtain hard disk failure information, mark the information set in the time period before the hard disk failure as a positive sample, and mark the information set in the rest of the time as a negative sample.

Furthermore, the process of performing useless data cleaning operation and undersampling processing on the original data set includes:

Obtain the fixed data in the hard disk and treat it as useless data and remove it from the original data set; select the corresponding number of negative sample information sets according to the number of information sets marked as positive samples in the original data set, and remove the remaining negative sample information sets.

Furthermore, the process of filling missing values in the original data set includes:

Get the hard disk model of the current hard disk, and collect the SMART information of all hard disks with the same hard disk model as a filling set;

Get the SMART items corresponding to the lost data in the current hard disk as the missing set;

The mode of each SMART item in the intersection of the filled set and the missing set is extracted, and the mode is used as the missing data for filling.

Furthermore, the process of introducing the ASFD feature includes:

Traversing each element in the time series, and calculating the difference between the current element and the previous element, wherein the element is the hard disk SMART information converted into a time series;

The absolute value of the difference is taken, and the absolute value corresponding to each element is added to obtain an ASFD value, and the ASFD value is added to the original data set, wherein each ASFD value corresponds to a SMART item.

Furthermore, the process of building and training the prediction model includes:

The prediction model is trained using the LGB algorithm, and the optimal hyperparameters of the prediction model are obtained using the OPTUNA framework.

Furthermore, the process of completing the migration and repair of the faulty hard disk based on the bipartite graph maximum matching strategy and the repair scheduling strategy includes:

According to the bipartite graph maximum matching strategy, each data block in the hard disk is allocated to multiple reconstruction sets, and the reconstruction sets have the ability to be reconstructed in parallel, and according to the repair scheduling strategy, the number of data blocks in each migration reconstruction is maximized.

Compared with the prior art, the hard disk failure prediction and data migration method for low-quality data sets described in the present invention has the following beneficial effects:

In the case of low-quality hard disk SMART, a high hard disk failure prediction accuracy rate can be achieved, and data on the hard disk that is about to fail can be actively migrated and repaired in advance.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

Fig. 1 is a schematic diagram of the workflow of the present invention;

FIG2 is a schematic diagram showing the distribution of three stripes under RS (4,2) coding conditions in the present invention;

FIG3 is a schematic diagram of identifying blocks to be transmitted in the present invention;

FIG4 is a schematic diagram of identifying how to store repair blocks in the present invention;

FIG5 is a schematic diagram of performing repair in the present invention;

FIG. 6 is a schematic diagram of performing migration and reconstruction in a repair round in the present invention.

DETAILED DESCRIPTION

It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

The hard disk failure prediction and data migration method for low-quality data sets includes the following steps:

S1. Obtain the SMART information of the hard disk to obtain an information set, mark the information set with positive and negative samples, and use the information set with missing data as the original data set.

The process of labeling the information set with positive and negative samples in step S1 includes:

Obtain hard disk failure information, mark the information set in the time period before the hard disk failure as a positive sample, and mark the information set in the rest of the time as a negative sample.

Preferably, a specific execution process of step S1 and its explanation are as follows:

Collect SMART information from hard disks in large data centers regularly, for example, collect information from each hard disk once a day, and mark the data of the failed disk within 30 days before the failure as 1, which is called a positive sample; the data in other time periods is marked as 0, which is called a negative sample. The negative sample is the data of healthy hard disks;

Part of the data may be lost due to objective or subjective reasons during the collection, transmission, and storage of hard disk SMART data. Common loss situations include: SMART data loss of the same hard disk for several consecutive days, SMART data loss of several hard disks on the same day, and partial SMART field loss of the same hard disk on the same day. Therefore, the data set that is actually collected and has partial data loss is selected as the original data set.

S2. Cleaning useless data and performing under-sampling processing on the original data set.

The process of performing useless data cleaning and undersampling processing on the original data set in step S2 includes:

Obtain the fixed data in the hard disk and treat it as useless data and remove it from the original data set; select the corresponding number of negative sample information sets according to the number of information sets marked as positive samples in the original data set, and remove the remaining negative sample information sets.

The purpose of filtering out useless data in the SMART dataset is to reduce the workload of subsequent analysis. At the same time, due to the large disparity between the number of positive and negative samples in the hard disk dataset, undersampling is used to artificially change the ratio of positive and negative samples.

S3. Fill missing values in the original data set, convert the original data set after missing value filling into time series data, and introduce ASFD features.

By constructing time series, we can effectively make up for the low quality of the SMART data set. In order to better calculate and process the data in the original data set, we create and use time windows to convert the original data set into time series data. The time window can be used to more representatively discover the relationship between data and facilitate subsequent calculations. By constructing a small sliding window and sliding on the constructed time window, we can better discover the changes in data within the time window and enrich the calculation of high-order data. The features of multiple time series are merged, and the features that perform well after interaction are selected to strengthen the prediction model.

S31, the process of filling missing values in the original data set in step S3 includes:

Get the hard disk model of the current hard disk, and collect the SMART information of all hard disks with the same hard disk model as a filling set;

Get the SMART items corresponding to the lost data in the current hard disk as the missing set;

The mode of each SMART item in the intersection of the filled set and the missing set is extracted, and the mode is used as the missing data for filling.

Preferably, a specific execution process of step S31 and its explanation are as follows:

After converting the original data set into time series data, even if a large amount of original data is lost, the changing trend of the original data can still be obtained from the high-order values, making the prediction model more accurate and capable of dealing with the situation of low quality data set;

In order to obtain better prediction results, healthy disks and faulty disks are classified and processed separately. For healthy disks, the missing value of hard disk i in the jth SMART item is filled with the mode of all jth SMART items of the model to which the healthy disk belongs. For faulty disks, the missing value of hard disk i in the jth SMART item is filled with the mode of all jth SMART items of the model to which the faulty disk belongs.

S32, the process of introducing the ASFD feature in step S3 includes:

Traversing each element in the time series, and calculating the difference between the current element and the previous element, wherein the element is the hard disk SMART information converted into a time series;

The absolute value of the difference is taken, and the absolute value corresponding to each element is added to obtain an ASFD value, and the ASFD value is added to the original data set, wherein each ASFD value corresponds to a SMART item.

Preferably, a specific execution process of step S32 and its explanation are as follows:

ASFD (Absolute Sum of First Difference) is a measure of the change or variation between values in a time series. It is calculated by taking the difference between each element in a time series and its previous element. The absolute sum is the sum of the absolute values of these differences. The resulting value of ASFD is a non-negative number. A value of zero means that all values in the time series are the same. A higher value means a greater change in the values.

The calculation formula of the ASFD is: .

Wherein, y is the desired ASFD value; X is the desired SMART attribute value of the desired SMART attribute column; I is the i-th row; N is the number of rows in the desired SMART attribute column; _xi is the SMART attribute value of the i-th row of the desired SMART attribute column; _xi+1 is the SMART attribute value of the i+1-th row of the desired SMART attribute column.

For each SMART item of each hard disk, the ASFD value is calculated one by one and added to the original data set as a new attribute. The number of entries in the processed original data set remains unchanged, but the number of SMART items for each entry is doubled. For example, for the hard disk numbered 4214, the attribute value of the SMART data item ID 1 on June 1, 2024 is 15, and the ASFD value of the SMART data item ID 1 is 0.9. That is, a corresponding ASFD value is added for each SMART data value.

S4. Build and train a prediction model, input the time series data and the corresponding ASFD features into the prediction model to obtain the prediction result of the hard disk failure.

S41, the process of constructing and training the prediction model in step S4 includes:

The prediction model is trained using the LGB algorithm, and the optimal hyperparameters of the prediction model are obtained using the OPTUNA framework.

Preferably, the reasons for selecting the term explanation, i.e., the technical solution, proposed in step S41 are as follows:

LGB (lightgbm) is common knowledge in this field: Gradient Boosting Decision Tree (GBDT) is a commonly used model in machine learning. Its main idea is to use weak classifiers for iterative training to obtain the optimal model. This model has the advantages of high accuracy and low overfitting. GDBT is widely used in the industry and is usually used for tasks such as classification, regression and sorting. LGB is an open source framework for GBDT implemented by the Microsoft DMTK team. As an implementation of GBDT, the LGB algorithm has made many optimizations on the traditional GBDT algorithm and has the advantages of supporting efficient parallel training, lower memory consumption, and higher accuracy.

Compared with the traditional GBDT algorithm, the LGB algorithm has the following advantages:

Histogram optimization is used. Histogram optimization simplifies the expression of data and reduces the use of memory. Histograms can avoid overfitting to a certain extent. The histogram of a leaf node can be obtained by subtracting the histogram of a brother node from the histogram of the father node. Based on this, the histogram of a leaf node with a smaller amount of data can be calculated first, and then the histogram of a leaf node with a larger amount of data can be obtained by subtracting the histogram to achieve the acceleration effect.

The leaf-wise method is used to improve the accuracy of the model. Compared with the level-wise method, the leaf-wise method is more efficient. When the number of leaf nodes is the same, the leaf-wise method can obtain smaller training errors and higher accuracy. If the leaf-wise method is used alone to grow, overfitting may occur on small data sets. Therefore, LGB adds restrictions on the depth of the tree in leaf-wise.

Support for category features. Traditional machine learning algorithms generally cannot directly input categories. Category features need to be discretized and converted into multi-dimensional 0/1 features. This approach is inefficient both in space and time. LGB directly supports category features by changing the decision rules of the decision tree algorithm without the need for additional discretization.

Supports parallelization. Feature Parallelism is suitable for scenarios with small data but more features, Data Parallelism is suitable for scenarios with large data volume but fewer features, and Voting Parallelism is suitable for scenarios with large data volume and more features. Feature Parallelism vertically splits the data so that different machines have all sample points but different features, and then finds the optimal segmentation points of different features on different machines, and then synchronizes them to obtain the global optimal segmentation point. Data Parallelism horizontally splits the data so that different machines have part of the data but all the features, and then constructs histograms on different machines, merges the histograms, and finds the optimal segmentation point on the merged global histogram. Voting Parallelism is an optimization based on data parallelism. The bottleneck of data parallelism is mainly the merging of histograms. Voting Parallelism only merges the histograms of some feature values by voting, thereby reducing the amount of communication. First, the optimal features of TOP K are obtained through local data, and then the features that may be the global optimal segmentation points are obtained by voting. When merging histograms, only these selected features are merged, thereby reducing the amount of communication when merging histograms.

OPTUNA is common knowledge in this field: Regarding the hyperparameter tuning of the model, many scholars use the brute force search method of GridSearch. Although it is simple and easy to understand, its disadvantage is obvious and it is too time-consuming to run. In contrast, the parameter tuning method based on the Bayesian framework, such as HyperOPT and OPTUNA, is lightweight and more powerful, and the speed is also very fast. As the parameter tuning method adopted by the present invention, generally, the hyperparameters of the tree-based model can be divided into 4 categories: 1) parameters that affect the structure and learning of the decision tree; 2) parameters that affect the training speed; 3) parameters that improve the accuracy; 4) parameters that prevent overfitting.

In most cases, there is a lot of overlap between these categories. Improving the efficiency of one category may reduce the efficiency of another category. Relying entirely on manual parameter tuning is not only time-consuming and labor-intensive, but also difficult to find the optimal balance. However, if a suitable parameter grid is provided, OPTUNA can automatically search and find the most balanced parameter combination between these categories, thereby optimizing the overall performance.

The above-mentioned LGB and OPTUNA methods are an optimal combination obtained after long-term experiments and comparisons based on a large number of model algorithms and parameter tuning. Other model algorithms, such as LSTM, random forest, and SVM, cannot achieve the effect of the LGB method. Other parameter tuning methods, such as GridSearch and HyperOPT, also cannot achieve the effect of the OPTUNA method.

S42, the process of inputting the time series data and the corresponding ASFD features into the prediction model in step S4 to obtain the prediction result of the hard disk failure includes:

The collected hard disk SMART data is checked for partial data loss. If there is data loss, i.e., low quality, the missing data is filled with the mode of the SMART item of the hard disk in the last N days, where N is generally 7 or 10 days. Then, the ASFD value of each SMART item of the hard disk is calculated to obtain the latest sample of the hard disk, which is input into the model. The input result is 0 or 1, where 0 represents the prediction result that the hard disk is healthy, and 1 represents that the hard disk is about to fail.

S5. Identify the hard disk that is about to fail based on the prediction results, and complete the migration and repair of the failed hard disk based on the bipartite graph maximum matching strategy and the repair scheduling strategy.

The process of completing the repair of the faulty hard disk data based on the bipartite graph maximum matching strategy and the repair scheduling strategy in step S5 includes:

According to the bipartite graph maximum matching strategy, each data block in the hard disk is allocated to multiple reconstruction sets, and the reconstruction sets have the ability to be reconstructed in parallel, and according to the repair scheduling strategy, the number of data blocks in each migration reconstruction is maximized.

Preferably, the specific execution processes of the bipartite graph maximum matching strategy and the repair scheduling strategy in step S5 are respectively:

The maximum matching strategy for bipartite graphs is:

Construct a set C and an initial reconstruction set R, wherein the set C represents the set of all blocks of the faulty disk that need to be repaired;

For each data block in the set C, determine in turn whether the block set in the newly constructed reconstruction set R can be reconstructed in parallel if it is added to the reconstruction set R;

If parallel reconstruction is possible, the data block is added to the reconstruction set R and deleted from the set C until all blocks in the set C are organized into the reconstruction set R.

In the bipartite graph maximum matching strategy, the specific process of sequentially judging and adding data blocks to the reconstruction set includes:

For each data block _Ci , use the MATCH function to construct a bipartite graph containing N−1 nodes, one part of which consists of blocks from healthy hard disks and the other part consists of check blocks;

Identify the maximum matching in the bipartite graph. If there are k(1+|R|) edges in the maximum matching, it means that the current data block _Ci can be rebuilt in parallel through k(1+|R|) different healthy hard disks and check blocks. At this time, the MATCH function returns true, adds the current data block _Ci to the reconstruction set R, and deletes the current data block _Ci from the set C.

Multiple repair rounds are set, and in each repair round, the above operations performed on each data block _Ci are repeated until all data blocks in the set C are added to the reconstruction set R, and the set of all reconstruction sets R is obtained D = { _R1 , _R2 , ..., _Rd }, where d is the total number of reconstruction sets.

In order to increase the number of blocks in the reconstruction set and thus improve the capability of parallel reconstruction, it may be checked whether the reconstruction set R can be expanded by exchanging a data block in the reconstruction set R with other data blocks not in the reconstruction set R.

Repair scheduling strategy:

Sort all reconstruction sets R in set D in monotonically decreasing order according to the number of data blocks they contain, so as to ensure that reconstruction sets containing more blocks are given priority during the repair process;

Set two indices l and u, where l represents the index of the current smallest reconstruction set index, and u represents the current largest reconstruction set index, that is, the first reconstruction set is R _l , and the last reconstruction set is R _u ;

Establish a block set C _m that is repaired by migration, and establish a block set C _r that is repaired by collaborative reconstruction using healthy hard disks;

Calculate C _m and determine whether the current C _m satisfies |R _(l+1) ∪…∪R _u |≤ C _m ;

If it is satisfied, the reconstruction set between R _(l+1) to _Ru can be repaired by migration, R _{l can} be repaired by reconstruction, and the migration and reconstruction processes are performed in parallel, and the repair round is interrupted to complete the repair;

Otherwise, find the reconstruction set with the least number of data blocks in the current set D, so that , x is the index of the reconstruction set index of the current reconstruction set, and obtains the subset R _x 'of the current reconstruction set belonging to R _x , such that ;

Let _Rx = _Rx - _Rx ', let _Ml = _Rx ' ∪Rx ₊₁ … _∪Ru , update the indicators l and u, let l = l +1, u = x, so that the new reconstruction set is considered in the next repair round, and iterate this process until all reconstruction sets have been processed.

Working process:

The present invention discloses a hard disk failure prediction and data migration method for low-quality data sets. In order to better illustrate the working process of the present invention, a specific implementation process is given as follows:

1) Create a time window: In low-quality data sets, we use a standard time window size of 10 days to create time series features, which facilitates the subsequent addition of time series features.

2) Set sliding window: Preset two different sliding windows of 7 days and 1 day in advance to facilitate the subsequent addition of time feature sequences. The sliding window will slide on the above time windows for calculation;

Add ASFD feature. Use the sliding window method to add the time series feature AbsoluteSum of First Difference (ASFD) to the data set to reflect the absolute fluctuation between adjacent observations of hard disk SMART data. If a time series x = [1,1,0,1,0] is given, then the first-order difference is [0,0,1,1,1], and ASFD is y = [0,0,1,2,3];

3) CID features can also be added as a supplement. The time series feature Complexity Invariant Distance (CID) is added using the sliding window method to reflect the time series complexity of the hard disk SMART data. For time series Q and C, the CID calculation formula is: ,Through multiple experiments, adding CID features will improve the ,prediction accuracy in most cases, but it will increase the processing time, and ,engineering technicians need to decide whether to add CID based on actual ,needs.

4) Overview of the training dataset: After preprocessing the original dataset, each SMART attribute is added with ASFD features and CID features. The training dataset has a total of 121 dimensions of features.

5) Model training: Use LGB (lightgbm) developed by Microsoft and the preprocessed training set to train the hard disk failure prediction model.

6) Model hyperparameter tuning: Use the OPTUNA method to find the optimal hyperparameters of the LGB algorithm and tune the model.

7) Hard disk failure prediction: After obtaining the failure prediction model with the optimal hyperparameters, the test set data is input into the failure prediction model to obtain the disk-level prediction result, that is, whether the hard disk will fail.

8) Data migration and reconstruction:

As shown in Figures 2, 3, 4, and 5: A bipartite graph with two sets of left and right vertices is constructed, and the node vertices of {N1, N2, N3, N4, N5} and {N4, N5, N6, N7, N8} are connected to the k = 5 block vertices of strips S1 and S2 respectively. The maximum matching is found, and it is concluded that blocks can be retrieved from N1, N2, N3, and N4 to reconstruct the blocks of S1. Another bipartite graph is constructed, and the node vertices of {N6, N7, N8}, {N1, N2, N3}, and {N3, N4, N5} are connected to the strip vertices of strips S1, S2, and S3 respectively. The maximum matching shows that the repair block of S1 can be stored in N6.

As shown in Figure 6: Another embodiment of data migration and reconstruction is given, taking a reconstruction set of d = 7, Cm = 4. For the first round of repair, it is found that the largest x is 5, so that |R5|+|R6|+|R7|>Cm = 4, and then further find a subset R5'⊂R5, R5' contains a block in R5, so that M1 = R5'∪R6∪R7. Through migration, the number of remaining data blocks in R5 is reduced to 2 in parallel with the reconstruction of R1 in the first repair round, and finally all blocks are repaired in the third round of repair.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (7)

1. The hard disk fault prediction and data migration method for the low-quality data set is characterized by comprising the following steps of:

obtaining SMART information of a hard disk to obtain an information set, reconstructing positive and negative samples of the information set, and taking the information set of lost data as an original data set;

carrying out useless data cleaning operation on the original data set and undersampling;

Filling the original data set with missing values, converting the original data set filled with the missing values into time sequence data, and introducing ASFD features;

constructing and training a prediction model, and inputting time sequence data and corresponding ASFD features into the prediction model to obtain a prediction result of the hard disk fault;

And identifying the hard disk which is in imminent failure according to the prediction result, and completing migration repair of the failure hard disk based on the maximum matching strategy and the repair scheduling strategy of the bipartite graph.

2. The method for predicting hard disk failure and data migration for low quality data sets according to claim 1, wherein the process of reconstructing positive and negative samples of said information sets comprises:

Acquiring an information set of a fault hard disk in a period of time before a fault, and marking the information set as a positive sample; according to the length of the time period, acquiring an information set of the healthy hard disk in the same time period, and marking the information set as a negative sample;

wherein, the SMART information of the hard disk is recorded with the fault state of the hard disk.

3. The method for predicting hard disk failure and data migration for low-quality data sets according to claim 1, wherein said process of performing garbage collection operation on the original data set and performing undersampling process comprises:

The method comprises the steps of obtaining fixed and unchanged data in a hard disk, and regarding the fixed and unchanged data as useless data to be cleared from an original data set; and selecting a corresponding number of negative sample information sets according to the number of the information sets marked as positive samples in the original data set, and clearing the rest negative sample information sets.

4. The method for predicting hard disk failure and data migration in accordance with claim 1, wherein said process of missing value filling of said original data set comprises:

acquiring the hard disk model of the current hard disk, and acquiring SMART information of all the hard disks with the same hard disk model as a filling set;

Acquiring a SMART item corresponding to lost data in a current hard disk, and taking the SMART item as a deletion set;

And extracting the mode of each SMART item in the intersection of the filling set and the missing set, and filling the mode as missing data.

5. The method for hard disk failure prediction and data migration for low quality data sets according to claim 1, wherein the process of introducing ASFD features comprises:

Traversing each element in the time sequence, and calculating a difference value between a current element and a previous element, wherein the element is hard disk SMART information converted into the time sequence;

And taking absolute values of the differences, adding the absolute values corresponding to each element to obtain ASFD values, and adding the ASFD values to an original data set, wherein each SMART item corresponds to one ASFD value respectively.

6. The method for predicting hard disk failure and data migration in accordance with claim 1, wherein said process of constructing and training a predictive model comprises:

and training the prediction model by using an LGB algorithm, and acquiring the optimal super parameters of the prediction model by using a OPTUNA framework.

7. The method for predicting and repairing hard disk failure based on low-quality data set according to claim 1, wherein the process of completing migration repair of the failed hard disk based on the bipartite graph maximum matching policy and the repair scheduling policy comprises:

And distributing each data block in the hard disk to a plurality of reconstruction sets according to the maximum matching strategy of the bipartite graph, wherein the reconstruction sets have parallel reconstruction capability, and the number of the data blocks in each migration reconstruction is maximized according to the repair scheduling strategy.