CN115905860A - Method and system for deep learning test sample selection based on feature distribution analysis - Google Patents

Abstract

The invention discloses a deep learning test sample selection method and a system based on feature distribution analysis, wherein the method comprises the following steps: dividing the feature distribution of the training samples by adopting a clustering method to obtain a feature distribution cluster of the training samples; calculating a characteristic difference value of each training sample/test sample distributed on each characteristic distribution cluster according to the characteristic distribution cluster of the training sample; calculating the characteristic dispersion degree of the training sample according to the output vector of the training sample; constructing a sequencing model based on a sequencing learning algorithm, and realizing prediction and sequencing of test samples; and generating a synthetic test set, sequencing the synthetic test set by using a sequencing model, setting a sampling ratio, and selecting test samples with the highest sequencing to form a test subset. The method screens out test input capable of quickly and sufficiently detecting the DNN model fault under limited resources, so that the influence of data distribution change on the DNN model precision is relieved, the test marking cost is reduced, and the DNN model test efficiency is improved.

Description

Deep learning test sample selection method and system based on feature distribution analysis

Technical Field

The present invention belongs to the field of deep learning test technology, and specifically relates to a deep learning test sample selection method and system based on feature distribution analysis.

Background Art

With the rapid development of deep learning (DL) technology, deep neural network (DNN) models trained based on a large amount of data have been widely used in many fields such as autonomous driving, face recognition, speech recognition, medical diagnosis, aircraft collision avoidance systems, software engineering, etc. Although DNN has achieved great results, its quality problems have also led to many safety accidents. In practice, DNN is mostly tested on a test set, which comes from the same data set as the training set and has the same data distribution as the training data. However, when DNN is deployed in real scenarios, new test data is continuously generated over time, and the distribution of data becomes more and more diverse. As the data distribution shifts, the effectiveness of the model in the test environment will continue to decline.

Therefore, in order to ensure that the model can adapt to data with different distributions in the new environment, in DNN testing, testers retrain the original model by collecting new unlabeled test data, thereby effectively detecting DNN model failures while updating the model's weight parameters, further improving the model quality to adapt to the new data distribution.

However, a large amount of test data needs to be correctly labeled before it can be put into testing, and manual labeling is currently the main method. In order to ensure the correctness of the labeling, multiple users are usually required to collaborate to complete the labeling task. Some test data involving professional fields (such as images, pictures and texts from the medical field) require field experts to label, which results in a large amount of labeling costs and seriously affects the efficiency of DNN model testing. Therefore, maintaining a small-scale test set with good testing capabilities has become an important research issue in the current field of deep learning testing. The goal of deep learning test selection methods is to select representative test input data from the original unlabeled test set to form a test subset, and then label the test subset to reduce the overall labeling and test execution costs.

In order to improve the efficiency of DNN model testing, many test selection methods have been proposed. Shen et al. proposed a test selection method MCP, which clusters test samples into multiple boundary regions based on the maximum confidence and second largest confidence of the test input prediction of the DNN model to be tested, and assigns priorities to uniformly select test samples from all boundary regions to form a test subset, so as to guide the retraining of the DNN model and improve the quality of the DNN model.

However, the problems with the prior art are:

1) Existing methods do not consider the problem of data distribution changes and cannot effectively select test sample data with different distributions from training samples to retrain the model, thereby improving the model's ability to adapt to new data distributions.

2) The test subsets screened by existing methods fail to ensure the diversity of their test samples and lack test adequacy.

Summary of the invention

In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a deep learning test sample selection method and system based on feature distribution analysis, so as to solve the problems of insufficient testing and unknown data distribution changes in the test subsets selected in the prior art; the method of the present invention screens out test inputs that can quickly and fully detect DNN model faults under limited resources, so as to alleviate the impact of data distribution changes on the accuracy of the DNN model, reduce test marking costs, and improve DNN testing efficiency.

To achieve the above object, the technical solution adopted by the present invention is as follows:

The present invention provides a deep learning test sample selection method based on feature distribution analysis, comprising the following steps:

1) Using clustering method to divide the characteristic distribution of training samples and obtain characteristic distribution clusters of training samples;

2) According to the feature distribution clusters of the training samples obtained in step 1), the feature difference value of each training sample/test sample distributed in each feature distribution cluster is calculated;

3) Calculate the feature dispersion of the training sample based on the output vector of the training sample;

4) Build a sorting model based on the sorting learning algorithm and realize the prediction and sorting of test samples;

5) Use the adversarial attack method to generate a synthetic test set, use the sorting model to sort the synthetic test set, set the sampling ratio, and select the top-ranked test samples to form a test subset.

Furthermore, the step 1) specifically includes: inputting all training samples into a DNN model for prediction to obtain respective output vectors, i.e., feature vectors of all training samples; clustering the training samples according to the feature vectors using a K-means clustering algorithm to divide the feature distributions of different categories of the training samples on the corresponding DNN model to obtain multiple feature distribution clusters, each feature distribution cluster represents a type of feature distribution, and the training samples under the same feature distribution cluster have the same type of feature distribution.

Furthermore, the step 2) specifically includes:

Calculation of feature difference value (FDF): Given a sample set S of size m (the sample set includes a training subset R and a test set T), a model D under test, and w feature distribution clusters of the training samples obtained according to the K-means clustering algorithm on the model D under test, the set of feature distribution clusters is C = {c ₁ , c ₂ , ..., c _w }; let s _h be the hth sample in the sample set S, c _j be the jth cluster, select k prototypes from c _j according to the MMD-critic algorithm to form a set X _j = {x _{j, 1} , x _{j, 2} , ..., x _{j, k} }, x _{j, p} is the pth prototype in the set X _j ; according to the feature vector (i.e., output vector) of sample s _h , calculate the distance between sample s _h and all prototypes in the set X _j , and take the average of all distances as the feature difference value FDF _{h, j} of sample s _h distributed in cluster c _j ; the calculation is as follows:

Then the set of feature difference values of the sample _sh distributed on the cluster set C = {c ₁ , c ₂ , ... , c _w } is FDF _h = {FDF _{h, 1} , FDF _{h, 2} , ... , FDF _{h, w} }.

Furthermore, the step 3) specifically includes:

Feature dispersion (FDS) calculation: Given a training set R and a model under test D, the training set R has n categories. Assume that the predicted probability vector of a training sample r (r∈R) on the model under test D is P(r)＝<pr _,1 , _pr,2 ,..., _pr,n >, where _pr,n represents the probability value of the training sample r being predicted as the nth category. Assume that the true category of the training sample r is the tth category. According to the predicted probability _pr,t on the tth category, construct a reference vector P′ _(r) ＝< _pr,t , _pr,t ,..., _pr,t > with a length of n. pr _,t represents the probability value of the training sample r being predicted as the tth category. The feature dispersion (FDS) of the training sample r is calculated as follows:

The higher the feature dispersion FDS value of the training sample, the more dispersed its feature distribution is.

Furthermore, the step 4) specifically includes:

41) Construction of new training set: The new training set is denoted as FP = {fp _{1, 2} , fp ₂ , 3, ..., fp _{n-1, n} }, fp _{n-1, n} represents a feature pair consisting of the feature difference values of the original training samples r _n-1 and r _n ; given a feature pair fp _{p, i} consisting of the feature difference values of the original training samples r _i and r _j , denoted as fp _{p, i} = <FDF _i , FDF _j >; the label label _{i, j} of the feature pair fp _{p, i} takes values of -1, 0 or 1; the label label _{i, j} is determined by the relative size of the feature dispersion of the original training samples r _i and r _j ; the value of the label reflects the relative size of the training model capability between the original training samples in the feature pair; when FDF _i >FDF _j , label _{i, j} = 1, indicating that the feature distribution of the original training sample r _i is more dispersed than that of the original training sample r _j and the training model capability is stronger; when FDF _i = FDF _j , label _{i, j} =0, indicating that the dispersion of the feature distribution of the original training sample r _i is the same as that of the original training sample r _j , and the training model has the same ability; when FDF _i ＜FDF _j , label _{i, i} =-1, indicating that the feature distribution of the original training sample r _j is more dispersed than that of the original training sample r _i , and the training model has stronger ability; the obtained new training set FP is input into the xgboost sorting algorithm, which uses the tree ensemble model and sets the number of generated trees to m. New trees are added by continuously splitting the basic features of the training samples to learn more complex features from the training set, while continuously fitting the prediction residuals to minimize the training error; after generating the set number of m trees, the sorting model is constructed.

42) Construction of new test set: construct new test samples by extracting the feature difference value set of the test samples; assuming that the feature difference value set of a test sample _ti is FDF _i , then the new test sample constructed based on the test sample _ti is a feature pair containing only one feature difference value set, that is, <FDF _i >;

43) Prediction of test samples: Input new test samples into the constructed sorting model, add up the scores of each feature of the new test samples on the corresponding leaf nodes of each tree in the sorting model, and get the predicted score of each test sample. The predicted score reflects the correlation between the test sample and revealing model errors. The larger the predicted score, the easier it is for the test sample to reveal model errors. According to the size of the predicted score, the test samples with large predicted scores are prioritized to achieve sorting of the test samples.

Furthermore, the step 5) specifically includes: using five adversarial attack methods, namely FGSM, BIM-A, BIM-B, CW and JSMA, to perturb each test sample in the test set to generate an adversarial sample set, randomly selecting 20% of the adversarial samples from the adversarial sample set, and selecting 80% of the test samples from the test set to form a synthetic test set with different data distributions; using a sorting model to sort each synthetic test set, setting a sampling ratio, and according to the ratio, selecting the test samples ranked first from the sorted synthetic test set to form a test subset to complete the selection of test samples.

The present invention also provides a deep learning test sample selection system based on feature distribution analysis, comprising:

A feature distribution partitioning module is used to partition the feature distribution of training samples using a clustering method to obtain feature distribution clusters of training samples;

A feature difference value calculation module is used to calculate the feature difference value of each training sample/test sample distributed on each feature distribution cluster according to the feature distribution cluster of the training sample;

A feature dispersion calculation module is used to calculate the feature dispersion of the training sample according to the output vector of the training sample;

The sorting model building module builds a sorting model based on the sorting learning algorithm and realizes the prediction and sorting of test samples;

A generation module for generating a synthetic test set using an adversarial attack method;

The test sample selection module is used to sort the synthetic test set using the sorting model, set the sampling ratio, and select the top-ranked test samples to form a test subset.

The present invention also provides a selection terminal, comprising:

one or more processors;

A memory for storing one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the deep learning test sample selection method based on feature distribution analysis.

The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep learning test sample selection method based on feature distribution analysis.

Beneficial effects of the present invention:

The present invention has effective error detection capability: Based on the analysis of sample feature distribution, the present invention uses learning to rank to build a ranking model by learning the relationship between the feature distribution information of training samples and the ability of training models to predict the correlation between the feature distribution information of test samples and the ability to reveal model errors, thereby obtaining a set of correlation scores. The size of the score reflects the size of the test sample's ability to reveal errors. By sorting and selecting test samples according to the scores, effective test cases can be quickly found for marking, faults in the DNN model can be detected, the marking cost can be greatly reduced, and the DNN test efficiency can be improved.

The present invention has the ability to mitigate data distribution shift: the present invention measures the difference in the feature distribution of the test sample by extracting the difference between the feature distribution of the test sample and the feature distribution of the training samples under each category. According to the feature difference value, the test sample with a larger distribution difference is selected to retrain the model, so that the retrained model can effectively adapt to the new data distribution and mitigate the impact of the change in data distribution on the model accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of the method of the present invention.

FIG. 2 is a schematic diagram of clustering training samples in the present invention.

FIG. 3 is a schematic diagram of extracting feature difference values in the present invention.

FIG. 4 is a schematic diagram of calculating feature dispersion in the present invention.

DETAILED DESCRIPTION

In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

1, a method for selecting a deep learning test sample based on feature distribution analysis of the present invention comprises the following steps:

1) Using a clustering method to divide the feature distribution of training samples to obtain feature distribution clusters of training samples; as shown in Figure 2, specifically including: inputting all training samples (regardless of their labels) into a DNN model for prediction to obtain respective output vectors, i.e., feature vectors of all training samples; according to the feature vectors, using a K-means clustering algorithm to cluster the training samples to divide the feature distributions of different categories of training samples on the corresponding DNN model to obtain multiple feature distribution clusters, each of which represents a type of feature distribution, and the training samples under the same feature distribution cluster have the same type of feature distribution.

2) According to the feature distribution clusters of the training samples obtained in step 1), the feature difference value (FDF) of each training sample/test sample distributed on each feature distribution cluster is calculated; as shown in FIG. 3, specifically including:

Calculation of feature difference value (FDF): Given a sample set S of size m (the sample set includes a training subset R and a test set T), a model D under test, and w feature distribution clusters of the training samples obtained according to the K-means clustering algorithm on the model D under test, the set of feature distribution clusters is C = {c ₁ , c ₂ , ..., c _w }; let s _h be the hth sample in the sample set S, c _j be the jth cluster, select k prototypes from c _j according to the MMD-critic algorithm to form a set X _j = {x _{j, 1} , x _{j, 2} , ..., x _{j, k} }, x _{j, p} is the pth prototype in the set X _j ; according to the feature vector (i.e., output vector) of sample s _h , calculate the distance between sample s _h and all prototypes in the set X _j , and take the average of all distances as the feature difference value FDF _{h, j} of sample s _h distributed in cluster c _j ; the calculation is as follows:

Then the set of feature difference values of the sample _sh distributed on the cluster set C = {c ₁ , c ₂ , ... , c _w } is FDF _h = {FDF _{h, 1} , FDF _{h, 2} , ... , FDF _{h, w} }.

3) Calculating the feature dispersion (FDS) of the training sample according to the output vector of the training sample; as shown in FIG4 , specifically including:

Feature dispersion (FDS) calculation: Given a training set R and a model under test D, the training set R has n categories. Assume that the predicted probability vector of a training sample r (r∈R) on the model under test D is P(r)＝<pr _,1 , _pr,2 ,..., _pr,n >, where _pr,n represents the probability value of the training sample r being predicted as the nth category. Assume that the true category of the training sample r is the tth category. According to the predicted probability _pr,t on the tth category, construct a reference vector P′ _(r) ＝< _pr,t , _pr,t ,..., _pr,t > with a length of n. pr _,t represents the probability value of the training sample r being predicted as the tth category. The feature dispersion (FDS) of the training sample r is calculated as follows:

The higher the feature dispersion FDS value of the training sample, the more dispersed its feature distribution is.

4) Build a ranking model based on the learning to rank algorithm and implement the prediction and ranking of test samples; specifically include:

41) Construction of new training set: The new training set is denoted as FP = {fp _{1, 2} , fp ₂ , 3, ..., fp _{n-1, n} }, fp _{n-1, n} represents a feature pair consisting of the feature difference values of the original training samples r _n-1 and r _n ; given a feature pair fp _{p, i} consisting of the feature difference values of the original training samples r _i and r _j , denoted as fp _{p, i} = <FDF _i , FDF _j >; the label label _{i, j} of the feature pair fp _{p, i} takes values of -1, 0 or 1; the label label _{i, j} is determined by the relative size of the feature dispersion of the original training samples r _i and r _j ; the value of the label reflects the relative size of the training model capability between the original training samples in the feature pair; when FDF _i >FDF _j , label _{i, j} = 1, indicating that the feature distribution of the original training sample r _i is more dispersed than that of the original training sample r _j and the training model capability is stronger; when FDF _i = FDF _j , label _{i, j} =0, indicating that the dispersion of the feature distribution of the original training sample r _i is the same as that of the original training sample r _j , and the training model has the same ability; when FDF _i ＜FDF _j , label _{i, j} =-1, indicating that the feature distribution of the original training sample r _j is more dispersed than that of the original training sample r _i , and the training model has stronger ability; the obtained new training set FP is input into the xgboost sorting algorithm, which uses the tree ensemble model and sets the number of generated trees to m. New trees are added by continuously splitting the basic features of the training samples to learn more complex features from the training set, while continuously fitting the prediction residuals to minimize the training error; after generating the set number of m trees, the sorting model is constructed.

42) Construction of new test set: construct new test samples by extracting the feature difference value set of the test samples; assuming that the feature difference value set of a test sample _ti is FDF _i , then the new test sample constructed based on the test sample _ti is a feature pair containing only one feature difference value set, that is, <FDF _i >;

43) Prediction of test samples: Input new test samples into the constructed sorting model, add up the scores of each feature of the new test samples on the corresponding leaf nodes of each tree in the sorting model, and get the predicted score of each test sample. The predicted score reflects the correlation between the test sample and revealing model errors. The larger the predicted score, the easier it is for the test sample to reveal model errors. According to the size of the predicted score, the test samples with large predicted scores are prioritized to achieve sorting of the test samples.

5) Generate a synthetic test set using an adversarial attack method, sort the synthetic test set using a sorting model, set the sampling ratio, and select the top-ranked test samples to form a test subset; specifically, the following steps are used: use five adversarial attack methods, namely FGSM, BIM-A, BIM-B, CW, and JSMA, to perturb each test sample in the test set to generate an adversarial sample set, randomly select 20% of the adversarial samples from the adversarial sample set, and select 80% of the test samples from the test set to form a synthetic test set with different data distributions; sort each synthetic test set using a sorting model, set the sampling ratio, and select the top-ranked test samples from the sorted synthetic test set to form a test subset according to the ratio to complete the selection of test samples.

In the example:

Select the more mature image dataset CIFAR10 and the DNN network model VGG-16. Use the training set of CIFAR10 to train ResNet-20 and obtain a trained model (referred to as L). Use five adversarial attack methods FGSM, BIM-A, BIM-B, CW, and JSMA to slightly perturb the test set of CIFAR10, and synthesize 20% of the perturbed images and 80% of the original images into a new test set. The method of the present invention is used to sample 1% of the synthetic test set (10,000 test images), that is, 1,000 valid test samples are selected from the test set to form a small-scale test subset, so as to fully and effectively detect the faults of model L, improve the model accuracy, and effectively alleviate the impact of data distribution transfer on model quality. The specific implementation process is as follows:

1. Set the number of clusters k = 10, use the K-means clustering algorithm to cluster the CIFAR10 training samples, and obtain 10 different feature distribution clusters C = {c ₁ , c ₂ , ..., c ₁₀ }.

2. Select 1% of the training samples from the CIFAR10 training set to form the training subset R. Input the selected training samples into the model under test to obtain its output vector, i.e., the feature vector of the training sample. Construct a benchmark vector based on the predicted value of the training sample under the true category. Use the feature vector of the training sample and the benchmark vector to calculate the feature dispersion (FDS) of the training sample.

3. Using the MMD-critic algorithm, select 5 prototypes from each feature distribution cluster to approximately represent the entire cluster. Given the training subset R and the synthetic test set T, obtain the sample set S (S = R ∪ T); given the h-th sample s _h (s _h ∈ S) and a cluster c _j , calculate the Euclidean distance between s _h and the 5 prototypes of c _j according to the output vector of s _h on the tested model L. The average of all Euclidean distances is taken as the feature difference value FDF _{h, j} of the sample s _h distributed on cluster c _j . Similarly, the set of feature difference values of s _h distributed on 10 feature clusters is FDF _h = {FDF _{h, 1} , FDF _{h, 2} , ..., FDF _{h, 10} }.

4. Based on the feature difference value set of any two training samples in the training subset R, construct feature pairs, and use the set of feature pairs as a new training set for training the ranking model based on learning to rank. Then use the feature difference value set of each test sample in the synthetic test set T as a new test sample for the ranking model. Input the new test sample into the ranking model, and the ranking model automatically scores the test sample according to the feature difference value of the sample. The size of the score value reflects the correlation between the test sample and the error of revealing model L; the test samples are sorted according to the size of the correlation based on the score value; the higher the ranking of the test sample, the more likely it is to reveal the error of model L.

5. Select 1% of the top test samples from the sorted test set and mark them, input them into the original tested model L, perform the test after retraining, and report the improvement effect of the model accuracy after retraining, so as to prove the effectiveness of the present invention.

The present invention also provides a deep learning test sample selection system based on feature distribution analysis, comprising:

A feature distribution partitioning module is used to partition the feature distribution of training samples using a clustering method to obtain feature distribution clusters of training samples;

A feature difference value calculation module is used to calculate the feature difference value of each training sample/test sample distributed on each feature distribution cluster according to the feature distribution cluster of the training sample;

A feature dispersion calculation module is used to calculate the feature dispersion of the training sample according to the output vector of the training sample;

The sorting model building module builds a sorting model based on the sorting learning algorithm and realizes the prediction and sorting of test samples;

A generation module for generating a synthetic test set using an adversarial attack method;

The test sample selection module is used to sort the synthetic test set using the sorting model, set the sampling ratio, and select the top-ranked test samples to form a test subset.

The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims (9)

1. A deep learning test sample selection method based on feature distribution analysis is characterized by comprising the following steps:

1) Dividing the feature distribution of the training samples by adopting a clustering method to obtain a feature distribution cluster of the training samples;

2) Calculating a characteristic difference value of each training sample/test sample distributed on each characteristic distribution cluster according to the characteristic distribution cluster of the training samples obtained in the step 1);

3) Calculating the characteristic dispersion degree of the training sample according to the output vector of the training sample;

4) Constructing a sequencing model based on a sequencing learning algorithm, and realizing prediction and sequencing of test samples;

5) And generating a synthetic test set by using an anti-attack method, sequencing the synthetic test set by using a sequencing model, setting a sampling ratio, and selecting test samples ranked in front to form a test subset.

2. The method for selecting the deep learning test sample based on the feature distribution analysis according to claim 1, wherein the step 1) specifically comprises: inputting all training samples into a DNN model for prediction to obtain respective output vectors, namely the characteristic vectors of all training samples; and clustering the training samples by adopting a K-means clustering algorithm according to the feature vectors to divide the feature distribution of different types of the training samples on the corresponding DNN model to obtain a plurality of feature distribution clusters, wherein each feature distribution cluster represents one type of feature distribution, and the training samples under the same feature distribution cluster have the same type of feature distribution.

3. The method for selecting the deep learning test sample based on the feature distribution analysis according to claim 1, wherein the step 2) specifically comprises:

calculating a characteristic difference value: giving w sample sets S with the size of m, measured models D and feature distribution clusters of training samples on the measured models D obtained according to a K-means clustering algorithm, wherein the set of the feature distribution clusters is C = { C = { (C) } ₁ ，c ₂ ，...，c _w }; let s _h Is the h sample in sample set S, c _j Is the jth cluster, from c according to the MMD-critic algorithm _j In which k prototypes are selected to constitute a set X _j ＝{X _j，1 ，x _j，2 ，...，x _j，k }，x _j，p Is a set X _j The p-th prototype in (1); according to the sample s _h Computing a sample s _h And set X _j The average value of all the distances is used as the sample s _h Distributed in cluster c _j Characteristic difference value FDF of _h，j (ii) a The calculation is as follows:

then the sample s _h Distributed in cluster set C = { C = } ₁ ，c ₂ ，...，c _w The set of feature difference values on (i) } is FDF _h ＝{FDF _h，1 ，FDF _h，2 ，...，FDF _h，w }。

4. The method for selecting the deep learning test sample based on the feature distribution analysis according to claim 1, wherein the step 3) specifically comprises:

calculating the characteristic dispersion degree: given a training set R and a tested model D, the class of the training set R is n, and the prediction probability vector of a training sample R (R belongs to R) on the tested model D is assumed to be P (R) = R<p _r，1 ，p _r，2 ，...，p _r，n >，p _r，n Representing the magnitude of the probability value that the training sample r is predicted to be in the nth category; assuming that the true category of a training sample r is the t-th category, according to the prediction probability p on the t-th category _r，t Constructing a reference vector P 'of length n' _(r) ＝<P _r，t ，p _r，t ，...，p _r，t >，p _r，t Representing the magnitude of the probability value that the training sample r is predicted to be in the t-th category; the Feature Dispersion (FDS) of the training samples r is calculated as follows:

the higher the FDS value of the training sample, the more dispersed the feature distribution.

5. The method for selecting the deep learning test sample based on the feature distribution analysis according to claim 1, wherein the step 4) specifically comprises:

41 Construction of a new training set: the new training set is denoted as FP = { FP = _1，2 ，fp _2，3 ，…，fp _n-1，n }，fp _n-1，n Representing the data from the original training sample r _n-1 And r _n A feature pair is formed by the feature difference values; given a training sample r consisting of the original _i And r _j A feature pair fp consisting of feature difference values of _p，i Denoted fp _p，i ＝<FDF _i ，FDF _j >(ii) a The characteristic pair fp _p，i Label of (2) _i，j The value is-1, 0 or 1; label label _i，j From the original training sample r _i And r _j Relative size determination of the characteristic dispersion of (a); the value of the label reflects the relative size of the training model capability between the original training samples in the feature pair; when FDF _i ＞FDF _j When, label _i，j =1, indicates original training sample r _i Is compared with the original training sample r _j The characteristic distribution of the model is dispersed, and the capability of training the model is strong; when FDF _i ＝FDF _j When, label _i，j =0, indicating the feature distribution of the original training sample ri and the original training sample r _j The feature distribution has the same dispersity and the model training capability is the same; when FDF _i ＜FDF _j When, label _i，j = -1, shows original training sample r _j Is compared with the original training sample r _i The characteristic distribution of the model is dispersed, and the capability of training the model is strong; inputting the obtained new training set FP into an xgboost sorting algorithm to complete the construction of a sorting model;

42 Construction of a new test set: constructing a new test sample by extracting a characteristic difference value set of the test sample; assume a test sample t _i Is FDF _i According to the test sample t _i Constructing a new test sample as the features only containing one set of feature difference valuesTo, i.e.<FDF _i >；

43 Prediction of test samples): inputting new test samples into the constructed sequencing model, and adding the score values of leaf nodes corresponding to each feature of the new test samples distributed on each tree in the sequencing model to obtain the predicted score value of each test sample; according to the size of the prediction score value, the test samples with large prediction score values are preferentially arranged in front, so that the test samples are sorted.

6. The method for selecting the deep learning test sample based on the feature distribution analysis according to claim 1, wherein the step 5) specifically comprises: using five counterattack methods of FGSM, BIM-A, BIM-B, CW and JSMA to disturb each test sample in the test set to generate a counterattack sample set, randomly selecting 20% of the counterattack samples from the counterattack sample set, and selecting 80% of the test samples from the test set to form a synthetic test set with different data distribution; and sequencing each synthetic test set by using a sequencing model, setting a sampling ratio, and selecting the test samples sequenced at the front from the sequenced synthetic test sets according to the proportion to form a test subset so as to finish the selection of the test samples.

7. A deep learning test sample selection system based on feature distribution analysis, comprising:

the characteristic distribution dividing module is used for dividing the characteristic distribution of the training samples by adopting a clustering method to obtain a characteristic distribution cluster of the training samples;

the characteristic difference value calculation module is used for calculating the characteristic difference value of each training sample/test sample distributed on each characteristic distribution cluster according to the characteristic distribution cluster of the training sample;

the characteristic dispersion degree calculation module is used for calculating the characteristic dispersion degree of the training sample according to the output vector of the training sample;

the sequencing model building module builds a sequencing model based on a sequencing learning algorithm and realizes prediction and sequencing of the test samples;

a generation module for generating a synthetic test set using an anti-attack method;

and the test sample selection module is used for sorting the synthetic test set by using the sorting model, setting a sampling ratio and selecting the test sample with the front sorting to form a test subset.

8. A selection terminal, comprising:

one or more processors;

a memory for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-6.

9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the selection method according to any one of claims 1 to 6.

Images