CN107256245A - Improved and system of selection towards the off-line model that refuse messages are classified - Google Patents

Abstract

Improve and system of selection, comprise the following steps towards the off-line model that refuse messages are classified the invention discloses a kind of：(1) feature selecting and extension, select feature using feature selection approach, construct feature term vector, original short message text is represented using Feature Words vector model；(2) offline sorting algorithm and improved tuning training and test, the improvement for making to classify towards refuse messages to offline sorting algorithm, data preparation is carried out to the training set obtained by step (1) and test set according to each offline sorting algorithm and improvement, using training set is to each off-line algorithm and improves progress tuning training and test；(3) the offline sorting algorithm selection based on evaluation index, is proposed the evaluation index classified towards refuse messages, the test result obtained by step (2) is analyzed using the evaluation index and optimal offline sorting algorithm is selected.

Description

Offline Model Improvement and Selection Method for Spam SMS Classification

technical field

The invention relates to an offline text classification algorithm, in particular to an offline model improvement and selection method for classification of junk short messages, and belongs to the technical field of text content-based identification of junk short messages.

Background technique

The most important thing in the text classification problem is to select and train the text classification model, and the performance of text classification depends on the text classification model to a large extent. Recently, researchers have proposed various text classification models based on machine learning, combined with multidisciplinary theories such as statistics and informatics.

Naive Bayesian classification algorithm is a machine learning method based on statistics, which is widely used in text classification problems. The algorithm is based on the assumption of feature independence. Although there is often correlation between features in practical problems, this assumption simplifies the calculation of the naive Bayesian classification model. In the content-based spam classification problem, the Naive Bayesian classification algorithm has achieved very good predictive performance.

The decision classification tree algorithm is also a common algorithm for text classification problems. It uses the training data set to learn a decision classification tree. Each node in the tree corresponds to a feature, and each branch of the node corresponds to a division based on the feature of the node. The tree Leaf nodes correspond to category labels. At present, there are many decision tree construction methods, such as ID3 algorithm based on information gain, C4.5 algorithm based on information gain ratio, and CART algorithm based on Gini index. The decision tree classification algorithm is applied to the text classification problem to obtain a set of rules, and the corresponding features of the test text are judged along these rules, and finally the category of the test text can be determined.

Perceptrons were first applied to text classification problems by Schutze et al. Later, the perceptron algorithm applied to the text classification problem has been greatly improved and optimized, such as POSITIVE WINNOW, BALANCED WINNOW, WIDROW-HOFF and so on. The perceptron is actually the simplest neural network. The difference between the two is that the perceptron learns a linear classification model, while the neural network obtains a nonlinear classification model, but the perceptron can obtain a classification similar to the neural network. performance, and the training time complexity is low.

The KNN algorithm selects the k training samples closest to the test sample based on the distance measurement function, and then uses the majority voting method to determine the category of the test sample. No training is required, but the classification error is also large. If the k value is selected too small, it is easily affected by the noise data. If the k value is selected too large, then the training samples with a large distance (dissimilarity) from the test sample will also Act on the forecast, produce wrong forecast results. In text classification problems, more classification models combining KNN and other classification algorithms are used, such as nearest neighbor and clustering algorithms, nearest neighbor and maximum a posteriori estimation, etc.

Support vector machine classification algorithm is widely used in text classification problems, and a large number of experiments show that support vector machine is a classification model with high accuracy.

Recently, integrated classifiers have attracted more and more attention. The basic idea is that "three cobblers are better than one Zhuge Liang". The prediction results of multiple classifiers must be more credible than that of a single classifier. Learning multiple weak classifiers, Finally, the classification results of each weak classifier are integrated as the final prediction result. Classifier integration rules mainly include majority voting rules (Majority Voting), dynamic classifier selection (dynamic selection), linear weighted combination rules (Weighted Linear Combination), adaptive classifier combination rules (Adaptive Combination) and so on. AdaBoost is an algorithm for building integrated classifiers. This algorithm learns multiple weak classifiers by dynamically changing the weight distribution of samples. The integration rule adopted is a linear weighted combination rule, and the weight is calculated according to the classification error rate of the weak classifiers.

Facing the relatively stable static short message data of spam short message feature information, the present invention improves and selects the offline classification algorithm, wherein the offline classification algorithm includes: LR, AdaBoost decision tree, SVM and GBDT, and proposes an evaluation index facing spam short message classification, and based on Evaluation metrics for offline model selection.

Contents of the invention

Purpose of the invention: the present invention proposes the improvement and selection of the offline classification algorithm based on the relatively stable static short message data set of spam short message feature information, aiming at obtaining the optimal spam short message classification model. Offline classification algorithms and improvements include LR based on a specific data format, which reduces the space-time resource consumption of the LR model and improves the efficiency of model training and testing; AdaBoost decision tree for differential loss, differential loss is based on spam messages The misjudgment cost of normal SMS is higher than that of spam SMS misjudgment in recognition. Since AdaBoost can dynamically update the sample weight distribution in the process of iteratively generating weak classifiers, it is possible to differentially adjust the misclassified samples in the iterative process. The weight makes the samples misjudged as spam text messages by normal text messages in the previous round receive more attention than misjudged spam text messages in this round of iteration; SVM and SVM which have certain advantages in dealing with nonlinear separable problems and classification performance Automatically do feature selection and use only part of the features in each iteration, GBDT with good performance, and introduce their parameter tuning methods. Secondly, an evaluation index for spam SMS classification is proposed, parameter tuning training and testing are carried out for the offline classification algorithm, and the offline model is selected based on the evaluation index.

Technical solution: an offline model improvement and selection method for spam classification, including the following four aspects:

(1) SMS text preprocessing, the main preprocessing content includes: word segmentation, unified conversion of SMS text to simplified expressions, conversion of desensitized strings such as numbers to single characters, and removal of stop words;

(2) feature selection and expansion, according to the preprocessing result obtained in step (1), use the feature selection method to select features, construct the feature word vector, and use the feature word vector model to represent the original text message;

(3) Offline classification algorithm and improved tuning training and testing, the offline classification algorithm is improved for spam classification, according to each offline classification algorithm and improvement, the training set and test set obtained in step (2) are prepared for data, Use the training set to perform tuning training and testing for each offline algorithm and improvement;

(4) Based on the selection of the offline classification algorithm of the evaluation index, an evaluation index for spam SMS classification is proposed, and the evaluation index is used to analyze the test results obtained in step (3) and select the optimal offline classification algorithm.

The content (1) SMS text preprocessing, the main preprocessing content includes: word segmentation, SMS text unified to simplified expression, number and other desensitized character strings to single characters, stop words removal, specifically:

(1.1) Use Ansj to segment the SMS text and keep the part-of-speech tag;

(1.2) Unified conversion of SMS text to simplified expressions, numbers and other desensitized strings to single characters;

(1.3) Remove stop words according to the stop word list.

Described content (2) feature selection and expansion, according to the preprocessing result that step (1) obtains, use feature selection method to select feature, construct feature word vector, use feature word vector model to represent original SMS text, specifically:

(2.1) Frequent word feature selection based on statistical threshold and average information gain, the threshold is an adjustable parameter, select frequent words as the feature word set according to the threshold, and decide whether to continue to adjust the threshold according to the average information gain of the feature word set;

(2.2) Feature selection based on the N-Gram algorithm for double-character words and compound words, generate text fragment sequences based on the N-Gram algorithm, filter out infrequent sequences according to the optimal statistical threshold obtained in step (2.1), and construct the remaining sequences Relevance matrix, the matrix element is the frequency of occurrence of the corresponding row and column combination sequence in the spam text, and the combination text sequence is screened according to certain standards;

(2.3) Combining non-modifying content words into tuple features, traversing all spam texts to find noun+verb\adjective combinations, and screening the tuple features obtained according to certain standards.

(2.4) Based on the feature selection of cumulative information gain, for the merging result of the word and compound word features obtained by the above steps, select the feature word whose cumulative information gain reaches 95% of the original feature word information gain sum, and then construct the feature word vector.

Described content (3) off-line classification algorithm and improved tuning training and test, the off-line classification algorithm is done to the improvement facing spam classification, according to each off-line classification algorithm and improvement to the training set and the test set of step (2) gained Data preparation, using the training set to optimize training and testing of each offline algorithm and improvement, specifically:

(3.1) Improve the offline classification algorithm for spam classification, including LR based on a specific data format. The specific data format is: label index1:value1 index2:value2..., using this specific data format, LR calculates the coefficient vector The formula for the inner product of and instance is: Among them, w represents the coefficient vector (matrix), x _i represents the i-th instance vector (matrix), index _j represents the subscript of the non-zero element of the instance vector x _i , since the 0, 1 dictionary model is used, the non-zero element is 1 , label is an instance category label, usually an integer, such as 0 and 1, and value is a corresponding feature value, because what the present invention uses is a dictionary model; during the AdaBoost decision-making of differential loss, in the spam classification, the normal short message is wrong The judgment cost is higher than the misjudgment of spam text messages, so an improvement of the differential loss is proposed. When updating the weight of the training sample in each iteration, if the classification is correct in the last iteration, the update formula is If it was misclassified in the previous iteration, the update formula is

w _m,i is the weight of the i-th instance in the m-th iteration, Z _m is the normalization factor, and α _m is the weight of the m-th base classifier.

(3.2) carry out data preparation to the training set and the test set of step (2) gained according to each off-line classification algorithm and improvement;

(3.3) Use the training set to optimize training and testing of each offline algorithm and improvement, use cross-validation to tune the model parameters of SVM, and use grid search to find the optimal parameters of GBDT. The basic idea is: according to the order of importance of parameters Tuning, if only one parameter is tuned, then construct a parameter vector according to the value range of the parameter, traverse all the values in the vector, and select the best according to the prediction result; if tuning two parameters at the same time, then according to The value interval of the two parameters constructs a two-dimensional parameter matrix, which is shaped like a grid. Each grid corresponds to the value combination of the two parameters. All the grids are traversed, and the optimal parameter combination is selected based on the prediction results. For LR and AdaBoost, the optimal model is obtained by adjusting the number of iterations, and each optimal model is optimally used to predict the test set.

Described content (4) selects based on the off-line classification algorithm of evaluation index, proposes the evaluation index facing spam classification, uses this evaluation index to analyze the test result that step (3) obtains and selects optimal offline classification algorithm, specifically :

(4.1) Propose evaluation indicators for spam SMS classification, including accuracy and accuracy recall rate with Wherein TP is the number of samples whose real category is 1 (spam message) and is predicted to be 1, FP is the number of samples whose real category is 0 and is predicted to be 1, and FN is the number of samples whose real category is 1 and predicted to be 0;

(4.2) Use the evaluation index proposed in step (4.1) to analyze the test results obtained in step (3) and select the optimal offline classification algorithm.

The present invention adopts the above-mentioned technical scheme, and has the following beneficial effects:

1. Text preprocessing improves the accuracy and effectiveness of feature selection on the one hand, and avoids the loss of important information on the other hand;

2. The features selected by the feature selection method based on statistical threshold and average information gain are more discriminative than the simple method based on statistical threshold. Using N-Gram algorithm-based feature selection of double-character words and compound words and combining non-modifying entities The method of combining words into tuple features effectively avoids the information loss of feature selection based on word segmentation results, and the selected combination word features more accurately describe the unique information of spam text messages, and the average information gain cannot represent all feature words , it is proposed to further select the combined results of the above selected feature words based on the cumulative information gain, this method overcomes the difficult problem of setting the threshold based on information gain;

3. The improvement and selection of offline classification algorithms have effectively improved the recognition performance of spam text messages, and the optimal offline model has a high recognition accuracy and efficiency.

Description of drawings

Fig. 1 is the broken line graph of LR classification result about the number of iterations (a) is the broken line graph of each index of the LR classification result about the number of iterations, (b) is the broken line graph of the number of iterations of the F2-Measure of the LR classification result;

Figure 2 is a graph of the training results when the LIBSVM parameter C is set to 8;

Fig. 3 is a diagram of the optimization selection results of LIBSVM parameters C and g;

Fig. 4 is to use optimal parameter C and g to train support vector machine model on training set;

Figure 5 is the ROC curve of the AdaBoost classification results for 10 iterations;

Figure 6 is the ROC curve of the AdaBoost classification results for 20 iterations.

detailed description

Below in conjunction with specific embodiment, further illustrate the present invention, should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention, after having read the present invention, those skilled in the art will understand various equivalent forms of the present invention All modifications fall within the scope defined by the appended claims of the present application.

The offline model improvement and selection method for spam classification includes the following four aspects:

(1) SMS text preprocessing, the main preprocessing content includes: word segmentation, unified conversion of SMS text to simplified expressions, conversion of desensitized strings such as numbers to single characters, and removal of stop words;

(1.1) Use Ansj to segment the SMS text and keep the part-of-speech tag;

(1.2) Unified conversion of SMS text to simplified expressions, numbers and other desensitized strings to single characters;

(1.3) Remove stop words according to the stop word list.

(2) feature selection and expansion, according to the preprocessing result obtained in step (1), use the feature selection method to select features, construct the feature word vector, and use the feature word vector model to represent the original text message;

(2.1) Frequent word feature selection based on statistical threshold and average information gain, the threshold is an adjustable parameter, select frequent words as the feature word set according to the threshold, and decide whether to continue to adjust the threshold according to the average information gain of the feature word set;

(2.2) Feature selection based on the N-Gram algorithm for double-character words and compound words, generate text fragment sequences based on the N-Gram algorithm, filter out infrequent sequences according to the optimal statistical threshold obtained in step (2.1), and construct the remaining sequences Relevance matrix, the matrix element is the frequency of occurrence of the corresponding row and column combination sequence in the spam text, and the combination text sequence is screened according to certain standards;

(2.3) Combining non-modifying content words into tuple features, traversing all spam texts to find noun+verb\adjective combinations, and screening the tuple features obtained according to certain standards.

(2.4) Based on the feature selection of cumulative information gain, for the merging result of the word and compound word features obtained by the above steps, select the feature word whose cumulative information gain reaches 95% of the original feature word information gain sum, and then construct the feature word vector.

(3) Offline classification algorithm and improved tuning training and testing, the offline classification algorithm is improved for spam classification, according to each offline classification algorithm and improvement, the training set and test set obtained in step (2) are prepared for data, Use the training set to perform tuning training and testing for each offline algorithm and improvement;

(3.1) Improve the offline classification algorithm for spam classification, including LR based on a specific data format. The specific data format is: label index1:value1 index2:value2..., using this specific data format, LR calculates the coefficient vector The formula for the inner product of and instance is: Wherein label is instance class label, is usually an integer, such as 0 and 1, and index is the index of ordered non-zero feature, and value is the corresponding feature value, because what the present invention adopts is dictionary model; AdaBoost decision-making of differential loss When , in the classification of spam text messages, the misjudgment cost of normal text messages is higher than that of spam text messages. Therefore, an improvement of differential loss is proposed. When updating the weight of training samples in each iteration, if the classification is correct in the previous iteration, the update formula is If it was misclassified in the previous iteration, the update formula is

(3.2) carry out data preparation to the training set and the test set of step (2) gained according to each off-line classification algorithm and improvement;

(3.3) Use the training set to optimize training and testing of each offline algorithm and improvement, use cross-validation to tune the model parameters of SVM, and use grid search to find the optimal parameters of GBDT. The basic idea is: according to the order of importance of parameters Tuning, if only one parameter is tuned, then construct a parameter vector according to the value range of the parameter, traverse all the values in the vector, and select the best according to the prediction result; if tuning two parameters at the same time, then according to The value interval of the two parameters constructs a two-dimensional parameter matrix, which is shaped like a grid. Each grid corresponds to the value combination of the two parameters. All the grids are traversed, and the optimal parameter combination is selected based on the prediction results. For LR and AdaBoost, the optimal model is obtained by adjusting the number of iterations, and each optimal model is optimally used to predict the test set.

(4) Based on the selection of the offline classification algorithm of the evaluation index, an evaluation index for spam SMS classification is proposed, and the evaluation index is used to analyze the test results obtained in step (3) and select the optimal offline classification algorithm.

(4.1) Propose evaluation indicators for spam SMS classification, including accuracy and accuracy recall rate with Wherein TP is the number of samples whose real category is 1 (spam message) and is predicted to be 1, FP is the number of samples whose real category is 0 and is predicted to be 1, and FN is the number of samples whose real category is 1 and predicted to be 0;

(4.2) Use the evaluation index proposed in step (4.1) to analyze the test results obtained in step (3) and select the optimal offline classification algorithm.

For the step (3) offline classification algorithm and improved tuning training and testing, the following is a detailed description:

1) Offline classification algorithm and improvement

1.1) LR based on a specific data format

For the offline logistic regression algorithm, if the sample size of the data set is large and the feature dimension is high, because all the data needs to be loaded into the memory, the memory consumption will be very large. At the same time, SMS texts are usually short, and after being represented by the word vector space model, an extremely sparse matrix is often obtained. If the original matrix can be compressed and stored, it will not only reduce memory consumption, but also improve the training and testing efficiency of the LR model. This paper uses the data format of LIBSVM to convert the original data set:

Table 1 LIBSVM data format example

label is the instance category label, usually an integer, such as 0 and 1. index is the index of the ordered non-zero features, and value is the corresponding feature value. After using this data format, the size of the data set has changed from 2.07G to 52MB. It can be seen that the SMS matrix is very sparse. The converted example is shown in Table 2:

Table 2 Example of converting SMS data based on a specific data format

1.2) SVM tuned by cross-validation

The original data set is usually not linearly separable, including approximately linearly separable and linearly inseparable. For these two problems, support vector machines are solved by introducing slack variables and kernel techniques respectively. In addition, the optimization problem finally solved by the support vector machine is to maximize the geometric interval between the support vector and the separating hyperplane, which makes the support vector machine not only seek the correct classification, but also require the correct classification of the sample with the maximum degree of certainty, so the support vector machine can achieve very high classification accuracy. To sum up, support vector machine is applicable to more classification scenarios than logistic regression, and the classification accuracy is higher. However, the calculation of support vector machine is more complicated, and the training process is more time-consuming. The present invention uses the LIBSVM software package to train and test the support vector machine model, and optimizes parameters C and g based on S-fold cross-validation. The training function of LIBSVM is used as follows:

svmtrain [options] training_set_file [model_file]

Among them, options is used to set the parameters, training_set_file is the training data file, and model_file is used to save the training result file, that is, the model file. In this paper, RBF is selected as the kernel function, and the parameters -g r(gama) and -c cost are tuned by setting the cross-validation parameter -v n, where -g r(gama) sets the function in the kernel function, the default is 1/k, and k is the feature dimension, -c cost sets the coefficient of the penalty item, the default is 1, -v n means to use n-fold cross-validation. Finally, use the optimal parameters C and g to train the support vector machine classification model and predict the test set.

1.3) AdaBoost decision tree for differential loss

The basic idea of the AdaBoost algorithm is to train multiple classifiers. When predicting, weighted voting is used to determine the final classification result. In the process of iteratively generating each sub-classifier, the sample weight is dynamically adjusted and the weight is calculated according to the accuracy of the sub-classifier. , this idea makes the samples misclassified in the previous round of iteration get more attention in the current round of iteration, the model training is more targeted, and the prediction performance is also higher. In the classification of spam text messages, the cost of misjudgment of normal text messages is higher than that of misjudgment of spam text messages. Therefore, it is necessary to dynamically modify the sample weights during the training process, and further expand the weight of misjudgment samples from normal text messages on the basis of increasing the weight of misjudgment samples in the previous round of iteration. AdaBoost can easily do this for the sample weights judged as spam messages. Weak classifiers use decision trees because the training time complexity of decision trees is low, and overfitting can be effectively avoided through pruning techniques. The AdaBoost decision tree classification algorithm for differential loss is:

Initialize M decision trees, the weight of each decision tree is initialized to 1/M, the weight of each sample in the training set is initialized to 1/N, N is the number of training samples

Loop M times and generate a decision tree each time:

Use the decision tree algorithm to train the mth decision tree G _m (x) on the training set

Calculate the classification error rate of the mth decision tree according to the sample weight distribution

Calculate the weight of the mth decision tree

Update training sample weights:

if If a sample is correctly classified by the current round classifier

1.4) GBDT with parameter grid search

In scikit-learn, Gradient Boosting Classifier is a classification of GBDT. Similar to Adaboost, we divide important parameters into two categories. The first category is the important parameters of the Boosting framework, and the second category is the weak learner, that is, the important parameters of the CART regression tree. parameter.

a) GBDT class library Boosting framework parameters

n_estimator: The maximum number of iterations, that is, the maximum number of weak learners. If n_estimators is set too small, it is easy to underfit. If n_estimators is set too large, it is easy to overfit, so it needs to be tuned to get a moderate value. The default value of this parameter is 100. In the actual tuning process, it is usually considered together with learning_rate.

learning_rate: The weight reduction coefficient of each weak classifier, or step size, in order to avoid falling into a local optimal solution, each iteration only believes a part of the weak classifier, and makes up for it by learning more classifiers. This parameter is also A regularization method, the iterative formula of the strong classifier after introducing shrinkage technology is:

f _k (x)＝f _k-1 (x)+νh _k (x) (4-1-3)

The value range of ν is 0<ν≤1, f _k (x) represents the decision formula of the integrated classifier obtained in the k-th iteration, v is the reduction factor, and h _k (x) is the weak classifier obtained in the k-th iteration . For the same training set, in order to achieve the same fitting effect, if ν is set smaller, then it is necessary to learn more weak classifiers, that is, increase the number of iterations, so these two parameters are usually used together to optimize the algorithm The classification effect, that is, n_estimators and learning_rate are tuned together. The initial value of ν usually takes a small value, and the default value is 1.

subsample: the ratio of subsampling, the value interval is (0,1]. If the value is 1, it means that all samples are used, and subsampling is not used. If the value is a number between 0 and 1, it means Only a part of the samples will be used for training the decision regression tree of GBDT. Selecting a subsampling ratio less than 1 can avoid overfitting, reduce variance, and improve the generalization ability of the model, that is, the prediction accuracy on the test data set. But relatively Yes, it will also increase the deviation of the training samples, making the model perform poorly on the training set, and the learning is not sufficient, so the value should not be too small. Generally, a better value is between [0.5,0.8], and the default is 1.0 , that is, no subsampling is used.

b) GBDT class library weak classifier parameters

Since the weak classifier used by GBDT is a decision regression tree, the parameters of the weak classifier come from these decision regression trees.

The maximum number of features considered during division max_features: multiple types of values can be used, the default is "None", which means that all features are considered during division, if it is "log2", it means that at most log ₂ N features are considered during division; if it is "sqrt" or "auto" indicates that the most features, where N is the total feature number of the sample, that is, the dimension; if it is an integer, it indicates the absolute number of features considered; if it is a floating point number, it indicates the percentage of the number of features considered, that is, the number of features considered after the specified percentage is rounded. If the number of sample features is small, such as less than 50, use the default value. If the number of features is very large, you need to use other values to control the maximum number of features considered in the division, so as to control the generation time of the decision regression tree.

Decision tree maximum depth max_depth: If this parameter is not set, the algorithm will not limit the depth of the tree when building a decision regression tree. When the data scale is small, that is, the number of samples and the feature dimension are not very large, you can not set it This parameter, but when the sample size is large and the data dimension is high, it is necessary to control the training time of the model by limiting the maximum depth of the decision tree to avoid overfitting and improve the generalization ability of the model. The specific value depends on the distribution of the data, and the common value range is 10 to 100.

The minimum number of samples required for internal node re-division min_samples_split: This value limits the conditions for subtrees to continue to be divided. If the number of samples of a node is less than min_samples_split, it will not continue to try to select the optimal feature to analyze the data on the node. divided. The default value of this parameter is 2. If the amount of data is not large, this parameter does not need to be set. If the amount of data is very large, a larger value can be selected to improve the prediction performance of the model.

Minimum number of samples of leaf nodes min_samples_leaf: This value limits the minimum number of samples of leaf nodes. If the number of samples of a leaf node is less than the minimum number of samples, it will be pruned together with sibling nodes. The default value of this parameter is 1, which can be set to the minimum absolute number of samples, or the ratio of the minimum sample to the total number of samples. If the amount of data is not large, this parameter does not need to be set. If the amount of data is large, this parameter can be increased value.

The present invention adopts the method of grid search to optimize the parameters of GBDT. The basic idea is: to optimize according to the order of importance of the parameters. All values in the vector are selected according to the prediction results; if two parameters are tuned at the same time, then a two-dimensional parameter matrix is constructed according to the value range of the two parameters, which is shaped like a grid, and each grid corresponds to The value combination of the two parameters traverses all the grids and selects the optimal parameter combination based on the prediction results.

2) Tuning training and testing of offline classification algorithms

2.1) LR: Find the optimal regression coefficient by increasing the number of iterations to obtain the optimal classification performance. Calculate the value of each index, use F ₂ -Measure to evaluate the performance of the LR classification algorithm, and select a reasonable number of iterations based on the time complexity.

Table 3 LR classification results

Analysis of Table 3 shows that when the number of iterations is 200, each index reaches the maximum value, when the number of iterations is 200-400, the index value drops, and when the number of iterations exceeds 500, each index rises again, but is still inferior. This shows that the weight coefficient of logistic regression will gradually approach the global optimal solution as the number of iterations increases, and continue to iterate after reaching the optimum, but there will be large fluctuations due to the linear inseparability of the original problem or some misclassified samples. , of course, this phenomenon will gradually stabilize. The above analysis is clearly reflected in Figure 1.

2.2) SVM: In this experiment, the parameters -g r(gama) and -c cost are tuned by setting the cross-validation parameter -v n, and then the optimal parameters C and g are used to train the support vector machine classification model, and finally based on the support vector machine The model makes predictions on the test set. The model training results when the parameter C is set to 8 are shown in Figure 2.

In Figure 2, #iter is the number of iterations, rho is the constant item b in the decision function, nSV is the number of support vectors, and nBSV is the number of support vectors on the boundary. The trained model file contains specific support vector information, which gives the geometric interval from the support vector to the separating hyperplane. Table 4 gives some results:

Table 4 Support vectors and their geometric intervals

Because the penalty parameter and the kernel function are more important to the support vector machine, the parameters C and g are tuned by cross-validation. Here, set the value of cross validation (-v) to 5, that is, use 5-fold cross-validation to optimize parameters C and g. Specifically, you need to modify the grid.py in the python subdirectory of libsvm. At the same time, you need to unzip the gnuplot tool, then modify the grid.py file in the /libsvm/tools directory, and modify the gnuplot path in the file to the actual path. Then, copy the grid.py file and the python.exe file in the python installation directory to the libsvm/windows directory, enter the python grid.py trainData command, and after execution, the optimal parameters C and g can be obtained. Using the spam SMS training data set for experiments, the optimal parameters C and g are obtained as shown in Figure 3.

It can be seen from the experimental results in Figure 3 that the optimal penalty parameter C is 8, and the optimal g value is 0.03125. For the selected training samples, the accuracy of the support vector machine classification model trained by these parameters reaches 99.4848%. During the training process, with the gradual tuning of parameters C and g, the accuracy rate is getting higher and higher, and the model is gradually converging. Using the optimal values of parameters C and g to train the SVM model is shown in Figure 4.

Test the trained support vector machine model, predict 140,000 text messages to be identified, and calculate the results of each indicator, as shown in Table 5:

Table 5 LIBSVM test results

It can be seen from Table 5 that the overall classification performance of the support vector machine classification model is very good, and the classification accuracy and F value are higher when the penalty parameter is set to 8. But compared to the recall rate, we accept that spam text messages are identified as normal text messages, but the loss will be greater because normal text messages are misjudged and intercepted, so we pay more attention to precision, that is, the real spam text messages among the text messages identified as spam text messages The ratio, the larger the ratio, the better (the larger the number of normal text messages in the identified spam text messages). From the training time of the support vector machine, we can see that when faced with massive SMS text data, the support vector machine classification model may not meet the real-time requirements. If the algorithm model with low complexity is used and the indicators can achieve similar prediction accuracy and performance at the same time, it should be given priority.

2.3) AdaBoost: Iterates round by round, repeating the process of training and adjusting sample weights until the training error rate is 0 or the number of iterations, that is, the base classifier reaches a certain number.

Table 6 Improved AdaBoost classification results

It can be seen from Table 6 that the improved AdaBoost classification results converge very quickly, and after 20 iterations, the values of each index will not change. From the data of various indicators, AdaBoost is better than LR and worse than SVM. Judging from the indicator of precision, the improved AdaBoost hardly recognizes normal text messages as spam text messages. Experiments show that the improved AdaBoost achieves the desired effect. And the time complexity of algorithm training is obviously lower than that of support vector machine.

Yet another tool for measuring disequilibrium in classification is the ROC curve, which stands for Receiver Operating Characteristic. The ROC curve shows the change of the false positive rate and true rate with the threshold. The upper right corner corresponds to discriminating all samples as spam text messages, and the upper left corner corresponds to discriminating all samples as normal text messages. The dashed lines are the result of random guessing. The ROC curve corresponding to a good classifier should be in the upper left corner as much as possible, that is, while the false positive rate is very low, the true rate is still high. In spam text message identification, this means that the trained classification model can identify almost all spam text messages, and rarely misjudge normal text messages as spam text messages.

In this experiment, the ROC curves of 10 iterations and 20 iterations are shown in Figure 5 and Figure 6. In the ROC curves of Figure 5 and Figure 6, two lines are given, a dashed line and a straight line. The horizontal axis in the figure is the proportion of false positives That is, the proportion of the original normal text messages identified as spam text messages, and the vertical axis is the proportion of real cases That is the recall rate.

An analysis index of the ROC curve is the area under the dotted line AUC, the area under the two curves in the figure is greater than 0.98, and it can be clearly seen in the figure that after 20 iterations, the curve moves closer to the upper left corner, and the area increase. In terms of performance, the classifier achieves the best effect when iterating to 20 times. The closer the AUC is to 1, the better the classifier, and the value of AUC at random guessing is 0.5.

2.4) GBDT: The experimental results obtained by step-by-step parameter adjustment in this experiment are as follows: In the case of selecting the default for any parameter, the Accuracy of the training data training = 99.82%, AUC Score (Train) = 0.999899, it can be seen that the model fits the training data well . The following will improve the generalization ability of the model through parameter adjustment. First start with the step size (learning rate) and the number of iterations (n_estimators). In general, start by choosing a smaller step size to grid search for the best number of iterations. Here, the initial value of the step size is set to 0.1. Grid search is performed on the number of iterations, and the results are shown in Table 7:

Table 7 GBDT grid search optimal n_estimators result table

It can be seen from Table 7 that the optimal number of iterations is 90, at this time the mean is the largest, reaching 0.9961, and the std is the smallest, only 0.00367. The optimal number of iterations has been found, and now the parameters of the decision tree are adjusted. Firstly, conduct a grid search on the maximum depth max_depth of the decision tree and the minimum number of samples min_samples_split required for re-division of internal nodes. The experimental results are shown in Table 8:

Table 8 GBDT grid search optimal max_depth and min_samples_split result table

It can be seen from Table 8 that the optimal maximum tree depth is 6, and the optimal value of the minimum number of samples required for internal node subdivision is 100. Since the depth of the decision tree is 6 is a relatively reasonable value, the optimal value of the parameter max_depth can be set to 6. The minimum number of samples min_samples_split required for internal node subdivision cannot be determined for the time being, because this parameter is still It is associated with other parameters of the decision tree. Next, adjust the parameters of the minimum number of samples min_samples_split and the minimum number of samples of leaf nodes min_samples_leaf for internal node subdivision. The output results are shown in Table 9:

Table 9 GBDT grid search optimal min_samples_leaf and min_samples_split result table

It can be seen from Table 9 that the optimal min_samples_leaf value is 6, and the min_samples_split value is 100, both of which are at the search boundary value, and further debugging is necessary to be smaller than the boundary value. Further set the lower limit of min_samples_leaf to 4, and the lower limit of min_samples_split to 60, and the upper limit is intersected with the previous one, so as to find the optimal result. The debugging results are shown in Table 10:

Table 10 Table 9 Continued Further Grid Search Debugging

It can be seen from Table 10 that the final optimal values of the drop-down parameters min_samples_leaf and min_samples_split have not changed, which are 6 and 100, respectively. Put the new tuned parameters into the GBDT class to train the model and predict the test data. The output results show: Accuracy=99.83%, AUC Score=0.999989. Compared with the result of not adjusting parameters at all at the beginning, it can be seen that the accuracy has improved. Now perform a grid search on the sub-sampling ratio, and the output results are shown in Table 11:

Table 11 GBDT grid search optimal subsample result table

It can be seen that the optimal sub-sampling ratio is 0.6. Now that all the tuning parameter results have been basically obtained, the parameters learning_rate and n_estimators can be adjusted to enhance the generalization ability of the model. The test results are shown in Table 12:

Table 12 GBDT optimization iteration number and step size result table

Table 13 GBDT optimal parameter table

From Table 13, it can be seen that doubling the number of iterations and halving the learning rate AUC score will decrease slightly. The reason is to enhance the generalization ability of the model and prevent overfitting. At the same time, the subsample ratio is reduced, thereby reducing the training data. The degree of fit of the set. At the same time, it can be found that reducing the step size and increasing the number of iterations can reduce a certain deviation, that is, the degree of fitting of the training data, on the basis of ensuring the generalization ability of the model. Furthermore, if the step size is too small, the fitting effect will deteriorate instead, that is to say, it is not usually possible to set it too small. Finally, use all the optimal parameters after tuning (see Table 13) to train the model and use it for testing data, and the obtained index values are shown in Table 14:

Table 14 GBDT classification results

3) Offline classification algorithm selection based on evaluation index

Analysis of Table 14 shows that the classification accuracy of GBDT, F ₂ -Measure is the best among all offline models, and the training time of the model is better than that of the support vector machine. When dealing with static SMS data sets with relatively stable spam feature information , the classification performance using GBDT is the best.

After conducting experiments on all offline models, the result list of predicting the test SMS data after each model is tuned is shown in 15:

Table 15 Optimal classification results of each offline classification algorithm

Analysis of Table 15 shows that logistic regression is the fastest classification algorithm with the worst accuracy. SVM improves the classification accuracy, but the time complexity is also the highest. The improved AdaBoost increases the penalty for misjudgment of normal SMS, and the precision is 100. %, that is, all the real categories of text messages identified as spam text messages are spam text messages, and there is no misjudgment of normal text messages. But while GBDT achieves better accuracy, the precision is also 100%. It can be seen from the last column in the table that the time complexity of GBDT is better than that of SVM.

Claims (4)

1. a kind of improve and system of selection towards the off-line model that refuse messages are classified, it is characterised in that comprises the following steps：

(1) feature selecting and extension, select feature using feature selection approach, construct feature term vector, use feature term vector Model represents original short message text；

(2) offline sorting algorithm and improved tuning training and test, make what is classified towards refuse messages to offline sorting algorithm Improve, data preparation is carried out to the training set obtained by step (1) and test set according to each offline sorting algorithm and improvement, instruction is used Practice each off-line algorithm of set pair and improve and carry out tuning training and test；

(3) the offline sorting algorithm selection based on evaluation index, is proposed the evaluation index classified towards refuse messages, is commented using this Valency index is analyzed the test result obtained by step (2) and selects optimal offline sorting algorithm.

2. according to claim 1 improve and system of selection towards the off-line model that refuse messages are classified, it is characterised in that Step (1) is concretely comprised the following steps：

(1.1) the frequent word feature selecting based on statistical threshold and average information gain, threshold value is adjustable parameter, is selected according to threshold value Frequent word is selected as feature word set, is decided whether to continue to adjust threshold value according to the average information change in gain situation of feature word set；

(1.2) two-character word and portmanteau word feature selecting based on N-Gram algorithms, word segment sequence is produced based on N-Gram algorithms Row, non-Frequent episodes are fallen according to the optimal statistical threshold filtering that step (1.1) is obtained, by remaining sequence construct incidence matrix, Matrix element is occurrence frequency of the correspondence ranks composite sequence in refuse messages text, and text is combined according to certain standard screening Word sequence；

(1.3) non-modified property notional word is combined into tuple feature, travel through all refuse messages texts find noun+verbs describe Word combination, is screened according to certain standard to gained tuple feature；

(1.4) feature selecting based on cumulative information gain, to the word and the merging knot of portmanteau word feature obtained by above step Really, selection cumulative information gain reaches 95% Feature Words of primitive character word information gain summation, so construction feature word to Amount.

3. according to claim 1 improve and system of selection towards the off-line model that refuse messages are classified, it is characterised in that Step (2) are concretely comprised the following steps：

(2.1) improvement classified towards refuse messages, including the LR based on format, certain number are made to offline sorting algorithm It is according to form：label index1:value1index2:Value2..., using the format, LR design factor to Formula is when amount and the inner product of example：Wherein label For example class label, usually integer, the index for the non-zero characteristics that index is ordered into, value is corresponding feature value, by In using dictionary model, therefore the value of non-zero characteristics is 1；During the AdaBoost decision-makings of differentiation loss, in refuse messages classification, Normal short message erroneous judgement cost is judged by accident higher than refuse messages, therefore proposes the improvement of differentiation loss, and training sample is updated in each iteration During this weight, if correctly classified in upper once iteration, newer isIf in upper once iteration Mistake is classified, and newer is

(2.2) data preparation is carried out to the training set obtained by step (1) and test set according to each offline sorting algorithm and improvement；

(2.3) using training set is to each off-line algorithm and improves progress tuning training and test, using cross validation tuning SVM's Model parameter, GBDT optimized parameter is found using grid search, is specifically：Tuning is carried out according to parameter significance sequence, such as Fruit is only to an arameter optimization, then according to the interval constructing variable of parameter vector, all values during traversal is vectorial, Select optimal according to predicting the outcome；If carrying out tuning to two parameters simultaneously, then constructed according to the interval of two parameters The parameter matrix of two dimension, shape such as grid, the valued combinations of two parameters of each grid correspondence travel through all grids, based on pre- Survey result selection best parameter group.For LR and AdaBoost, optimal models is obtained by adjusting iterations, it is optimal to use Each optimal models is predicted to test set.

4. according to claim 1 improve and system of selection towards the off-line model that refuse messages are classified, it is characterised in that Step (3) are concretely comprised the following steps：

(3.1) evaluation index classified towards refuse messages, including accuracy rate accuracy, accuracy are proposed Recall rateWithWherein TP is that true classification is that 1 (rubbish is short Letter) and be predicted as 1 number of samples, FP is the number of samples that true classification is predicted as 1 for 0, FN be true classification be 1 and It is predicted as 0 number of samples；

(3.2) evaluation index proposed using step (3.1) is analyzed and selected most to the test result obtained by step (2) Excellent offline sorting algorithm.