CN113590819B - Large-scale category hierarchical text classification method - Google Patents

Abstract

The present invention provides a large-scale category-level text classification method. This method applies a deep learning network to a plane classifier method and a global classifier method, and performs classification calculations on them respectively. On the one hand, the plane classifier method is different from the machine learning method. Compare the classification performance. On the other hand, compare the deep learning method to whether the global classifier method performs better than the flat classifier after learning the hierarchical dependence information; when using the flat classifier, it is consistent with the machine learning method and does not consider the hierarchical structure. , the loss function only considers the empirical loss function of the training set; when using a global classifier, the hierarchical structure is taken into consideration, and the penalty of the regularization term is added to the loss function; based on the classic neural network models CNN and RNN in text classification tasks, attention will be paid to The force mechanism is combined with the RNN model and CNN model to capture the key information of the text, which is conducive to further promoting the informatization and automation of government affairs.

Description

A large-scale category-level text classification method

Technical field

The present invention relates to the field of hierarchical text classification, and more specifically, to a large-scale category hierarchical text classification method.

Background technique

Government procurement refers to the behavior of state agencies, institutions and organizations at all levels using fiscal funds to purchase goods, projects and services that are within the centralized procurement catalog formulated in accordance with the law or above the procurement quota. Compared with the procurement behavior of many other social entities, there are significant differences in the procurement behavior of the government as a subject: the funds for government procurement mainly come from government fiscal revenue, and ultimately the ultimate source is all taxpayers. Therefore, It has strong public attributes and needs to be responsible to taxpayers. At the same time, government procurement behavior is non-commercial. It does not aim to obtain profits, but to meet the basic needs for the operation of government affairs or to provide basic and inclusive public services to the society through procurement. . Therefore, during the government procurement process, how to ensure the openness and standardization of procurement behavior has always attracted much attention from the government. At present, although information system management has brought many conveniences to the management of government procurement business, there are still many serious challenges that need to be solved. In order to make better use of government public procurement information and tap its value, this invention will explore how to classify appropriate procurement projects based on procurement project information based on hierarchical text classification research and government procurement data to further promote government affairs information. ization and automation.

Facing the trend of government affairs disclosure and the requirements of the big data era, it is of practical significance to conduct in-depth analysis and research on massive government affairs disclosure data and explore its value. Among them, the main purpose of the present invention is to explore the classification of government procurement projects and find effective hierarchical classification methods, in order to use past government procurement project data to give automatic classification suggestions, thereby helping future staff to be more efficient and efficient. Accurately classify procurement items. According to the actual background, the project names, items and other procurement information of a large number of government procurement projects are obtained from the official government website as the data set used in the research of this invention, and the data set is analyzed and preprocessed.

In hierarchical text classification, training samples are mainly labeled by establishing a hierarchical classifier. The last category node at the end of each branch of the hierarchical category structure is defined as a leaf node, and the rest are defined as trunk nodes. The present invention considers a planar classifier. , the advantages and disadvantages of local classifiers and global classifiers. In machine learning methods, the plane classifier method is used as the baseline, that is, all leaf nodes are placed in the same plane for classification. The deep learning network is applied to the flat classifier method and the global classifier method, and classification calculations are performed respectively. On the one hand, the classification performance of the flat classifier method is compared with the machine learning method, and on the other hand, the deep learning method is compared with the global classifier method. Whether classifier methods perform better than flat classifiers after learning hierarchical dependence information. In order to solve the problem of long-distance dependence in deep learning methods, Wang et al. proposed a relationship extraction method that combines LSTM and attention mechanism to classify corpus. Hao et al. proposed a full-time improved formula P-IDF for the purpose of large-scale text classification to improve the performance of traditional machine learning classification methods. Wu et al. proposed the NMF-SMVM text classification algorithm based on the hierarchical model and SEAN attention mechanism to solve the problem of lack of hierarchical structure features in the text classification process. However, these methods either use machine learning algorithms or ignore the hierarchical structure in large-scale text classification.

Contents of the invention

The present invention provides a large-scale category-level text classification method. This method is based on the classic neural network models CNN and RNN in text classification tasks, and combines the attention mechanism with the RNN model and the CNN model to capture the key information of the text. It is conducive to further promoting the informatization and automation of government affairs.

In order to achieve the above technical effects, the technical solutions of the present invention are as follows:

A large-scale category-level text classification method, including the following steps:

S1: Collect government procurement announcement data;

S2: Use machine learning to classify the data collected in step S1;

S3: Use the classification result of step S2 as the benchmark, and use the ARCNN model to classify the data collected in step S1.

Further, the specific process of step S1 is:

Collect government procurement publicity data, including transaction announcements and bidding announcements, and preprocess the collected government procurement publicity data:

(1) Remove illegal characters from text data and convert traditional characters into simplified characters;

(2) Segment Chinese words and remove stop words;

(3) Training word vectors.

Further, in step S2, the plane classifier method for hierarchical classification problems is used for classification. In the feature engineering stage, BOW and TF-IDF are selected as word features; in the classifier training stage, the naive Bayes classifier and Logistic regression classifier are selected. , support vector machine classifier, and decision tree classifier to classify experimental data; the plane classifier does not consider the hierarchical structure, so the loss function in this part only considers the empirical loss function of the training set.

Further, in the step S3, Chinese text segmentation is performed on the collected government procurement announcement data, and the word sequence D=[w ₁ ; w ₂ ; ...; w _n ] is obtained, and let w represent the parameter set in the network structure, p(k|D, w) represents the probability that the text instance belongs to category k. The word sequence D = [w ₁ ; w ₂ ; ...; w _n ] is mapped to the word vector space through Word Embedding to obtain the vector representation of the word sequence. : [e(w ₁ ); e(w ₂ ); ...; e(w _n )];

For the input word sequence, the bidirectional recurrent neural network model Bi-RNN is used to learn the contextual information of the current word w _i , and obtain the upper information c _l ( _wi ) and the lower information c _r (wi ₎ , where, e(w _i ) is a non-sparse vector with |e| real-valued elements, c _l ( _wi ) and _cr (wi ₎ are both non-sparse vectors with |c| real-valued elements.

Further, in step S3, based on the structure of the bidirectional recurrent neural network model Bi-RNN, the context information c _l ( _wi ) and _cr ( _wi ) are hidden layer information, and their calculation method is as follows:

c _l ( _wi )=f(W ^(l) c _l ( _wi-1 )+W ^(sl) e( _wi-1 ))

c _r ( _wi )=f(W ^(r) c _r ( _wi+1 )+W ^(sr) e(wi ₊₁ ))

Among them, c _l (wi _-1 ) comes from the hidden layer information at the word w _i-1 above, e(wi _-1 ) comes from the WordEmbedding value of the word w _i-1 above, W ^{(l )} is the transfer matrix that transfers the hidden layer information backward, W ^(sl) is the transfer matrix that transfers the input WordEmbedding value information backward, f is the nonlinear activation function, and performs the aggregation of the above hidden layer information and the input information After activation, it is passed backward to the current hidden layer. For the first word in the document, since there is no context, the shared parameter c _l (w ₁ ) is used.

Furthermore, based on the bidirectional recurrent neural network model Bi-RNN, through forward and reverse cycles, for the current word w _i , the current input information e(wi ₎ and its hidden layer information c _l (wi ₎ can be obtained , c _r ( _wi ); cascade and splice the current input information and hidden layer information to obtain the text representation:

x _i =[c _l ( _wi ); e ( _wi ); _cr ( _wi )].

Furthermore, contextual information is now obtained at each word, but the degree of utilization is not exactly the same, and not all words contribute equally to the representation of text meaning. In order to allow the model to better focus on the key parts of the text, For the output x _i of the Bi-RNN module, the self-attention model is used to extract more important words in the text, and the text representation after attention marking is obtained:

The text representation x _i obtained in the RNN module is encoded, and the query encoding Query: q _i , key encoding Key: k _i and value encoding Value: v _i are obtained through linear mapping f _q , f _k , and f _v respectively;

q _i =f _q (x _i )

k _i = f _k ( _xi )

v _i = f _v (x _i )

Calculate the similarity between Query and Key to obtain the attention score as a measure of word importance and attention; then, use the softmax function to normalize the score to obtain the attention weight value:

e _{i, j} = a (q _i , k _j )

a _{i, j} = softmax(e _{i, j} )

According to the obtained normalized weight, the value of the text is weighted and summed to obtain the text representation after attention marking:

Furthermore, the current word w _i obtains the text representation after attention transformation It is further convolved and the tanh function is used as the activation function to obtain the latent semantic vector of w _i />

In the convolutional layer, let the kernel size of the filter=1. This is because It actually contains the context information of w _i . After obtaining the latent semantic vectors of all words, the maximum pooling layer is used to calculate, and we get/> Among them/> The kth element of is/> The maximum value of the kth element:

After the pooling layer, text representations of various lengths are converted into fixed-length vectors, and the information in the entire text is captured. The maximum pooling layer is used. After the maximum pooling of the pooling layer, it enters the output layer and passes through a full The connection layer and softmax function get the probability that the text instance belongs to the category:

In the above forward propagation process, the parameters that need to be learned through model training include:

w={E, b ⁽²⁾ , b ⁽⁴⁾ , c _l (w ₁ ), _cr (w _n ), W ⁽²⁾ , W ⁽⁴⁾ , W ^(l) , W ^(r) , W ^(sl) , W ^(sr) }.

Further, a category hierarchical tree N, the nodes of the tree are labels of each category at all levels, and the mandatory leaf node prediction method is selected for study. For the M text instances included in the input training set, it is recorded as where x _i ∈X is the instance space of text embedding, t _i ∈T represents the category,/> For the set of all leaf nodes in the hierarchical structure, define the function π: N→N, indicating that π(n) is the ancestor node of node n; let C _n represent the set of all child nodes of node n, define y _in = (2I( t _i =n)-1)∈{-1, 1} is a binary variable used to indicate whether sample x _i belongs to category n∈T; the prediction model is recorded as function f: TRAIN obtains the minimum loss of f.

Furthermore, the loss function adopted consists of the empirical risk of the data set and a regularization term used to penalize the complexity of the prediction function f, which is defined as follows:

The loss function adopted consists of the empirical risk of the data set and a regularization term used to penalize the complexity of the prediction function f, defined as follows:

Loss＝λ(w)+C×R _emp

Among them, λ(w) is the regularization term, R _emp is the empirical risk of the data set, and C is the weight parameter that controls the balance between the empirical risk and the complexity of the control f;

For the parameter set w, in addition to the forward propagation method, it is decomposed into each node to form a parameter set based on the node composition: w = {w _n : n∈N}, that is, each node in the hierarchical structure n is associated with the parameter group w _n ; for the regularization term, the hierarchical structure is learned during the training process by introducing the hierarchical dependency of the tree structure into the regularization term of w. The form of the regularization term is as follows:

For the empirical loss function, the loss due to instances at leaf nodes in the category hierarchy:

Among them, L can be all convex loss functions in different models.

Compared with the existing technology, the beneficial effects of the technical solution of the present invention are:

The present invention uses the deep learning method to classify experimental data, applies the deep learning network to the plane classifier method and the global classifier method, and performs classification calculations on them respectively. On the one hand, the classification performance of the plane classifier method is compared with the machine learning method. , on the other hand, compare the deep learning method to whether the global classifier method has better performance than the flat classifier after learning hierarchical dependence information; when using the flat classifier, it is consistent with the machine learning method, regardless of the hierarchical structure and loss function Only the empirical loss function of the training set is considered; when using a global classifier, the hierarchical structure is taken into consideration, and the penalty of the regularization term is added to the loss function; based on the classic neural network models CNN and RNN in text classification tasks, the attention mechanism is combined with The combination of RNN model and CNN model can capture the key information of the text, which is conducive to further promoting the informatization and automation of government affairs.

Description of drawings

Figure 1 is a schematic diagram of the topology structure of the ARCNN model in the present invention;

Figure 2 is a schematic diagram of the Self-Attention mechanism used in RNN output (redrawing self-attention) in the present invention;

Figure 3 is a definition diagram of the category hierarchy tree and related function sets.

Detailed ways

The drawings are for illustrative purposes only and should not be construed as limitations of this patent;

In order to better illustrate this embodiment, some components in the drawings will be omitted, enlarged or reduced, which does not represent the size of the actual product;

It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

The technical solution of the present invention will be further described below with reference to the accompanying drawings and examples.

As shown in Figure 1, a large-scale category-level text classification method uses the RNN model to extract contextual information and the CNN model to capture local correlations, and introduces the Attention mechanism to obtain the more critical internal structure. Word information also improves the intuitiveness and interpretability of the model, which mainly includes the following steps:

(1) Obtain government procurement announcement data, which usually includes two types of announcements: transaction announcements and bidding announcements. and preprocess text data.

(2) Use common machine learning methods to classify the experimental data as a baseline for the data set, which can be used as a reference for comparison of model performance between subsequent deep learning methods and the ARCNN model used in the present invention.

(3) Use the ARCNN model to classify experimental data.

The overall framework diagram of the algorithm of the present invention is shown in Figure 1. The RNN model and the CNN model are combined. After learning the context information of the text through the RNN structure, the text information is input into the CNN structure for convolution.

The input of the model is text data, and the output is the hierarchical category label corresponding to the text data. For the input text example, in the present invention, it is a Chinese text, so the Chinese text word segmentation preprocessing is first performed on it to obtain a set of word sequences D = [w ₁ ; w ₂ ; ...; w _n ] (if it is an English text, Then the word sequence can be obtained naturally according to the space character). Let w represent the parameter set in the network structure, and p(k|D, w) represent the probability that the text instance belongs to category k.

The word sequence D = [w ₁ ; w ₂ ; ...; w _n ] is mapped into the word vector space through Word Embedding, and the vector representation of the word sequence is obtained: [e(w ₁ ); e(w ₂ ); ...; e (w _n )]. At the same time, for the input word sequence, a bidirectional recurrent neural network model (Bi-RNN) is used to learn the contextual information of the current word w _i , and obtain the above (left) information c _l (w _i ) and the below (right) information c _r (w _i ). Among them, e( _wi ) is a non-sparse vector with |e| real-valued elements, and c _l ( _wi ) and _cr ( _wi ) are both non-sparse vectors with |c| real-valued elements.

Based on the structure of the Bi-RNN module, the context information c _l ( _wi ) and _cr (w _t ) are hidden layer information, and their calculation method is as follows:

c _l ( _wi )=f(W ^(l) c _l ( _wi-1 )+W ^(sl) e( _wi-1 ))

c _r ( _wi )=f(W ^(r) c _r ( _wi+1 )+W ^(sr) e(wi ₊₁ ))

Among them, c _l (wi _-1 ) comes from the hidden layer information at the word w _i-1 above, e(wi _-1 ) comes from the WordEmbedding value of the word w _i-1 above, W ^{(l )} is the transfer matrix that transfers the hidden layer information backward, W ^(sl) is the transfer matrix that transfers the input WordEmbedding value information backward, f is the nonlinear activation function, and performs the aggregation of the above hidden layer information and the input information After activation, it is passed backward to the current hidden layer. For the first word in the document, since there is no context, the shared parameter c _l (w ₁ ) is used.

In the same way, the calculation method of _cr ( _wi ) is similar to c _l ( _wi-1 ), except that the hidden layer information and input information at the following word w _i-1 are used, and the information is passed forward. For the last word in the document, since there is no context, the shared parameter c _r (w _n ) is used.

Here, the two-way loop process of the neural network can use a more complex RNN structure to learn to obtain the context information of w _i , such as LSTM, GRU, etc.

After the Bi-RNN module, through forward and reverse cycles, for the current word w _i , the current input (self) information e(wi ₎ and its hidden layer (context) information c _l (wi ₎ , c can be obtained _r (w _i ). By cascading the current input information and the hidden layer information, a text representation can be obtained:

x _i =[c _l ( _wi ); e ( _wi ); c _r ( _wi )]

Compared with traditional neural network models that only use partial information of the text, this text representation method can better eliminate possible ambiguities in the word w _i by using contextual information.

Although contextual information is now obtained at every word, the degree of utilization is not exactly the same, and not all words contribute equally to the representation of text meaning. Therefore, in order to allow the model to better focus on the key parts of the text, for the output x _i of the Bi-RNN module, a self-attention model is used, see Figure 2, to extract more important words in the text and obtain the attention mark The text representation after.

The specific calculation process in the Self-Attention module is as follows. First, the text representation x _i obtained in the RNN module is encoded, and Query (query encoding) q _i , Key (key encoding) k _i and Value (value encoding) _v are obtained respectively through linear mapping f _q , f _k , and f v _i :

q _i =f _q (x _i )

k _i = f _k ( _xi )

v _i = f _v (x _i )

Subsequently, the similarity between Query and Key is calculated, and the attention score is obtained as a measure of word importance attention. Then, the score is normalized using the softmax function to obtain the attention weight value:

e _{i, j} = a (q _i , k _j )

a _{i, j} = softmax(e _{i, j} )

According to the obtained normalized weight, the value of the text is weighted and summed, and the text representation after attention marking can be obtained:

After passing through the Attention module, the current word w _i obtains the text representation after attention transformation. It is further convolved and the tanh function is used as the activation function to obtain the latent semantic vector of w _i />

In the convolutional layer, the kernel size of the filter can be set=1, because Actually contains the context information of w _i . However, in practice, using a larger kernel size (such as [2, 3, 4]) can achieve better results. This may be because the context information of w _i can be further learned when the filter window is larger than 1.

After obtaining the latent semantic vectors of all words, the maximum pooling layer is used to calculate, and we get Among them/> The kth element of is/> The maximum value of the kth element:

After the pooling layer, various text representations of different lengths are converted into fixed-length vectors, and the information in the entire text is captured. A max pooling layer is used here because the model only needs to focus on the most important latent semantic factors in the text.

After the maximum pooling of the pooling layer, the output layer is entered. Through a fully connected layer and the softmax function, the probability that the text instance belongs to the category can be obtained:

In the above forward propagation process, the parameters that need to be learned through model training include:

w={E, b ⁽²⁾ , b ⁽⁴⁾ , c _l (w ₁ ), _cr (w _n ), W ⁽²⁾ , W ⁽⁴⁾ , W ^(l) , W ^(r) , W ^(sl) , W ^(sr) }

The text classification task in the present invention belongs to text classification of large-scale hierarchical categories, and there is a serious category imbalance. In this regard, the present invention adopts a recursive regularization term for solving large-scale hierarchical category text classification, which not only utilizes the hierarchical dependencies between class labels to improve performance, but also maintains the scalability of the model across large-scale hierarchical categories ( That is, to solve the problem of class imbalance).

For the category hierarchical structure of the research object of the present invention, see Figure 3. It is regarded as a category hierarchical tree N. The nodes of the tree are labels of each category at all levels, and the method of mandatory leaf node prediction is selected for research. For M text instances contained in the input training set, let where x _i ∈X is the instance space of text embedding, t _i ∈T represents the category,/> A collection of all leaf nodes in the hierarchy. Define the function π: N→N, indicating that π(n) is the ancestor node of node n. Let C _n represent the set of all child nodes of node n, and define y _in =(2I(t _i =n)-1)∈{-1, 1} as a binary variable used to indicate whether sample x _i belongs to the category n∈T. The prediction model in the present invention is recorded as function f: X→T, and the goal is to obtain the minimum loss of f for a given TRAIN.

The loss function used in this invention consists of the empirical risk of the data set and a regularization term used to penalize the complexity of the prediction function f, and is defined as follows:

Loss＝λ(w)+C×R _emp

Among them, λ(w) is the regularization term, R _emp is the empirical risk of the data set, and C is the weight parameter that controls the balance between the empirical risk and the complexity of the control f.

For the parameter set w, in addition to the forward propagation method, it can also be decomposed into each node to form a parameter set based on the nodes: w = {w _n : n∈N}, that is, there is a hierarchical structure Each node n of is associated with a parameter group w _n .

For the regularization term, we hope to learn the hierarchical structure during the training process by introducing the hierarchical dependence of the tree structure into the regularization term of w. The form of the regularization term is as follows:

This regularization term is based on modeling hierarchical dependencies. As can be seen from its form, it encourages the parameters of a node to be as similar as possible to the parameters of its ancestor nodes and neighboring nodes (the ancestor nodes are the same and at a lower level). On the one hand, this helps to train model parameters while utilizing the information of class-level dependencies. On the other hand, it helps to share the sample strength throughout the hierarchy. For those classes with few samples, it can also be used from classes with more samples. Obtain information from classes to produce better classification models when samples are limited.

For the empirical loss function, defined as the loss incurred by instances at leaf nodes in the category hierarchy:

Among them, L can be all convex loss functions in different models.

Observe and understand the basic situation of the government procurement data and information obtained. The government procurement data information contains a total of 1,313,025 pieces of data, each of which contains a total of 7 elements including announcement title, procurement project name, item, purchasing unit, administrative region, announcement time, extended information, and website address. In the text classification task of the present invention, the "purchasing item name" is input as the sample of the classification task, and the "category" is output as the category corresponding to the sample. Before conducting the experiment, first understand the basic situation of the experimental data and perform data preprocessing. Before performing the classification task, the present invention divides the experimental data set according to the ratio of 7:2:1 to obtain a training set, a verification set, and a test set, and uses the evaluation index of the test set as a reference standard.

The present invention uses common machine learning methods to classify experimental data, which serves as the baseline of the present invention's data set and serves as a reference for comparison of model performance between subsequent deep learning methods and the ARCNN model used in the present invention. In the construction of the baseline, the present invention adopts the plane classifier method of hierarchical classification problem, placing all leaf nodes in the same plane for classification. In the feature engineering stage, BOW and TF-IDF are selected as word features. In the classifier training stage, the naive Bayes classifier, logistic regression classifier, support vector machine classifier, and decision tree classifier are selected to classify the experimental data. Since the flat classifier does not consider the hierarchical structure, the loss function in this section only considers the empirical loss function of the training set.

The present invention uses a relatively popular deep learning method proposed in recent years to classify experimental data as a reference for the classification performance of the ARCNN model proposed by the present invention. Among them, the deep learning network is applied to the flat classifier method and the global classifier method, and classification calculations are performed respectively. On the one hand, the classification performance of the flat classifier method is compared with the machine learning method, and on the other hand, the deep learning method is compared. Compare whether the global classifier method performs better than the flat classifier after learning hierarchical dependence information. When using a flat classifier, consistent with the machine learning method, the hierarchical structure is not considered, and the loss function only considers the empirical loss function of the training set. When using a global classifier, the hierarchical structure is taken into consideration and the penalty of the regularization term is added to the loss function.

The parameters of the deep learning method used in this invention are set as follows:

(1)Word vector model parameters

After word segmentation preprocessing is performed on the procurement project name, the resulting word segments are represented by word vectors. In this invention, the Skip-gram model in Word2Vec is used to perform word vector pre-training on all name segmentations obtained from government procurement data, and a 200-dimensional word vector is obtained. During the training process, the context word range is set to 2, that is, a total of 4 contexts are considered. word.

(2) Deep learning training parameters

In the neural network, the present invention sets the batch size used in training to 64, the number of epochs to 5, the learning rate decay step to 1000, the learning rate decay rate to 0.1, and the gradient threshold to 100. Use "Adam" as the optimizer of the neural network and set the learning rate to 0.008. During training, the dropout of the hidden layer is set to 0.5. The activation function of the fully connected layer is set to the tanh function, and the activation function of the output layer is set to the sigmoid function. In the loss function, L2 lambda is set to 0, and a loss function with a recursive regularization term is used. The empirical risk function of the training set uses the binary cross entropy loss function (Binary Cross Entropy Loss). When predicting, set the prediction threshold to 0.5.

In the hidden layer of the present invention, a variety of neural network model structures are used, including FastText, TextCNN, TextRNN, RCNN, ARCNN and other models. Among them, except for the ARCNN model proposed by the present invention, the parameters of the other models are the parameters corresponding to the best classification results obtained after multiple parameter adjustments. Their parameters are not the focus of the present invention and will not be described again.

The core of the ARCNN model mainly has a 3-layer network structure. When compared with other models, the parameter settings used are as follows. In the RNN layer, a bidirectional RNN module is set, the number of hidden layer neurons is 64, and the structure of the RNN module is set to the Bi-GRU structure. In the Attention layer, set the attention dimension to 16, the Attention method is the self-attention method and uses DOT Attention. In the CNN layer, set the number of convolution kernel channels to 64, the convolution kernel size to [2, 3, 4], and the pooling layer uses the maximum pooling method.

The present invention combines two feature extraction methods with each machine learning classifier method, and calculates its Micro-F1 and Macro-F1 scores respectively. The results are shown in Table 1:

Table 1

The experimental data set is studied using a flat classifier strategy and a machine learning algorithm. Observing the experimental results shows that the machine learning algorithm has certain difficulties when facing large-scale category imbalance data. The highest Micro-F1 is BOW+Logistic Regression model, reaching 0.4617. In terms of feature engineering comparison, the BOW method performs better than the TF-IDF method on this data set, except in In addition to the slightly better results achieved by the TF-IDF method on the Bayes model, the BOW method has obvious advantages on other models, especially the difference between the Logistic Regression model and the DT model, which is about 0.04. In terms of comparison of machine learning classifiers, the Logistic Regression model achieved the best results, followed by the DT model, /> Bayes model and KNN model, while the SVM model not only performs very poorly in terms of calculation speed when facing large-scale category imbalance data, but also is unsatisfactory in terms of classification effect, with its Micro-F1 being only 0.13.

This invention calculates Micro-F1 and Macro-F1 of different neural networks on the prediction set based on the same deep learning training parameters and prediction parameters. The results are as shown in Table 2:

Table 2

The experimental data set is studied using the flat classifier strategy and the global classifier strategy and applying the neural network algorithm. Observing the experimental results, it can be seen that in terms of the flat classifier strategy, when the problem is transformed into a multivariate single-label problem, when facing large-scale category imbalance data, the effect of the neural network is generally significantly better than that of the traditional machine learning algorithm. . From the comparison of experimental results, it can be seen that the Micro-F1 of the least effective neural network classifier FastText on Flat Classification is almost the same as the best BOW+Logistic Regression, while most neural network classifiers have Micro-F1 on Flat Classification. It is about 0.05 higher than the optimal machine learning method.

In terms of comparing the performance of the neural network algorithm itself on different strategies, the effect of using the global classifier for Hierarchical Classification is significantly better than the Flat Classification of the flat classifier. As can be seen from Table 2, for each neural network method, the prediction set Micro-F1 predicted by Hierarchical Classification is generally 0.02-0.05 higher than the Micro-F1 predicted by Flat Classification. Obviously, the neural network in deep learning can perform better. Learn and exploit hierarchical dependency information to improve prediction performance.

Comparing the performance of different neural networks on Hierarchical Classification, FastText has a relatively simple structure and achieved a Micro-F1 of 0.4961, which is worse than the machine learning method, but not as good as other network structures with more complex structures and full utilization of information. The Micro-F1 value of TextCNN is 0.5580, and the Micro-F1 value of TextRNN is 0.5798. This shows that the CNN model and the RNN model can indeed achieve good results in the field of hierarchical text classification, and the RCNN model combines the CNN model and the RNN model to achieve better classification performance. Improved, Micro-F1 value reached 0.5824. The ARCNN model proposed by this invention, based on the advantages of the RNN model and the CNN model, has achieved the best classification performance by introducing the Attention mechanism. The Micro-F1 value is 0.5864, which is further improved on the basis of the RCNN model. At the same time Also better than all other models.

The same or similar numbers correspond to the same or similar parts;

The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limitations of this patent;

Obviously, the above-mentioned embodiments of the present invention are only examples to clearly illustrate the present invention, and are not intended to limit the implementation of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. An exhaustive list of all implementations is neither necessary nor possible. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims (5)

1. A large-scale category-level text classification method, which is characterized by including the following steps: S1: Collect government procurement announcement data; S2: Use machine learning to classify the data collected in step S1; S3: Use the classification result of step S2 as the benchmark, and use the ARCNN model to classify the data collected in step S1; In the step S3, Chinese text segmentation is performed on the collected government procurement announcement data to obtain the word sequence D = [w ₁ ; w ₂ ; ...; w _n ], let w represent the parameter set in the network structure, p(k |D, w) represents the probability that the text instance belongs to category k. The word sequence D = [w ₁ ; w ₂ ; ...; w _n ] is mapped to the word vector space through Word Embedding, and the vector representation of the word sequence is obtained: [e (w ₁ ); e(w ₂ );…; e(w _n )]; For the input word sequence, the bidirectional recurrent neural network model Bi-RNN is used to learn the contextual information of the current word w _i , and obtain the upper information c _l ( _wi ) and the lower information c _r (wi ₎ , where, e(w _i ) is a non-sparse vector with |e| real-valued elements, c _l (w _i ) and c _r (wi ₎ are both non-sparse vectors with |c| real-valued elements; In step S3, based on the structure of the bidirectional recurrent neural network model Bi-RNN, the context information c _l ( _wi ) and _cr ( _wi ) are hidden layer information, and their calculation method is as follows: c _l ( _wi )=f(W ^(l) c _l ( _wi-1 )+W ^(sl) e( _wi-1 )) c _r ( _wi )=f(W ^(r) c _r ( _wi+1 )+W ^(sr) e(wi ₊₁ )) Among them, c _l (wi _-1 ) comes from the hidden layer information at the above word w _i-1 , e(wi _-1 ) comes from the Word Embedding value of the above word w _i-1 , W ^{( l)} is the transfer matrix that transfers the hidden layer information backwards, W ^(sl) is the transfer matrix that transfers the input WordEmbedding value information backwards, c _r (wi ₊₁ ) comes from the following word w _i+1 Hidden layer information, e(w _i+1 ) comes from the WordEmbedding value of the word w _i+1 below, W ^(r) is the transfer matrix that transmits the hidden layer information forward, and W ^(sr) is the input Word The transfer matrix for forward transmission of Embedding value information, f is a nonlinear activation function, which activates the aggregated above hidden layer information and input information and then passes it backward to the current hidden layer. For the first word in the document, Since the above does not exist, the shared parameter c _l (w ₁ ) is used; Based on the bidirectional recurrent neural network model Bi-RNN, through forward and reverse cycles, for the current word w _i , the current input information e(wi ₎ and its hidden layer information c _l (wi ₎ , c _r can be obtained ( _wi ); cascade and splice the current input information and hidden layer information to obtain the text representation x _i : x _i =[c _l ( _wi ); e ( _wi ); c _r ( _wi )]; Now the contextual information is obtained at each word, but the degree of utilization is not exactly the same, and not all words contribute equally to the representation of text meaning. In order to allow the model to better focus on the key parts of the text, for Bi- The output x _i of the RNN module uses the self-attention model to extract more important words in the text, and obtain the text representation after attention marking: The text representation x _i obtained in the RNN module is encoded, and the query encoding Query: q _i , key encoding Key: k _i and value encoding Value: v _i are obtained through linear mapping f _q , f _k , and f _v respectively; q _i =f _q (x _i ) k _i = f _k ( _xi ) v _i = f _v (x _i ) Calculate the similarity between Query and Key to obtain the attention score as a measure of word importance and attention; then, use the softmax function to normalize the score to obtain the attention weight value: e _{i, j} = a (q _i , k _j ) a _{i, j} = softmax(e _{i, j} ) According to the obtained normalized weight, the value of the text is weighted and summed to obtain the text representation after attention marking. The current word w _i has obtained the text representation after attention transformation. It is further convolved and the tanh function is used as the activation function to obtain the latent semantic vector of w _i /> where W ⁽²⁾ is the weight matrix and b ⁽²⁾ is the bias vector: In the convolutional layer, let the kernel size of the filter=1. This is because It actually contains the context information of w _i . After obtaining the latent semantic vectors of all words, the maximum pooling layer is used to calculate, and we get/> Among them/> The kth element of is/> The maximum value of the kth element: After the pooling layer, text representations of various lengths are converted into fixed-length vectors, and the information in the entire text is captured. The maximum pooling layer is used. After the maximum pooling of the pooling layer, we obtain Entering the output layer, the probability p _i of the text instance belonging to the category is obtained through a fully connected layer and the softmax function as follows, where W ⁽⁴⁾ is the weight matrix and b ⁽⁴⁾ is the bias vector: In the forward propagation process, the parameters that need to be learned through model training include: among them, c _l (wi _-1 ) comes from the hidden layer information at the word w _i-1 above, and e(wi _-1 ) comes from the above The Word Embedding value of the text word w _i-1 , W ^(l) is the transfer matrix that transfers the hidden layer information backward, W ^(sl) is the transfer matrix that transfers the input Word Embedding value information backward, c _r (wi ₊₁ ) comes from the hidden layer information at the following word w _i+1 , e(w _i+1 ) comes from the WordEmbedding value of itself at the following word w _i+1 , W ^(r) is the hidden layer information The transfer matrix that is passed forward, W ^(sr) is the transfer matrix that passes the input WordEmbedding value information forward, W ⁽²⁾ is the weight matrix of the convolution layer, b ⁽²⁾ is the bias vector of the convolution layer, W ⁽⁴⁾ is the weight matrix of the fully connected layer, and b ⁽⁴⁾ is the bias vector of the fully connected layer: w＝{b ⁽²⁾ , b ⁽⁴⁾ , c _l (w ₁ ) , c _r (w _n ) , W ⁽²⁾ , W ⁽⁴⁾ , W ^(l) , W ^(r) , W ^(sl ), W ^(sr) }. 2. The large-scale category-level text classification method according to claim 1, characterized in that the specific process of step S1 is: Collect government procurement publicity data, including transaction announcements and bidding announcements, and preprocess the collected government procurement publicity data: (1) Remove illegal characters from text data and convert traditional characters into simplified characters; (2) Segment Chinese words and remove stop words; (3) Training word vectors. 3. The large-scale category hierarchical text classification method according to claim 2, characterized in that, in step S2, the plane classifier method of the hierarchical classification problem is used for classification, and in the feature engineering stage, BOW and TF-IDF are selected as word features. ; In the classifier training stage, choose the naive Bayes classifier, or Logistic regression classifier, or support vector machine classifier, or decision tree classifier to classify the experimental data; the plane classifier does not consider the hierarchical structure, so this section The loss function only considers the empirical loss function of the training set. 4. The large-scale category-level text classification method according to claim 1, characterized in that a category-level tree N, the nodes of the tree are labels of each category at all levels, and the method of mandatory leaf node prediction is selected for research. Input M text instances contained in the training set, denoted as where x _i ∈X is the instance space of text embedding, t _i ∈T represents the category,/> For the set of all leaf nodes in the hierarchical structure, define the function π: N→N, indicating that π(n) is the ancestor node of node n; let C _n represent the set of all child nodes of node n, define y _in = (2I( t _i =n)-1)∈{-1, 1} is a binary variable used to indicate whether the sample x _i belongs to the category n∈T; the prediction model is recorded as the prediction function f _p : X→T, and the target is for The given TRAIN obtains the minimum loss of the prediction function f _p . 5. The large-scale category-level text classification method according to claim 4, characterized in that the loss function used is composed of the empirical risk of the data set and a regularization term used to penalize the complexity of the prediction function _fp , defined as follows: Loss＝λ(w)+C×R _emp Among them, λ(w) is the regularization term, R _emp is the empirical risk of the data set, and C is the weight parameter that controls the balance between the empirical risk and the complexity of the control prediction function f _p ; For the parameter set w, in addition to the forward propagation method, it is decomposed into each node to form a parameter set based on the node composition: w = {w _n : n∈N}, that is, each node in the hierarchical structure n is associated with the parameter group w _n , where the function π: N→N means that π(n) is the ancestor node of node n. For the regularization term, the hierarchical dependence of the tree structure is introduced into the regularization term of w , thereby learning the hierarchical structure during the training process. The form of the regularization term is as follows: For the empirical loss function, the loss due to instances at leaf nodes in the category hierarchy: Among them, x _i is the instance space of text embedding, w _n is a parameter group based on nodes, y _in is used to indicate whether sample x _i belongs to category n, and L is all convex loss functions in different models.