CN112287666B - Corpus topic distribution calculation method based on meta information - Google Patents

Abstract

The invention belongs to the technical field of topic modeling, and particularly relates to a corpus topic distribution calculation method based on meta-information. The invention designs the TWLLDA topic model of the document and vocabulary meta-information, and overcomes the defects of complex model structure, non-conjugation, single information acquisition channel and the like in the prior art. The invention converts meta information into tag information of the document and the word, the tag information is independent of the model, so that the document with similar tags has similar dirichlet priori vectors, and the words with similar tags also have similar distribution weights on the topic; the invention provides an effective closed Gibbs sampling method to complete the reasoning of TWLLDA; and carrying out a plurality of groups of experiments by taking the confusion degree and the theme consistency as evaluation indexes. Experiments show that the TWLLDA model based on meta information performs better under the same conditions than the LDA model and the like.

Description

A Meta-information-Based Calculation Method of Topic Distribution in Corpus

technical field

The invention belongs to the technical field of topic modeling, and in particular relates to a method for computing topic distribution of a corpus based on meta-information.

Background technique

The traditional LDA topic model does not perform well on short texts, and its defect is that there is not enough rich lexical information to make its statistical significance valid. How to enrich document information and apply it to short text topic modeling is one of the hot research directions of scholars in recent years.

Although the short text topic modeling methods are different, the main idea is the same, that is, to enrich the sparse word co-occurrence information of short texts by various means. Its main implementation methods can be roughly divided into two categories, namely, a class of methods based on word embedding and tags, and a class of methods that expand the content of short texts to increase the amount of information.

Nowadays, most improved topic models are improved by introducing external features such as word vectors or adding short text content. The general idea of introducing word vector features is to make words with similar semantics more likely to be assigned to the same topic through word vectors. In this way, the accuracy of topic-word distribution is increased. Such methods only rely on word vectors to enrich word co-occurrence information, while the meta information of documents is not effectively used. Therefore, although the model accuracy has improved compared with the LDA model, there is still room for improvement.

Existing improved topic models often ignore the rich meta-information in the corpus. Meta information is the information that describes information. The author of the text, the time stamp, the feature vector of the document, etc. are not included in the text content, but the document feature information that can describe the text attributes can be called the meta information of the document. Knowledge such as word features can also be used as meta-information of words. In the topic model, the information of a vocabulary is the topic distribution of the vocabulary, then the vocabulary meta-information is the information describing the topic distribution of the vocabulary, and the meta-information is the information that is known before reading the document. The introduction of meta-information increases the channels of information acquisition and can effectively improve the accuracy of modeling.

Contents of the invention

The purpose of the present invention is to provide a corpus topic distribution calculation method based on meta-information.

The purpose of the present invention is achieved through the following technical solutions: comprising the following steps:

Step 1: Input the corpus to be calculated, obtain the document meta-information and lexical meta-information of the corpus, and set the maximum number of iterations;

Step 2: Convert the document meta-information and vocabulary meta-information of the corpus into document tags and vocabulary tags; generate a document-document tag vector matrix F _d,l according to the document tags; generate a vocabulary-vocabulary tag vector matrix according to the vocabulary tags

Step 3: Use the Gamma function with hyperparameter u ₀ as the parameter λ _l,k corresponding to document label l and topic k to assign values, λ _l,k ～Gamma(u ₀ ,u ₀ ), to obtain the topic-document label correlation matrix; use the Gamma function with hyperparameter v ₀ as the vocabulary label l ^* parameter corresponding to topic k assignment, /> Obtain the correlation matrix of topic-vocabulary label; Wherein, the total number of topics is K; The total number of document labels is L; The total number of vocabulary labels is L ^* ;

Step 4: Calculate the parameter β _k,v corresponding to topic k and vocabulary v, is the vocabulary-vocabulary label vector matrix /> element; calculate the document corpus d and topic k and the corresponding parameters α _d,k , /> f _d,l is the element of the document-document label vector matrix F _d,l ; calculate the number n _k, v of each word v assigned to the topic k, Calculate the number of words m _d,k assigned to topic k in each document corpus d,

Step 5: Sampling parameter q _d through q _d ~Beta(α _d, , m _d, );

Among them, α _d,· is the linear sum of each topic α _d,k value in a document, m _{d, ·} is the number of all words in a document; />

Step 6: Through the CRP process, take α _{d, k} as the aggregation, m _{d, k} as the number of customers sampling parameters t _{d, k} ;

Step 7: Sample parameters λ' _l,k from the Gamma random function, and update parameters α _d,k ;

λ' _l,k ～Gamma(μ′,μ″)

The update formula of parameters αd _,k is:

Step 8: According to Sampling parameter/>

Among them, β _k, is the sum of the correlation between topic k and each word; n _k, is the total number of words contained in topic k;

Step 9: Through the CRP process, take β _{k, v} as the aggregation, n _{k, v} as the number of customers sampling parameters t′ _{k, v} ;

Step 10: Sampling Parameters from Gamma Random Function And update the parameter β _k,v ;

The update formula of parameter β _{k, v} is:

Step 11: Determine whether the maximum number of iterations is reached; if the maximum number of iterations is not reached, return to step 5; otherwise, output the topic distribution of the corpus, that is, the formula:

The beneficial effects of the present invention are:

The present invention designs a TWLLDA topic model of document and vocabulary meta-information, which overcomes the shortcomings of complex model structure, non-conjugation, and single information acquisition channel in the prior art. The present invention converts meta-information into label information of documents and words, and the label information is independent of the model itself so that documents with similar labels have similar Dirichlet prior vectors, and words with similar labels have similar distribution weights on topics; the present invention proposes an effective closed Gibbs sampling method to complete the reasoning of TWLLDA; multiple groups of experiments are carried out with perplexity and topic consistency as evaluation indicators. Experiments show that compared with LDA and other models, the meta-information-based TWLLDA model performs better under the same conditions.

Description of drawings

Fig. 1 is an overall frame diagram of the present invention.

Fig. 2 is a TWLLDA model diagram in the present invention.

Fig. 3 is a pseudocode diagram of the conversion processing algorithm from meta information to tags in the present invention.

Fig. 4 is a structure diagram of document-document label vector matrix in the present invention.

Fig. 5 is a structure diagram of vocabulary-vocabulary label vector matrix in the present invention.

Fig. 6 is a structural diagram of a subject-document tag correlation matrix in the present invention.

Fig. 7 is a structure diagram of a topic-word tag correlation matrix in the present invention.

Fig. 8 is a pseudocode diagram of the model generation algorithm of TWLLDA in the present invention.

Fig. 9 is a pseudocode diagram of the collapsed Gibbs sampling algorithm of TWLLDA in the present invention.

Fig. 10 is the perplexity map of each model in the present invention on the Reuters data set.

Figure 11 is a comparison chart of the iteration time of each model on the 20NewsGroups dataset.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings.

The present invention relates to a corpus topic distribution calculation method based on meta-information, comprising the following steps:

Step 1: Input the corpus to be calculated, obtain the document meta-information and lexical meta-information of the corpus, and set the maximum number of iterations;

Step 2: Convert the document meta-information and vocabulary meta-information of the corpus into document tags and vocabulary tags; generate a document-document tag vector matrix F _d,l according to the document tags; generate a vocabulary-vocabulary tag vector matrix according to the vocabulary tags

Step 3: Use the Gamma function with hyperparameter u ₀ as the parameter λ _l,k corresponding to document label l and topic k to assign values, λ _l,k ～Gamma(u ₀ ,u ₀ ), to obtain the topic-document label correlation matrix; use the Gamma function with hyperparameter v ₀ as the vocabulary label l ^* parameter corresponding to topic k assignment, /> Obtain the correlation matrix of topic-vocabulary label; Wherein, the total number of topics is K; The total number of document labels is L; The total number of vocabulary labels is L ^* ;

Step 4: Calculate the parameter β _k,v corresponding to topic k and vocabulary v, is the vocabulary-vocabulary label vector matrix /> element; calculate the document corpus d and topic k and the corresponding parameters α _d,k , /> f _d,l is the element of the document-document label vector matrix F _d,l ; calculate the number n _k, v of each word v assigned to the topic k, Calculate the number of words m _d,k assigned to topic k in each document corpus d,

Step 5: Sampling parameter q _d through q _d ~Beta(α _d, , m _d, );

Among them, α _d,· is the linear sum of each topic α _d,k value in a document, m _{d, ·} is the number of all words in a document; />

Step 6: Through the CRP process, take α _{d, k} as the aggregation, m _{d, k} as the number of customers sampling parameters t _{d, k} ;

Step 7: Sample parameters λ' _l,k from the Gamma random function, and update parameters α _d,k ;

λ' _l,k ～Gamma(μ′,μ″)

The update formula of parameters αd _,k is:

Step 8: According to Sampling parameter/>

Among them, β _k, is the sum of the correlation between topic k and each word; n _k, is the total number of words contained in topic k;

Step 9: Through the CRP process, take β _{k, v} as the aggregation, n _{k, v} as the number of customers sampling parameters t′ _{k, v} ;

Step 10: Sampling Parameters from Gamma Random Function And update the parameter β _k,v ;

The update formula of parameter β _{k, v} is:

Step 11: Determine whether the maximum number of iterations is reached; if the maximum number of iterations is not reached, return to step 5; otherwise, output the topic distribution of the corpus, that is, the formula:

The present invention designs a TWLLDA topic model of document and vocabulary meta information, which overcomes the shortcoming of low accuracy of the topic model on short documents, and the model has lower perplexity and higher topic consistency under the same conditions.

As shown in Fig. 3, it is the conversion processing method from meta information to tags in the present invention. Transformation from meta information to tags: First, the document meta information can be processed by algorithm to obtain the document tag matrix with elements of (0/1), and the vocabulary meta information can be processed by algorithm to obtain the vocabulary tag matrix with elements of (0/1). This can not _only reduce the burden of data processing on the model, but also _highlight the expression of high feature dimensions in meta-information; secondly _, according to the generated document _label matrix _, and then generate _a document-document label vector matrix; The ma function is assignment, /> , to the topic-vocabulary label correlation matrix. Figure 4 shows the document-document label vector matrix, and Figure 5 shows the vocabulary-vocabulary label vector matrix, where d is a document in the dataset, v is the vocabulary in the dataset, l is the document label, and l* is the vocabulary label; the topic-document label correlation matrix, and the topic-vocabulary label correlation matrix are shown in Figures 6 and 7, where k is the topic; the final model generation process is shown in Figure 8. The collapsed Gibbs sampling process of TWLLDA is shown in Figure 9.

The mathematical derivation based on the TWLLDA model includes: sampling of random variables, sampling of Gamma random variables λ' _{l, k} , sampling of Gamma random variables to sample;

The joint distribution of the TWLLDA model is exactly the same as that of LDA, as shown in the formula:

in is the number of times word v is assigned to topic k. is the number of words assigned to topic k among all words in a document d. The parameters φ and θ can be calculated using the conjugate properties of Dirichlet polynomials. In turn, the joint probability of the model can be calculated, as shown in the formula:

where Beta _N (.) is the N-dimensional Beta function. The Dirichlet prior parameters represent the features of documents and vocabulary. Given the prior parameters α _d and β _k , the standard LDA model can start to run at this point, and Z _d,i can be directly calculated by performing Gibbs sampling on the topic assignment of the document vocabulary, as shown in the formula:

TWLLDA does not assume that a parameter represents the characteristics of documents and vocabulary, but calculates the Dirichlet prior parameters assumed in LDA through a linear model, and obtains an accurate prior parameter by describing the external characteristics of documents and vocabulary. These parameters can be generated directly from the Gamma random function.

The topic distribution θ of documents is a multinomial distribution, and λ _l,k is used to calculate the Dirichlet prior distribution α of θ. In order to sample the variable λ _l,k , the formula (1-2) is first extended by the Gamma function, as shown in the formula:

in That is, the linear sum of the alpha values of each topic in a document. parameter /> That is, the number of all words in a document. Γ(.) is the Gamma function. use the formula /> Replace α _k . In formula (1-4), Gamma ratio1 can be regarded as a series of beta random variables, so it can be enhanced by the Pitman-Yor process, as shown in the formula:

For each document d, there is q _d ~Beta(α _d,· ,m _d,· ), so given q _1:D for all documents, Gamma ratio1 can be expressed as Gamma ratio 2 in formula (1-4) can use auxiliary variable t _d,k data enhancement, as shown in the formula:

parameters in the formula Represents the first type of unmarked Stirling number, and Gamma ratio 2 is actually the probability normalization constant in CRP (Chinese Restaurant Process). Through the CRP process parameters t _{d, k} can be aggregated with α _{d, k} , and m _{d, k} is the number of customers for sampling, as shown in the formula:

Bern(.) in formula (1-7) is Bernoulli distribution. Parameters t _d,k sample 0/1 data through Bernoulli distribution. By ignoring entries that are not related to the parameter α, Equation 1-(6) can be simplified to Through the conversion of the above steps, formula (1-4) can be simplified to formula (1-8):

Through the Gamma function simplification method mentioned above, formula (1-8) can be transformed into formula (1-9):

As mentioned before, all document labels and vocabulary labels in the TWLLDA model are 0/1 labels, and only labels with a value of 1 in the label matrix are activated and added to the model calculation parameter α _d,k . Extract all the labels activated in the parameter λ _l,k in formula (1-9), and get the posterior probability of the parameter λ _l,k Wherein α _d,k /λ _d,k represents the value of α d,k obtained by not adding λ _{d,k to} the calculation when f _d,l =1.

Through the above data enhancement techniques, the TWLLDA joint probability distribution has been transformed into the conjugate prior distribution of the Gamma function λ' _{l, k} , so just like the Gibbs sampling in LDA, TWLLDA can directly sample the parameters λ' _{l, k} , such as formula (1-10), (1-11), (1-12):

λ' _l,k ～Gamma(μ′,μ″) (1-10)

Before the first calculation of the TWLLDA model, λ' _{l, k} is still an empty matrix. The calculation process is the standard LDA process. After the first calculation, the λ' _{l, k} matrix is filled. At this time, the parameters α _{d, k} can be updated through λ' _{l, k} , as shown in the formula:

In the formula (1-13), λ' _{l, k} are newly calculated values according to the formula (1-10). Repeating calculations from (1-10) to (1-13) means sampling the Gamma random variable λ' _l,k . In this iterative process, all documents are only involved in the calculation when the document label is activated.

Gamma random variable and λ' _{l, k} have a similar derivation process, and both use the same data enhancement technology and beta function conversion. In formula (1-2), the second beta function is transformed into formula (1-14):

Continue to simplify the formula (1-15) with the above method:

Compared with the corresponding relationship above, where Beta(.) is the beta distribution, β _{k, ·} is the topic prior of the multinomial distribution of each topic k on the vocabulary, that is, the sum of the correlation between topic k and each word; n _k is the number of vocabulary owned by each topic. /> Bern(.) is a Bernoulli distribution, and β _k,v is a correlation matrix. Using the vocabulary label activation matrix in the model, the calculation method of β _k,v is /> Extract each entry related to δ _l′,k and add the Gamma prior distribution function to get the posterior probability of δ _l′,k : />

Now the parameter δ _l′,k can be sampled from the Gamma random function, such as formula (1-16), (1-17), (1-18):

After the above three processes, the formula of parameter β _{k, v} is:

v' is the average number of words for which the label is activated. So far, the model parameters have been deduced.

Research on Gibbs sampling algorithm based on TWLLDA model:

Unlike most existing topic models, TWLLDA can utilize meta-information at two levels, document level and vocabulary level, and there is no need for one-to-one correspondence between topics and tags, and the number of tags can be set freely. In addition, TWLLDA does not destroy the conjugate prior distribution structure of the model, so parameter inference can be performed by a collapsed Gibbs sampling method similar to LDA.

The TWLLDA model aims at the shortcomings of short text topic modeling, and calculates the unique Dirichlet prior parameters for the document-topic distribution and topic-word distribution by converting the meta-information of the corpus into vocabulary tags and document tags. In doing so, while effectively merging meta information, it enriches the sparse word co-occurrence in short texts, making the perplexity and topic quality of TWLLDA model short text modeling significantly improved compared with other models. In addition, TWLLDA also has good modeling accuracy for regular text.

文档元信息和词汇元信息是通过doc2vec和word2vev算法获取的，根据算法1，将得到的词汇元信息(wV ₁ ,wV ₂ ,wV ₃ ,…wV _N )和文档元信息(dV ₁ ,dV ₂ ,dV ₃ ,…dV _D )作为输入得到输出文档标签(dL ₁ ,dL ₂ ,dL ₃ ,…dL _D )和词汇标签(wL ₁ ^* ,wL ₂ ^* ,wL ₃ ^* ,…wL _N ^* )，过程如下，首先根据文档元信息列表(dV ₁ ,dV ₂ ,dV ₃ ,…dV _D )，会计算每一篇的文档元信息获取正数并计算正数的平均值得到(avg+)也会获取负数并计算负数的平均值得到(avg-)，将计算得到的(avg+)和(avg-)与维度dVn进行比较大小，如果dVn≥(avg+)或者dVn≤(avg-)，向文档标签中添加数据1，否则添加0，进而生成元素为(0/1)的文档标签矩阵和元素为(0/1)的词汇标签矩阵，其中文档-文档标签向量就是根据算法1得到的F _d,l ，词汇-词汇标签向量也是根据算法1得到的G _w,l ^* ，其次是通过以超参数为v ₀的Gamma函数为 assignment, /> Then, by assigning values to λ _l,k with the Gamma function whose hyperparameter is u ₀ , λ _l,k ~Gamma(u ₀ ,u ₀ ) to get the topic-document label correlation matrix A _k,l and the topic-vocabulary label correlation matrix/> Finally, the corpus, document label matrix F _d,j , word label matrix 主题文档标签相关性矩阵A、主题-词标签相关性矩阵B、超参数μ ₀ 、超参数v ₀ 、主题个数K作为输出，得到在文档层面基于矩阵F和矩阵A为每篇文档d计算其特有的先验参数α _d作为输出，并得到在词汇层面基于矩阵G和矩阵B为每个主题k计算其特有的先验参数β _k作为输出；最后根据得到参数α _d和β _k运行TWLLDA模型，通过GibbsSampling采样算法达到收敛得到文档与主题的相关性的矩阵θ _d,k和主题与词汇的相关性的矩阵φ _k,v ，该模型的生成过程就可以概括为一个联合分布即公式/>

At the end of the model training, put the corpus to be tested into the trained model, and you will get the corresponding topic distribution, that is, the formula

The present invention proposes a TWLLDA topic model method for document and vocabulary meta-information. This model overcomes the shortcomings of the previously mentioned model, such as complex structure, non-conjugation, and single information acquisition channel, by converting meta-information into document and word label information. The label information is independent of the model itself so that documents with similar labels have similar Dirichlet prior vectors, and words with similar labels have similar distribution weights on topics; an effective closed Gibbs sampling method is proposed to complete the reasoning of TWLLDA; multiple groups of experiments are carried out with perplexity and topic consistency as evaluation indicators. Experiments show that compared with LDA and other models, the meta-information-based TWLLDA model performs better under the same conditions.

The present invention proposes a topic model (Text-Word Labeled Latent Dirichlet Allocation) based on document and vocabulary meta-information (hereinafter abbreviated as TWLLDA); the present invention proposes a topic model TWLLDA that converts the meta-information of the corpus into vocabulary tags and document tags, and calculates unique Dirichlet prior parameters for document-topic distribution and topic-word distribution. The model can both analyze tags with topics and assign tags to each topic by incorporating document meta information tags into the generation process of the model, which makes TWLLDA better interpretable. In the implementation, it is first necessary to transform document and vocabulary meta information into labels, generate corresponding document-document label vectors and vocabulary-vocabulary label vectors, and obtain the topic-document label correlation matrix and topic-vocabulary label correlation matrix by setting hyperparameters, that is, integrate meta information into the model.

Experimental verification and analysis:

The data sets used in the TWLLDA model performance comparison experiment are 20NewsGroups and some Reuters data sets. 20NewsGroups is one of the international standard datasets for text classification, mining and retrieval research, including 20 different types of news documents with a total of about 20,000 pieces. The terminology of the corpus is standardized, and the number of texts in each category is equal, and the topics of some categories are particularly similar, and some are completely unrelated. The Reuters dataset extracts labeled documents from Reuters-21578, including 11,367 documents with a vocabulary of 8,817 and an average document length of 73.

The data sets used to calculate the topic consistency are ten novels and the NLTK data set. The novels are Sophie's World, The Three-Body Problem, Lord of the Flies, To Kill a Mockingbird, Sun Tzu's Art of War, The Old Man and the Sea, The Great Gatsby, A Study in Scarlet, Women in Love, and One Hundred Years of Solitude. Each novel represents a different category, and the Chinese versions of these novels are all available for download on the Internet.

In the experiments of the present invention, doc2vec and word2vec are used to generate document vectors and word vectors as input to TWLLDA, respectively. The hyperparameters ₀ and ₀ of the TWLLDA model are both set to 1.0, and the hyperparameters and sum of the standard LDA model are both set to 0.5. Unrecognized words in the experiment do not participate in the calculation. Divide all novels into multiple documents according to chapters as a short text dataset. In the long text dataset and the short text dataset, 80% are used as the training set and 20% are used as the test set.

As the number of topics increases, the perplexity of each model decreases. But this decrease is not necessarily linear, such as LDA model, LF-LDA model and WF-LDA model. When the number of topics reaches a certain value for these three models, the corresponding perplexity does not decrease much, which means that the model has a certain limit for distinguishing topic perplexity. In addition, although LF-LDA uses word embeddings for improvement, the characteristics of its latent features lead to an increase in its perplexity. Although WF-LDA introduces this feature, the discrimination is still not enough to make the topic perplexity drop more obviously. Compared with LDA, the DMR model performs better in perplexity, but the experimental results of TWLLDA are still slightly better. The reason is that TWLLDA combines the meta-information of the text, and the ability to enrich the co-occurrence of document words is even better:

In this experiment, two mainstream indicators are used to evaluate the quality of a topic model, that is, perplexity evaluation and topic consistency. At the same time, the iteration time of the model is compared.

Perplexity is used to measure how well a probability model predicts a sample. is an important evaluation index in the field of topic models, and the smaller the value of perplexity, the higher the accuracy of the model. Its formula is shown in formula 1-20.

Perplexity＝exp(-lnp(wd/ad)/Nd) (1-20)

For the evaluation of perplexity, the TWLLDA model is compared with five models in the above two data sets. The comparison results are shown in the table below.

Table 1 Perplexity of each model in the 20NewsGroups dataset

It can be found that under the 20NewsGroups dataset, the perplexity performance of each topic model is above 1500. The reason is that the data set has many categories and different contents. Even if a higher number of topics is specified, the uncertainty of the topics cannot be reduced to an extremely low level.

As shown in Figure 10, it can be seen that Reuters has different perplexity performances for different topic numbers. However, no matter which condition the perplexity of the TWLLDA model is lower than that of the comparison model, it proves the subject extraction quality of the proposed model of the present invention.

Next is a comparison of run and iteration times. Running time is an important indicator of model evaluation, that is, the time complexity of the model. An iterative process of the standard LDA model is to completely reassign the topics for each vocabulary of each document. The assignment is based on the topic distribution of the document and the vocabulary distribution of each topic. Then in each Gibbs sampling, two matrices, the topic distribution matrix of the document and the vocabulary distribution matrix of the topic, must be maintained at the same time. Then the factors that affect an iteration are the size of the document and vocabulary, that is, the size of the data set, and the number of topics, a total of 2 factors. A TWLLDA iterative process is the same as the LDA process to complete a round of topic assignment process for all documents. Unlike LDA, the TWLLDA model not only needs to maintain the topic distribution matrix of the document and the vocabulary distribution matrix of the topic, but also needs to perform Gibbs sampling on the correlation matrix between document tags and topics and the correlation matrix between vocabulary tags and topics. In TWLLDA, two types of Gibbs sampling are performed at the same time, and a total of four matrices are maintained. Then the factors that affect one iteration of the TWLLDA model include the data set, the number of topics, the number of document labels, and the number of vocabulary labels.

Table 2 records the time required for each iteration of the experimental model.

Table 2 Model iteration time comparison

According to the analysis of experimental data, the single iteration time of the model generally increases linearly with the increase of the number of topics. Since LDA is the most basic model and does not introduce lexical features or labels, the running time is the shortest and the iteration is the fastest. However, the LF-LDA model increases the complexity of the model due to the introduction of word embedding, and the corresponding iteration time is particularly long.

As shown in Figure 11, the iteration time of the model on the 20NewsGroups dataset, obviously, the iteration time of the model on the 20NewsGroups dataset with more text will increase. Although the TWLLDA model increases the amount of data to be calculated compared with the standard LDA model, its iteration time is not particularly long due to the sparseness of external data. Therefore, it has a time advantage in terms of the DMR model and the WF-LDA model.

Next, the difference between the TWLLDA model and the LDA model will be evaluated from the subject consistency. Topic coherence computes the semantic coherence of vocabulary sets belonging to the same topic in a topic model. In the topic model, if the topic consistency is high, it means that the vocabulary within the topic has similar semantics, and the classification effect of the topic model is good. If the topic consistency is low, it indicates that the internal expression of the topic is fragmented, and the effect of topic classification is not good. The present invention uses Normalized Pointwise MutualInformation (NPMI) to measure the consistency of each topic in the topic model, and its formula is shown in formula (1-21).

Through this formula, the topic consistency score of topic k based on T high-frequency words in the topic can be calculated. P(w) is the probability of word w appearing, and p(w _i , w _j ) is the probability of word w _i , w _j appearing simultaneously in a sliding window. In the experiment of the present invention, the topic consistency score of each topic k is obtained by participating in the calculation of 10 high-frequency words of the topic. At the same time, this experiment divides Chinese novels into short texts and long texts according to chapters and lengths, sets the number of topics to 100, and analyzes the theme consistency between all themes and TOP20 themes. The experimental results are shown in the table below.

Under the condition of ALL TOPICS, its overall topic consistency is lower than that of TOP 20TOPICS. This is because the topics have an aggregation effect, that is, the top 20% topics occupy 80% of the lexical information, so the topic consistency of the front topic is higher, that is, the word co-occurrence information is more abundant, while the topic consistency of the latter topic is lower, that is, the word co-occurrence information is relatively scarce.

Analysis of corpus experiments:

Table 3 Comparison of topic consistency between TWLLDA model and other models

From the data in Table 3, it can be seen that the topic consistency of the TWLLDA model proposed by the present invention is higher than that of the standard LDA model as a whole, indicating that the data enhancement technology of TWLLDA can also play a role in optimizing the topic consistency.

When only the 20 topics with the largest vocabulary are involved in the calculation, the topic consistency is significantly improved, because the topic has more vocabulary, and the number of co-occurrence samples will also increase. Among them, the optimization on the medium-short dataset is 2 points higher than the optimization on the medium-long dataset on average, which is less than the 4 points calculated on the full topic, indicating that the TWLLDA model has a better auxiliary effect on topics with a small number of topic vocabulary.

When all topics are added to the topic consistency calculation, the TWLLDA model has played an optimized role in the three data sets of Chinese long documents, Chinese short documents, and NLTK. Compared with the LDA model, the TWLLDA model optimizes an average of 4 points on the medium and short data sets, and the average optimization point on the medium and long data sets is 3 points, which shows that the TWLLDA model has better performance on short documents than long documents. The topic consistency of short documents is significantly lower than that of long documents. The obvious reason is that the word co-occurrence information in short documents is insufficient, which is why data enhancement technology has a better space for short documents. When only the 20 topics with the largest number of vocabularies are involved in the calculation, the topic consistency is significantly improved. This is because the number of vocabularies of the topics is larger, and the samples of word co-occurrence will also increase. Among them, the optimization on the medium-short data set is 1 point higher than the optimization on the medium-long data set, which is less than the 4 points calculated on the full topic, indicating that the TWLLDA model has a better auxiliary effect on topics with a small number of topic vocabulary.

On the NLTK dataset, due to the sufficient data of NLTK, all models have high topic consistency, and as the previous analysis, the TWLLDA model has the highest topic consistency, and the LDA model has the lowest topic consistency. The benefit of the DMR model is higher, which will be illustrated later in the running time experiments. From the experiments in Table 4, it can be concluded that the data enhancement of the TWLLDA model has a better optimization effect on indicators sensitive to data volume, and has a better optimization effect than LDA, DMR, and WF-LDA.

The present invention proposes a topic model TWLLDA that converts meta-information of corpus into vocabulary tags and document tags, and calculates unique Dirichlet prior parameters for document-topic distribution and topic-word distribution. The model can both analyze tags with topics and assign tags to each topic by incorporating document meta information tags into the generation process of the model, which makes TWLLDA better interpretable. In the implementation, it is first necessary to transform document and vocabulary meta information into labels, generate corresponding document-document label vectors and vocabulary-vocabulary label vectors, and obtain the topic-document label correlation matrix and topic-vocabulary label correlation matrix by setting hyperparameters, that is, integrate meta information into the model.

The method of the present invention conducts multiple experiments with perplexity and topic consistency as evaluation indicators, and the experiments show that compared with models such as LDA, the meta-information-based TWLLDA model performs better under the same conditions.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (1)

1. The corpus topic distribution calculation method based on the meta information is characterized by comprising the following steps of:

step 1: inputting a corpus to be calculated, obtaining document meta information and vocabulary meta information of the corpus, and setting the maximum iteration times;

step 2: converting the document meta information and the vocabulary meta information of the corpus into document tags and vocabulary tags; generating a document-document tag vector matrix F from the document tags _d，l The method comprises the steps of carrying out a first treatment on the surface of the Generating vocabulary-vocabulary tag vector matrix according to vocabulary tag

Step 3: taking super parameter as u ₀ Gamma function of (1) is a parameter lambda corresponding to a theme k of a document tag l _l，k Assignment, lambda _l，k ～Gamma(u ₀ ，u ₀ ) Obtaining a correlation matrix of the topic-document label; let the super parameter be v ₀ Gamma function of (1) is vocabulary label l ^* Parameters corresponding to topic kAssignment of->Obtaining a correlation matrix of the topic-vocabulary labels; wherein the total number of topics is K; the total number of the document labels is L; the total number of vocabulary labels is L ^* ；

Step 4: calculating parameter beta corresponding to topic k and vocabulary v _k，v ， For vocabulary-vocabulary tag vector matrix->Is an element of (2); calculating document corpus d, topic k and corresponding parameters alpha _d，k ，/>f _d，l For document-document tag vector matrix F _d，l Is an element of (2); counting the number of times n each word v is assigned to the topic k _k，v ，Calculating the vocabulary quantity m allocated to the topic k in each document corpus d _d，k ，

Step 5: through q _d ～Beta(α _d，· ，m _d，· ) Sampling parameter q _d ；

Wherein alpha is _d，· For each topic α in a document _d，k The linear sum of the values is used to determine,m _d，· is the number of words in a document; />

Step 6: by CRP process, at alpha _d，k To gather, m _d，k Sampling the parameter t for the number of customers _d，k ；

Step 7: sampling the parameter lambda 'from the Gamma random function' _l，k And update the parameter alpha _d，k ；

λ′ _l，k ～Gamma(μ′，μ″)

Parameter alpha _d，k The updated formula of (2) is:

step 8: according toSampling parameter->

Wherein beta is _k，· The sum of the relevance of the topic k to each word; n is n _k，· The total number of words contained for topic k;

step 9: by CRP process, at beta _k，v To gather, n _k，v Sampling the parameter t 'for the number of customers' _k，v ；

Step 10: sampling parameters from Gamma random functionAnd update the parameter beta _k，v ；

Parameter beta _k，v The updated formula of (2) is:

step 11: judging whether the maximum iteration times are reached; if the maximum iteration number is not reached, returning to the step 5; otherwise, outputting topic distribution of the corpus, namely the formula: