CN107145476A - One kind is based on improvement TF IDF keyword extraction algorithms - Google Patents

Abstract

The invention discloses a keyword extraction algorithm based on improved TF-IDF, which includes the following steps: S1: uniformly format the input form of the text; S2: load the configuration file to Stanford NLP; S3: obtain the text in the configuration file All sentence collection Sentences; S4: Take a sentence from Sentences each time; S5: Get all the word collection Tokens in the current sentence; S6: Take a token from Tokens each time; S7: Get the word/phrase and part of speech of the current token , and give different part-of-speech weights; S8: Calculate the total number of words/words in the current sentence and their position percentage; S9: Get all words/words in the text set Words; S10: Take a word from Words each time; S11: Calculate the current word TF and IDF; S12: Calculate the weight W of all words, and select keywords according to the weight W of words. The invention increases the part-of-speech factor, improves the extraction accuracy, and solves the problem of complex spaces such as constructing Pat-trees.

Description

A Keyword Extraction Algorithm Based on Improved TF-IDF

technical field

The invention relates to an improved TF-IDF keyword extraction algorithm.

Background technique

At present, in the related methods based on the improved TF-IDF keyword extraction algorithm, the main disadvantage is that simply measuring the importance of a word by "word frequency" is not comprehensive enough, and sometimes important words may not appear many times. Moreover, this algorithm cannot reflect the part-of-speech information of words. Words whose attributes are nouns and words whose attributes are particles are given the same importance, which is obviously unreasonable. In other related technologies, the accuracy of keyword extraction is relatively improved, but the complexity of the space is also improved, which cannot meet the actual requirements.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art and provide a keyword extraction algorithm based on improved TF-IDF.

The technical scheme adopted in the present invention is:

A keyword extraction algorithm based on improved TF-IDF, which comprises the following steps:

S1: Uniformly format the input form of the text;

S2: Format the text standard, and load the Properties configuration file for Stanford NLP;

S3: Obtain the set Sentences of all sentences in the text according to the defined sentence separator in the Properties configuration file;

S4: Each time a sentence is taken out from the collection Sentences;

S5: Obtain all the word set Tokens in the current sentence;

S6: Take out a token from the collection Tokens each time;

S7: Obtain the word/phrase and part of speech of the current token, and assign different part of speech weights to different parts of speech;

S8: Calculate the total number of words/words and their position percentages in the current sentence;

S9: In the previous operation, all the words/word sets Words in the text are obtained;

S10: take out a word from the set Words each time;

S11: Calculate the TF and IDF of the current word;

S12: After obtaining the part-of-speech weight, position weight, TF, and IDF of all words, calculate the weight W of the word=TF*IDF+part-of-speech weight+position weight, and select the top 5 weight W weights of words from large to small word is output as a keyword.

Further, the parameters included in the unified formatting in the step S1 are title, label, abstract and text respectively; if the corresponding parameters do not exist, enter "".

In said step S2, when the annotators included in the pipeline are first set in the text standard formatting, respectively select word segmentation, separation, part-of-speech tagging, and 4 annotators for identifying named entities, these four annotators are necessary for the text processing of the present invention; secondly , load the packages required by each annotator and set the corresponding parameters.

Further, when calculating the total number of characters/phrases in the current sentence in the step S8, the words of parts of speech that cannot become keywords are removed.

Further, in the step S11, the traditional algorithm of IDF mainly compares the number of documents containing the word with the total number of documents containing the word, and the formula is as follows:

Where |D| indicates the total number of documents, and |{j:t _i ∈ d _j }| indicates the total number of documents containing the word ti.

The size of the IDF is based on the collected 10,000 news articles as a test data set, and the IDF of each word contained therein is calculated and stored in a configuration file (the IDF of about 270,000 common words is included). When you need to use it, you can directly read the IDF value of the corresponding word from the configuration file. There is no need to count the number of documents in which the word appears in all documents. The calculation amount is relatively small and the operation speed is fast. For new or uncommon words not stored in the configuration file, the mean of all IDFs is used as the IDF of the word

Further, the length of each word of the first 5 words used as keywords in the step S12 is between 2-6.

The present invention adopts the above technical scheme and adds part-of-speech factors in the process of extracting keywords, which can effectively solve the problems of large space complexity caused by constructing Pat-trees in related solutions while improving the accuracy of keyword extraction.

Description of drawings

The present invention will be described in further detail below in conjunction with accompanying drawing and specific embodiment;

Fig. 1 is a schematic flow chart of an improved TF-IDF keyword extraction algorithm in the present invention.

detailed description

As shown in Figure 1, the present invention discloses a keyword extraction algorithm based on improved TF-IDF, which includes the following steps:

S1: Uniformly format the input form of the text;

S2: Format the text standard, and load the Properties configuration file for Stanford NLP;

S3: Obtain the set Sentences of all sentences in the text according to the defined sentence separator in the Properties configuration file;

S4: Each time a sentence is taken out from the collection Sentences;

S5: Obtain all the word set Tokens in the current sentence;

S6: Take out a token from the collection Tokens each time;

S7: Obtain the word/phrase and part of speech of the current token, and assign different part of speech weights to different parts of speech;

S8: Calculate the total number of words/words and their position percentages in the current sentence;

S9: In the previous operation, all the words/word sets Words in the text are obtained;

S10: take out a word from the set Words each time;

S11: Calculate the TF and IDF of the current word;

S12: After obtaining the part-of-speech weight, position weight, TF, and IDF of all words, calculate the weight W of the word = TF*IDF+ part-of-speech weight + position weight, and select the top 5 weight W weights of words from large to small Word is used as a keyword, and the length of each word is output between 2-6.

Further, the parameters included in the unified formatting in the step S1 are title, label, abstract and text respectively; if the corresponding parameters do not exist, enter "".

In said step S2, when the annotators included in the pipeline are first set in the text standard formatting, respectively select word segmentation, separation, part-of-speech tagging, and 4 annotators for identifying named entities, these four annotators are necessary for the text processing of the present invention; secondly , load the packages required by each annotator and set the corresponding parameters.

Further, stop words are removed when calculating the total number of characters/phrases in the current sentence in the step S8. Not only stop words are filtered on the text, but also parts of speech that cannot become keywords, such as function words, etc. are filtered.

Further, in the step S11, the traditional algorithm of IDF mainly compares the number of documents containing the word with the total number of documents containing the word, and the formula is as follows:

Where |D| indicates the total number of documents, and |{j:t _i ∈ d _j }| indicates the total number of documents containing the word ti. If the term is not in the document, the denominator will be zero. In order to avoid this situation, generally 1+|{j:t _i ∈ d _j }| is used as the denominator. But the size of the IDF discussed in the present invention is the test data set with 10,000 pieces of news collected, calculate the IDF of each word contained therein and store it in the configuration file (comprising the IDF of about 270,000 common words in total) . When you need to use it, you can directly read the IDF value of the corresponding word from the configuration file. There is no need to count the number of documents in which the word appears in all documents. The calculation amount is relatively small and the operation speed is fast. For new or uncommon words not stored in the configuration file, the mean of all IDFs is used as the IDF of the word.

The present invention adopts the above technical scheme and adds part-of-speech factors in the process of extracting keywords, which can effectively solve the problems of large space complexity caused by constructing Pat-trees in related solutions while improving the accuracy of keyword extraction.

Claims (6)

1. A keyword extraction algorithm based on improved TF-IDF, characterized in that: it comprises the following steps: S1: Uniformly format the input form of the text; S2: Format the text standard, and load the Properties configuration file for Stanford NLP; S3: Obtain the set Sentences of all sentences in the text according to the defined sentence separator in the Properties configuration file; S4: Each time a sentence is taken out from the collection Sentences; S5: Obtain all the word set Tokens in the current sentence; S6: Take out a token from the collection Tokens each time; S7: Obtain the word/phrase and part of speech of the current token, and assign different part of speech weights to different parts of speech; S8: Calculate the total number of words/words and their position percentages in the current sentence; S9: In the previous operation, all the words/word sets Words in the text are obtained; S10: take out a word from the set Words each time; S11: Calculate the TF and IDF of the current word; S12: After obtaining the part-of-speech weight, position weight, TF, and IDF of all words, calculate the word weight W=TF*IDF+part-of-speech weight+position weight, and select the top 5 weights W of words from large to small word is output as a keyword. 2. a kind of based on improved TF-IDF keyword extracting algorithm according to claim 1, it is characterized in that: in the described step S1, the parameter that unified formatting comprises is title, label, abstract and main text respectively; If corresponding If the parameter does not exist, enter "". 3. a kind of based on improved TF-IDF keyword extraction algorithm according to claim 1, it is characterized in that: in described step S2, text standard formatting selects word segmentation, separation, 4 annotators for part-of-speech tagging and identification of named entities, these four annotators are necessary for the text processing of the present invention; secondly, load the packages required by each annotator and set the corresponding parameters. 4. a kind of based on improved TF-IDF keyword extracting algorithm according to claim 1, it is characterized in that: remove the words that can not become the part of speech of keyword when calculating the total number of words/words in the current sentence in the described step S8. 5. A kind of keyword extraction algorithm based on improved TF-IDF according to claim 1, characterized in that: the size of the IDF is based on the 10,000 pieces of news that are collected as a test data set, and each included in it is calculated The IDF of the word is stored in the configuration file, which contains a total of 270,000 IDFs of common words; when needed, just read the IDF value of the corresponding word directly from the configuration file, without counting the number of documents in which the word appears in all documents, For new or uncommon words not stored in the configuration file, the mean of all IDFs is used as the IDF of the word. 6. A kind of keyword extraction algorithm based on improved TF-IDF according to claim 1, characterized in that: the length of each word of the first 5 words as keywords in the step S12 is between 2-6 .