CN117010366A - Text specific sentence-oriented content identification and correction method - Google Patents

Abstract

The application discloses a text specific sentence-oriented content recognition and correction method, and belongs to the field of natural language processing. The method combines a plurality of technologies, including methods of regular matching, hfl/rbt3 model up-and-down sentence prediction, position sensitive hash technology, jaccard similarity calculation and the like, and aims at specific contexts of specific sentence recognition and error correction of important leading speech and the like, and can accurately recognize golden sentence parts of important leading speech and the like in news articles by integrating various different methods; by improving the minhash algorithm, it is made more efficient at processing specific text and matches are computed using Jaccard similarity for short text therein, so that it also has higher accuracy in the error recognition part. The method can effectively extract and verify specific sentences such as the pilot important speaking content and plays an important role in improving efficiency and accuracy for news release and the like.

Description

A content recognition and error correction method for specific sentences in text

Technical field

The present invention relates to the technical field of natural language processing, and in particular to a content identification and error correction method for specific sentences in text.

Background technique

The scenario of identifying text content in an article is similar to the scenario of document citation recognition, and is aimed at the task of invisible document citations. Literature citation identification is the process of identifying and extracting citations from academic texts. At present, the more common method is to use deep neural networks and attention mechanisms to extract reference text from the citation context. The task can be regarded as a classification or sequence labeling task. For example, for the idea of sequence labeling, the model is trained to predict the attributes of words in the text. Thus, it is judged that the sentences composed of those words are quotation sentences. Of course, in the early days, it was also implemented based on the idea of sliding windows, that is, selecting an appropriate window size to select the citation segment, and then using text similarity matching to determine the invisible citation part.

In the existing technology, hfl/rbt3 is a Chinese pre-trained language model based on RoBERTa. This is a small model with 3 layers and 12 attention heads. It is released by Harbin Institute of Technology (HIT) as part of its Chinese-BERT-wwm project. hfl/rbt3 can be used for various natural language processing tasks, such as text classification, named entity recognition, sentiment analysis, etc. It can be easily loaded and used with Transformers or PaddleHub libraries.

Location-Sensitive Hashing (LSH) is a fast approximate nearest neighbor search technology for massive high-dimensional data, which is highly efficient for massive data. The basic idea is to hash the input high-dimensional feature vector into a low-dimensional feature space through a hash function. This hashing method can make the points that are close in the original space still close in the low-dimensional space after hashing. That is to say, the probability of collision between points that are closer to each other after being hashed by the hash function is greater than the probability of collision between points that are farther away. Position-sensitive hashing can effectively reduce the complexity of data storage and retrieval, and is suitable for various types of data such as text, images, and audio.

For the specific operation of position-sensitive hashing, you first need to define a series of hash functions to map high-dimensional input vectors to low-dimensional output values; the hash function should satisfy the property that similar input vectors are mapped to the same output The probability of a value is high, while the probability that different input vectors map to the same output value is low. Secondly, the obtained hash value is divided into several parts, which are respectively recorded as keys, and the key and the original input vector are stored in the hash table, where the key is the key and the value is the original input vector. After that, to find the nearest neighbor of the query vector, apply the same hash function to it and find the corresponding output value in the hash table, retrieve all input vectors that share at least one key with the query vector and calculate their relationship with the query vector distance. Returns the closest vector as an approximate neighbor.

Minhash is a technique for quickly estimating the similarity between two sets. It was invented by Andrei Broder and was originally used in the AltaVista search engine to detect duplicate web pages and remove duplicate parts. It has also been applied to large-scale clustering problems, such as clustering documents by the similarity of word sets. There is actually a certain relationship between the minimum hash value and the Jaccard similarity: the Jaccard similarity of two sets is equal to the probability value of their random permutation and conversion to obtain the same minimum hash value. Therefore, the minimum hash is often used to estimate the Jaccard similarity. The MinHash algorithm gives us a fast approximation of the Jaccard similarity between two sets.

The above technologies have different applications for different scenarios. However, there is no comprehensive method for specifically identifying specific sentences (such as the content of important speeches by leaders, poems in articles, famous sayings of important figures, etc.). For specific application scenarios, the application of the algorithm needs to be appropriately improved according to the specific scenario. In specific scenarios, deep learning may not necessarily be chosen. Simple rule recognition is sometimes more accurate and efficient than deep learning. The present invention uses a rule-based method to realize recognition in the scene of specific sentences in the article.

Contents of the invention

Aiming at the recognition scenario of specific sentences in the article, the present invention comprehensively uses existing methods to construct a text-oriented recognition and error correction method. The method is roughly divided into two parts, namely the recognition module and the matching module. The recognition module is mainly based on regular rules. Matching and Roberta models, the matching module is mainly based on position-sensitive hashing and Jaccard similarity, and the recognition results are accurate and reliable.

In order to achieve the above objects, the present invention provides the following technical solutions:

The present invention provides a content identification and error correction method for specific sentences of text, which includes the following steps:

S1. Apply the regular matching rules to initially identify the golden sentence part. The recognition situation is mainly divided into two categories. One category is included in double quotes. This category can be correctly matched after being regularized and enters step S3; the other category is not included in double quotes. , enter step S2 for further processing;

S2. Use the trained hfl/rbt3 model to predict the upper and lower sentences, determine the golden sentence part, and enter step S3;

S3. Collect the golden sentence database and segment the identified golden sentence parts into sentences. The matching method is divided according to the sentence length. When the sentence length is greater than or equal to the set threshold, the minhash matching method is used to calculate the sentence minhash. value, store the hash value segments and sentences corresponding to the sentences in the database into the hash table, use inverted index matching to find the most similar sentences, and enter step S4; when the sentence length is less than the set threshold, use Jaccard similarity matching method, calculate the Jaccard value of the target sentence and other sentences in the database, find the sentence with the largest Jaccard value, and enter step S4;

S4. Start from the first sentence as the target sentence and match it in the database. If it matches, compare the subsequent parts of the sentence and the recognized golden sentence part in the database one by one to see if there are any errors, and output the recognition result and the error part.

Furthermore, the hfl/rbt3 model training in step S2 uses positive and negative samples of the upper and lower sentences. The positive samples come from the upper and lower sentences in the golden sentence database, and the negative samples are collected manually. The negative samples are data enhanced and then input into the pre-training model. hfl/rbt3 is trained, and data enhancement methods include but are not limited to random deletion, random replacement, and random addition.

Further, the specific steps for further judgment using the trained model in step S2 are as follows:

S201. First, obtain a paragraph without quotation marks, and divide the paragraph into sentences according to periods, exclamation marks, question marks, and ellipses;

S202. Then, consider the first sentence as the golden sentence, then this sentence will be used as the previous sentence, and the next sentence will be used as the next sentence. After being processed by the tokenizer of the hfl/rbt3 model, it will be input into the hfl/rbt3 model. Determine the return value;

S203. Initialize the result. The first sentence is a golden sentence, and then input it into the model. If the return value is 1, it means that the next sentence is also a golden sentence. Add this sentence to the result sentence, repeat step S202, and use this sentence as the previous sentence. Then it is judged whether the following sentence is the next sentence; if the return value is 0, it indicates that the next sentence is not a golden sentence, then it exits and does not continue the judgment. It ends when all sentences are traversed, and the result sentence is obtained, which is the golden sentence part.

Further, the specific steps of performing minimum hash operation on sentences in step S3 are:

S301. First, perform 3-shingle processing on the sentence to obtain a shingles list;

S302. Define the number of random permutation transformations as perm_num, and then randomly produce the a and b coefficients of perm_num affine transformation functions, that is:

h(x)=(ax+b)%m

Where m is the Mersenne prime number, which takes 2 ⁶¹ -1, a and b are random integers less than the maximum value, and the maximum value takes 2 ³² -1;

S303. For each hash function, perform a hash operation on each shingle, and find the minimum value as the minimum hash value of this hash function. Perm_num minimum hash functions constitute the minimum hash value of the final original text.

Further, the specific steps for storing the database sentences in step S3 are:

(1) Define the number of random permutation transformations as perm_num, divide all random permutation into b segments, each segment has r hashes, assuming that the real jaccard similarity between two texts is js, then between the two texts The probability that a corresponding part of the perm_num minimum hashes is equal is equal to js. It can be deduced that the probability that all hashes of the minimum hash signatures of two texts in a certain segment are equal, that is, the probability of becoming a candidate pair is:

1-(1-s ^r ) ^b

The relationship curve between the document Jaccard similarity and the probability of becoming a candidate pair is an S-shaped curve. The similarity of the curve is the threshold at 1/2 of the candidate probability, that is, the Jaccard similarity of 0.5. At this time, the approximate relationship between b, r and the threshold is estimated as follows:

(1/b) ^1/r

Find the optimal values of b and r according to the provided threshold, perm_num, false positive example weight and false negative example weight, where false positive examples are dissimilar text pairs hashed to the same bucket, and false negative examples are For similar text pairs that are not hash-mapped into a bucket, set the false positive example weight and false negative example weight to 0.5 respectively;

(2) Calculate the minhash value of the sentences in the database respectively, and then divide the obtained perm_num hashes into b blocks. Each block has r minimum hashes. The content of the divided b blocks is used as the b segment hash value (key) of the sentence. );

(3) Record the hash value (key) of segment b into hash tables (HashTables), where each hash value (key) is an index, that is, a key, and the index value is the corresponding sentence representation, and a dictionary is created at the same time. The key is a sentence representation, denoted as mi, i is an integer starting from 0, and the value is the sentence itself.

Further, the specific steps of inverted index matching in step S3 are:

For the target sentence, first determine the length of the sentence. If it is greater than a fixed value, do minhash and get its corresponding hash value (key). Index all hash values in the hash table (HashTables) and find a part of the hash. Candidate sets with the same value, and then find the instance in the candidate set that is closest to the Hamming distance of the target sentence. If it is within the tolerance range, it will be output, otherwise it will be marked as unrecognized.

Compared with the prior art, the beneficial effects of the present invention are:

1. This invention identifies specific sentences such as important speeches by leaders based on regular matching, without directly using the deep learning model for citation identification. Relatively speaking, the identification results are more accurate, and there is no need to collect a large amount of data, while reducing the cost of manual annotation.

2. The present invention further identifies the golden sentences initially identified based on the Roberta model, and considers whether the parts directly quoted without quotation marks all belong to golden sentences. This method makes the recognition of golden sentences more accurate.

3. The present invention performs matching and error recognition of golden sentences based on position-sensitive hash minhash and Jaccard similarity calculation. The position-sensitive hashing method minhash maps high-dimensional feature vectors into low-dimensional feature vectors, stores long sentences in hash form, can be calculated and stored offline, and can use inverted indexing for similarity matching. For massive It is very efficient for long texts; for short sentences, it is more accurate to use the Jaccard similarity calculation method. The combination of the two balances efficiency and accuracy.

4. The present invention uses 3-shingle to process the target sentence instead of using jieba word segmentation, which has higher accuracy and processing speed. Due to the particularity of golden sentences such as important speech sentences by leaders, it is generally not allowed to make a single word error. However, jieba word segmentation calls a third-party library which takes longer, and minhash calculation after removing stop words and punctuation will increase the error in the sentence and There is a gap in the original sentence, so it is more effective to use a simple 3-shingle.

In summary, the present invention proposes a content recognition and error correction method for specific text sentences, which combines a number of technologies including regular matching, hfl/rbt3 model upper and lower sentence prediction, position-sensitive hashing technology minhash and Jaccard similarity calculation method , can more accurately identify specific sentences such as important speeches by leaders in news articles, and by improving the minhash algorithm, it is more effective in processing specific texts, making it also have higher accuracy in error recognition, for news releases, etc. play an important role in important national areas. This method can also be extended to the identification of poems in articles or the identification of famous quotes from other important figures.

Description of the drawings

In order to more clearly explain the embodiments of the present application or the technical solutions in the prior art, the drawings required to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

Figure 1 is a flowchart of a method for content identification and error correction for specific sentences in text according to an embodiment of the present invention.

Figure 2 is a flow chart of training and inference of the sentence prediction model provided by the embodiment of the present invention.

Figure 3 is a flow chart of the minhash operation provided by the embodiment of the present invention.

Figure 4 is a storage principle diagram of the minhash hash table provided by the embodiment of the present invention.

Figure 5 is a diagram showing the complete input and output results provided by the embodiment of the present invention.

Detailed ways

In order to better understand the technical solution, the method of the present invention will be described in detail below with reference to the accompanying drawings.

The following takes an important speech by a leader as an example to introduce a content identification and error correction method for text-specific sentences of the present invention, as shown in Figure 1, which includes the following steps:

Step 1: Regular matching identification

Apply regular matching rules to initially identify the leader’s important speeches. The leader’s important speeches are also recorded here as golden sentences. The identification situations are mainly divided into two categories. One category is included in double quotes, which can be matched correctly after regularization, and no further processing is required; the other category is not included in double quotes and requires further processing.

Step 2: Training and inference of the upper and lower sentence prediction model

Figure 2 shows the steps for fine-tuning training and inference of the pre-trained model hfl/rbt3. First, we need to collect positive and negative samples of the upper and lower sentences of the leader's important speech. The positive samples come from the upper and lower sentences in the golden sentence database, while the negative samples are collected manually. Therefore, there will be an imbalance in the positive and negative sample data, so the negative samples are collected. Enhancement, then perform word analysis on the data, and then input the pre-trained model hfl/rbt3 for model training.

The specific steps to use the trained model for inference are as follows:

S1. First get a paragraph without quotation marks, and divide the paragraph into sentences according to periods, exclamation marks, question marks, and ellipses;

S2. Then, consider the first sentence as a golden sentence, then this sentence will be used as the previous sentence, and the next sentence will be used as the next sentence. After being processed by the tokenizer of the hfl/rbt3 model, it will be input into the hfl/rbt3 model. Determine the return value;

S3. If the return value is 1, it indicates that the next sentence is also a golden sentence. Add it to the result sentence, repeat step S2, use this sentence as the previous sentence, and then determine whether the following sentence is the next sentence; if the return value is 0, it indicates If the next sentence is not a golden sentence, it will exit without further judgment. It will end when all sentences have been traversed, and the result sentence will be obtained, which is the golden sentence part.

Step 3: minhash operation

Figure 3 shows the general steps of minhash. The specific steps of minhash operation on sentences are described as follows:

S1. First of all, it is to process the text 3-shingle. Assume that the original sentence is: "We must continue to push forward the great historical cause of the great rejuvenation of the Chinese nation", then the result is: ['We want', 'We want' ,'should continue','continue to control','end to control','to control China','Chinese people','Chinese nation','national greatness','national greatness','great restoration','great rejuvenation' ', 'revival', 'history', 'history', 'historical greatness', 'historical great cause', 'great cause push', 'deed push', 'push forward', 'forward'] ;

S2. Define the number of random permutation transformations as perm_num, and then randomly produce the a and b coefficients of perm_num affine transformation functions, that is:

h(x)=(ax+b)%m

Where m is the Mersenne prime number, here it is 2 ⁶¹ -1, a and b are random integers less than the maximum value, the maximum value is 2 ³² -1.

S3. For each hash function, perform a hash operation on each shingle, and find the minimum value as the minimum hash value of this hash function. Perm_num minimum hash functions constitute the minimum hash value of the final original text.

Step 4: minhash index storage

Store each sentence in the collected leader's important speech database in a hash table to facilitate subsequent quick matching of similar sentences. As shown in Figure 4, the specific steps for storing sentences in the database are:

S1. Consider dividing perm_num into b segments, each segment has r hashes. Assume that the real jaccard similarity between the two texts is js, then a corresponding part of the perm_num minimum hashes between the two texts is equal. The probability is equal to js. It can be deduced that the probability that all hashes of the minimum hash signatures of two texts in a certain segment are equal is:

1-(1-s ^r ) ^b

That is to say, the probability of becoming a candidate pair. The relationship curve between the document Jaccard similarity and the probability of becoming a candidate pair is an S-shaped curve, which is basically not affected by b and r. The similarity of the curve is the threshold at 1/2 of the candidate probability, which is the Jaccard similarity of 0.5. At this time, it is approximate The estimated relationship between b, r and threshold is as follows:

(1/b) ^1/r

Find the optimal values of b and r according to the provided threshold, perm_num, false positive example weight and false negative example weight, where false positive examples are dissimilar text pairs hashed to the same bucket, and false negative examples are For similar text pairs that are not hash-mapped into a bucket, the weight of false positive examples and the weight of false negative examples are generally set to 0.5 respectively, that is, we hope that both situations will occur as little as possible.

S2. Find the minhash value of the sentences in the database respectively, and then divide the obtained perm_num hashes into b blocks. Each block has r minimum hashes. The content of the divided b blocks is used as the b segment hash value (key) of the sentence. .

S3. Record the hash value (key) of segment b into hash tables (HashTables), where each hash value is an index, that is, a key, and the index value is the corresponding sentence representation, such as "m1". At the same time, create A dictionary whose key is the sentence representation "mi", i is an integer starting from 0, and the value is the sentence itself.

Step 5: Misidentification of the recognized golden sentences

For the golden sentence part recognized by the input, segment the golden sentence part. Starting from the first sentence, it is used as the target sentence to match in the database. If there is a match, compare the beginning and subsequent parts of the sentence in the database with the identified golden sentence part one by one to see if there are any errors.

For the matching of the target sentence, first judge the length of the sentence. If it is greater than a fixed value, do minhash and get its corresponding hash value (key). Index all hash values in the hash table (HashTables) and find a part. Candidate sets with the same hash value, and then find the instance in the candidate set that is closest to the Hamming distance of the target sentence. If it is within the tolerance range, output it, otherwise mark it as unrecognized; if the sentence length is less than a fixed value, calculate the distance between the target sentence and the database Among the Jaccard similarities of other sentences, find the largest one. If the similarity value is within a certain range, it will be output, otherwise it will be marked as unrecognized.

Figure 5 shows the complete input and output results. After inputting the article, the above-mentioned golden sentence recognition and similarity matching are performed to obtain the original sentence that is similar to the target sentence, and the original sentence and the recognized sentence are output.

In summary, the content identification and error correction method for specific sentences in text proposed by this invention combines a number of technologies including regular matching, hfl/rbt3 model upper and lower sentence prediction, position-sensitive hashing technology minhash and Jaccard similarity calculation method, which can It can more accurately identify important speeches of leaders in news articles, and by improving the minhash algorithm, it is more effective in processing specific texts, making it more accurate in error recognition. This method can also be extended to the identification of poems in articles or the identification of famous quotes from other important figures.

The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions of the foregoing embodiments. The recorded technical solutions may be modified, or some of the technical features thereof may be equivalently replaced, but these modifications or substitutions shall not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims (6)

1. A content identification and error correction method for specific sentences in text, which is characterized by including the following steps: S1. Apply the regular matching rules to initially identify the golden sentence part. The recognition situation is mainly divided into two categories. One category is included in double quotes. This category can be correctly matched after being regularized and enters step S3; the other category is not included in double quotes. , enter step S2 for further processing; S2. Use the trained hfl/rbt3 model to predict the upper and lower sentences, determine the golden sentence part, and enter step S3; S3. Collect the golden sentence database and divide the identified golden sentence parts into sentences. The matching method is divided according to the sentence length. When the sentence length is greater than or equal to the set threshold, the minhash matching method is used to calculate the minimum sentence. Hash value, store the hash value segments and sentences corresponding to the sentences in the database into the hash table, use inverted index matching to find the most similar sentences, and enter step S4; when the sentence length is less than the set threshold , using the Jaccard similarity matching method, calculate the Jaccard value of the target sentence and other sentences in the database, find the sentence with the largest Jaccard value, and enter step S4; S4. Start from the first sentence as the target sentence and match it in the database. If it matches, compare the subsequent parts of the sentence and the recognized golden sentence part in the database one by one to see if there are any errors, and output the recognition result and the error part. 2. The content identification and error correction method for text-specific sentences according to claim 1, characterized in that the hfl/rbt3 model training in step S2 adopts positive and negative samples of the upper and lower sentences, wherein the positive samples are derived from the golden sentence database. In the upper and lower sentences, negative samples are collected manually. After data enhancement is performed on the negative samples, the pre-trained model hfl/rbt3 is input for training. The data enhancement methods include but are not limited to random deletion, random replacement, and random addition. 3. The content identification and error correction method for text-specific sentences according to claim 1, characterized in that the specific steps for further judgment using the trained model in step S2 are as follows: S201. First, obtain a paragraph without quotation marks, and divide the paragraph into sentences according to periods, exclamation marks, question marks, and ellipses; S202. Then, consider the first sentence as a golden sentence, then this sentence will be used as the previous sentence, and the next sentence will be used as the next sentence. After being processed by the lexical analyzer of the hfl/rbt3 model, it will be input into the hfl/rbt3 model to determine the return value. ; S203. Initialize the result. The first sentence is a golden sentence, and then input it into the model. If the return value is 1, it means that the next sentence is also a golden sentence. Add this sentence to the result sentence, repeat step S202, and use this sentence as the previous sentence. Then it is judged whether the following sentence is the next sentence; if the return value is 0, it indicates that the next sentence is not a golden sentence, then it exits and does not continue the judgment. It ends when all sentences are traversed, and the result sentence is obtained, which is the golden sentence part. 4. The content identification and error correction method for text-specific sentences according to claim 1, characterized in that the specific steps of performing a minimum hash operation on the sentence in step S3 are: S301. First, perform 3-shingle processing on the sentence to obtain a shingles list; S302. Define the number of random permutation transformations as perm_num, and then randomly produce the a and b coefficients of perm_num affine transformation functions, that is: h(x)=(ax+b)%m Where m is the Mersenne prime number, which takes 2 ⁶¹ -1, a and b are random integers less than the maximum value, and the maximum value takes 2 ³² -1; S303. For each hash function, perform a hash operation on each shingle, and find the minimum value as the minimum hash value of this hash function. Perm_num minimum hash functions constitute the minimum hash value of the final original text. 5. The content identification and error correction method for text-specific sentences according to claim 1, characterized in that the specific steps of storing the database sentences in step S3 are: (1) Define the number of random permutation transformations as perm_num, divide all random permutations into b segments, each segment has r hashes, assuming that the real jaccard similarity between the two texts is js, then between the two texts The probability that a corresponding part of the perm_num minimum hashes is equal is equal to js. It can be deduced that the probability that all hashes of the minimum hash signatures of two texts in a certain segment are equal, that is, the probability of becoming a candidate pair is: 1-(1-s ^r ) ^b The relationship curve between the document Jaccard similarity and the probability of becoming a candidate pair is an S-shaped curve. The similarity of the curve is the threshold at 1/2 of the candidate probability, that is, the Jaccard similarity of 0.5. At this time, the approximate relationship between b, r and the threshold is estimated as follows: (1/b) ^1/r Find the optimal values of b and r according to the provided threshold, perm_num, false positive example weight and false negative example weight, where false positive examples are dissimilar text pairs hashed to the same bucket, and false negative examples are For similar text pairs that are not hash-mapped into a bucket, set the false positive example weight and false negative example weight to 0.5 respectively; (2) Calculate the minhash value of the sentences in the database respectively, and then divide the obtained perm_num hashes into b blocks. Each block has r minimum hashes, and the content of the divided b blocks is used as the b segment hash value of the sentence; (3) Record the hash value of segment b into the hash table, where each hash value is an index, that is, a key, and the index value is the corresponding sentence representation. At the same time, a dictionary is created, whose key is the sentence representation, recorded as mi, i is an integer starting from 0, and the value is the sentence itself. 6. The content identification and error correction method for text-specific sentences according to claim 1, characterized in that the specific steps of inverted index matching in step S3 are: For the target sentence, first determine the length of the sentence. If it is greater than a fixed value, minhash is done and the corresponding hash value is obtained. All hash values are indexed in the hash table to find a candidate set with a part of the same hash value. Then find the instance in the candidate set that is closest to the Hamming distance of the target sentence. If it is within the tolerance range, it will be output. Otherwise, it will be marked as unrecognized.