CN110852056B - Method, device and equipment for obtaining text similarity and readable storage medium - Google Patents

Abstract

The invention discloses a method, a device, equipment and a readable storage medium for acquiring text similarity, belonging to the technical field of communication, wherein the method comprises the following steps: obtaining numerical characteristics of the text pairs according to the text pair data set; constructing a sample feature matrix through the numerical features of the text pairs; model training is carried out according to the sample feature matrix and the prediction vector, and a prediction model is obtained; obtaining a target text pair, and obtaining a similarity score of the target text pair according to the sample feature matrix and the prediction model; the text similarity is judged by acquiring a plurality of numerical characteristics of the text pairs and considering semantic and syntactic structures, and the method has the advantages of being trainable in weight, less in manual intervention, simple and quick, easy to implement, high in accuracy and the like, and user experience is improved.

Description

A method, device, equipment and readable storage medium for obtaining text similarity

Technical Field

The present invention relates to the field of communication technology, and in particular to a method, device, equipment and readable storage medium for obtaining text similarity.

Background Art

In the era of information explosion, people's demand for quickly and accurately obtaining the required content from massive information is increasing day by day. To meet this demand, many applications have emerged, such as information retrieval, intelligent question and answer, document duplicate checking, personalized recommendation, etc. Behind these applications, text similarity calculation technology is one of the key core technologies.

Text similarity has been widely discussed in different fields. Due to different application scenarios, its connotation varies, so there is no unified and recognized definition. From the perspective of information theory, text similarity is related to the commonalities and differences between texts. The greater the commonality and the smaller the difference, the higher the similarity between texts; conversely, the smaller the commonality and the greater the difference, the lower the similarity between texts.

Summary of the invention

This article aims to provide a method, device, equipment and readable storage medium for obtaining text similarity. By obtaining multiple numerical features of text pairs, the text similarity is judged taking into account semantic and syntactic structures. It has the advantages of trainable weights, less manual intervention, simple and fast, easy to implement, high accuracy, etc., which improves the user experience.

The technical solutions adopted in this paper to solve the above technical problems are as follows:

According to one aspect of this invention, a method for obtaining text similarity is provided, comprising:

Obtaining numerical features of the text pairs according to the text pair dataset;

Constructing a sample feature matrix through the numerical features of the text pairs;

Performing model training according to the sample feature matrix and the prediction vector to obtain a prediction model;

A target text pair is obtained, and a similarity score of the target text pair is obtained according to the sample feature matrix and the prediction model.

Optionally, the numerical features include: semantic features based on ordered edit distance, semantic features based on unordered edit distance, semantic features based on word meaning distance, and syntactic features based on dependency relationship.

Optionally, obtaining the numerical features of the text pair according to the text pair dataset includes:

Obtaining a training corpus file, wherein the training corpus file includes a plurality of text pairs and a similarity score of each text pair;

Obtaining a training data set according to the training corpus file;

Obtaining a word vector matrix from the training data set;

According to the word vector matrix and the edit distance, obtaining a first improved edit distance between the text pairs as a semantic feature based on the ordered edit distance;

According to the edit distance and the bag-of-words model, obtaining a second improved edit distance between the text pairs as a semantic feature based on the unordered edit distance;

According to the word vector matrix, obtaining the word meaning distance between the text pairs as a semantic feature based on the word meaning distance;

Perform dependency syntactic analysis on text pairs and obtain the syntactic distance between text pairs as a syntactic feature based on dependency relations.

Optionally, the acquiring the target text pair, and acquiring the similarity score of the target text pair according to the sample feature matrix and the prediction model comprises:

Obtain a target text pair, obtain numerical features of the target text pair, and construct a feature vector of the target text pair;

Substitute the feature vector of the target text pair into the prediction model to obtain a similarity score of the target text pair.

According to another aspect of the present invention, a device for obtaining text similarity is provided, comprising:

A training module, used for obtaining numerical features of the text pairs according to the text pair data set;

A matrix construction module, used for constructing a sample feature matrix through the numerical features of the text pairs;

A prediction module, used for performing model training according to the sample feature matrix and the prediction vector to obtain a prediction model;

The online acquisition module is used to acquire a target text pair and acquire a similarity score of the target text pair according to the sample feature matrix and the prediction model.

Optionally, the numerical features include: semantic features based on ordered edit distance, semantic features based on unordered edit distance, semantic features based on word meaning distance, and syntactic features based on dependency relationship.

Optionally, the training module includes:

An acquisition unit, used for acquiring a training corpus file, wherein the training corpus file includes a plurality of groups of text pairs and a similarity score of each group of text pairs;

An extraction unit, used to obtain a training data set according to the training corpus file;

A word vector acquisition unit, used to obtain a word vector matrix from the training data set;

An ordered edit distance acquisition unit, used to acquire a first improved edit distance between text pairs according to the word vector matrix and the edit distance as a semantic feature based on the ordered edit distance;

An unordered edit distance acquisition unit, used to acquire a second improved edit distance between text pairs according to the edit distance and the bag-of-words model as a semantic feature based on the unordered edit distance;

A word meaning distance acquisition unit, used to acquire the word meaning distance between text pairs according to the word vector matrix as a semantic feature based on the word meaning distance;

The syntactic distance acquisition unit is used to perform dependency syntactic analysis on the text pairs and acquire the syntactic distance between the text pairs as a syntactic feature based on the dependency relationship.

Optionally, the online acquisition module includes:

A feature vector acquisition unit, used to acquire a target text pair, acquire numerical features of the target text pair, and form a feature vector of the target text pair;

The similarity acquisition unit is used to substitute the feature vector of the target text pair into the prediction model to obtain the similarity score of the target text pair.

According to another aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and at least one application stored in the memory and configured to be executed by the processor, wherein the application is configured to execute the above-mentioned method for obtaining text similarity.

According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for obtaining text similarity described above is implemented.

The embodiments of the present invention provide a method, device, equipment and readable storage medium for obtaining text similarity. The method comprises: obtaining numerical features of text pairs according to a text pair data set; constructing a sample feature matrix through the numerical features of the text pairs; performing model training according to the sample feature matrix and a prediction vector to obtain a prediction model; obtaining a target text pair, and obtaining a similarity score of the target text pair according to the sample feature matrix and the prediction model; and determining text similarity by obtaining multiple numerical features of the text pairs and taking into account semantic and syntactic structures. The method has the advantages of trainable weights, less manual intervention, simplicity and speed, easy implementation, high accuracy, etc., thereby improving user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for obtaining text similarity provided by Embodiment 1 of the present invention;

FIG. 2 is a flow chart of a method of step S10 in FIG. 1 ;

FIG3 is a flow chart of a method of step S40 in FIG1 ;

FIG4 is an exemplary structural block diagram of an apparatus for obtaining text similarity provided in Embodiment 2 of the present invention;

FIG5 is an exemplary structural block diagram of the training module in FIG4;

FIG. 6 is an exemplary structural block diagram of the online acquisition module in FIG. 4 .

The realization of the purpose, functional features and advantages of this article will be further explained in conjunction with embodiments and with reference to the accompanying drawings.

DETAILED DESCRIPTION

In order to make the technical problems, technical solutions and beneficial effects to be solved by this article clearer and more understandable, this article is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain this article and are not used to limit this article.

Embodiment 1

As shown in FIG. 1 , in this embodiment, a method for obtaining text similarity includes:

S10, obtaining numerical features of the text pairs according to the text pair data set;

S20, constructing a sample feature matrix through the numerical features of the text pairs;

S30, performing model training according to the sample feature matrix and the prediction vector to obtain a prediction model;

S40: Obtain a target text pair, and obtain a similarity score of the target text pair according to the sample feature matrix and the prediction model.

In this embodiment, the text similarity is judged by obtaining multiple numerical features of the text pair, taking into account the semantic and syntactic structures. The semantic similarity between the texts, including features such as word meaning, edit distance, and bag-of-words model, is considered. The grammatical similarity including syntactic structure is also considered, and the semantics and syntax are combined. A neural network is used to perform feature extraction at a higher level, which has the advantages of trainable weights, less manual intervention, simple and fast, easy implementation, and high accuracy, thereby improving the user experience.

In step S10, first, a large amount of annotated text pair data is prepared as training corpus; each sample in the training corpus is a set of text pairs and corresponding annotated similarity scores, which can be formally expressed as [text1; text2; score], where text1 and text2 are text pairs for similarity acquisition, and score is the annotated similarity score. The annotation score can come from manual annotation or other prior information, such as the user's satisfaction with the system's answer in the question-and-answer system, the user's browsing of the system's arrangement information in the retrieval system, etc. All samples are saved in the file originalData.txt, each line is a training sample, and text1, text2, and score in each training sample can be separated by a tab. Among them, the annotated similarity score score is a real number between 0 and 1, and the larger the number, the higher the similarity between the text pairs, and vice versa; in particular, a score of 0 indicates that the text pairs are completely unrelated, and a score of 1 indicates that the text pairs are exactly the same. The accuracy of the score is not fixed depending on the source. For example, if it comes from manual annotation, it may be a single-digit decimal such as 0.3 or 0.6. If it comes from other application systems, it may be a multi-digit decimal such as 0.563 or 0.8192.

In this embodiment, the format of the file originalData.txt is as follows:

I would like to ask where I can buy ZTE mobile phones, where can I buy ZTE mobile phones, 0.769

ZTE Corporation in Nanjing Yuhuatai District, Nanjing Yuhuatai District ZTE Corporation, 0.816

The intelligent question-answering system team has made new achievements, the field of intelligent question-answering is changing with each passing day, 0.324

What are the channels for applying for credit cards, how to apply for debit cards, 0.814.

In this embodiment, assuming that the training corpus file has a total of M lines of text pairs, and N numerical features are obtained for each text pair in the training corpus, the sample feature matrix extracted from the training corpus can be expressed as X∈R ^M×N . For each text pair in the training corpus, the annotation similarity score is used as the predicted value of the sample, and a prediction vector y∈R ^M×1 can be extracted from the training corpus. Therefore, the training data set can be expressed as D=[X,y].

In this embodiment, the numerical features include: semantic features based on ordered edit distance, semantic features based on unordered edit distance, semantic features based on word meaning distance, and syntactic features based on dependency relationship.

In this embodiment, in addition to the ordered edit distance, the moving distance of disordered words is also considered, which has stronger adaptability to texts with simple word order reversal and can greatly improve the system recall rate. Moreover, this embodiment also obtains syntactic similarity based on the number of valid dependency pairs in the sentence, which can better measure the number of core words in the sentence and the words that have dependency relationships with them.

As shown in FIG. 2 , in this embodiment, step S10 includes:

S11, obtaining a training corpus file, wherein the training corpus file includes a plurality of text pairs and a similarity score of each text pair;

S12, obtaining a training data set according to the training corpus file;

S13, obtaining a word vector matrix from the training data set;

In this embodiment, the word vector training method can adopt the common method in the industry. Here, the Word2Vec method is adopted. The specific steps are as follows:

S131. Generate a new training corpus file originalDataForWord2Vec.txt from the file originalData.txt: obtain only text1 and text2 for each line of samples in the file originalData.txt, and then divide text1 and text2 into two lines for storage.

The corpus file originalDataForWord2Vec.txt is in the following format:

I would like to ask where I can buy a ZTE mobile phone

Where to buy ZTE mobile phones

ZTE Corporation in Yuhuatai District, Nanjing

ZTE Corporation in Yuhuatai District, Nanjing

The intelligent question-answering system team has made new achievements

The field of intelligent question answering is changing with each passing day

What are the channels for applying for credit cards?

How to apply for a debit card

S132. Use word2vec to perform word vector training, and the vector length is denoted as _dw (eg, _dw = 400).

S133. Record the trained wordv2ec model as a matrix Where V is the vocabulary consisting of all the words in the corpus file, |V| is the number of words in the vocabulary, represents a real matrix with |V| rows and _w columns.

S134, the word vector obtained by the model for word w can be expressed as in, The word vector is represented as a matrix with 1 row and d _w columns, where w is a variable and can refer to any word, such as "ZTE".

S14, obtaining a first improved edit distance between text pairs according to the word vector matrix and the edit distance as a semantic feature based on the ordered edit distance;

In this embodiment, the editing operations defined in the first improved edit distance c _A include: match (Mat), insert (Ins), delete (Del), and replace (Sub), and the corresponding operation costs are c _Mat , c _Ins , c _Del , and c _Sub . The specific calculation steps are as follows:

S141. After performing word segmentation and stop word removal operations on the texts text1 and text2, word sequences t1 and t2 are obtained.

For example, text1 is "I want to apply for in-app purchase of ZTE mobile phone", after word segmentation, it is [I|want|apply|in-app purchase|ZTE|mobile phone|], after removing stop words, the word sequence t1 is [apply|in-app purchase|ZTE|mobile phone]; text2 is "How to apply for in-app purchase of ZTE products", after word segmentation, it is [how to|apply|for|ZTE|product|of|in-app purchase|], after removing stop words, the word sequence t2 is [how to|apply|ZTE|product|in-app purchase]. Among them, "I", "want", "have", "a while", "of", "are" are all stop words.

S142: Calculate the edit path Path _A from the word sequence t1 to the word sequence t2 and the corresponding edit element sequence Elements _A using a general method (such as a method based on dynamic programming).

For example, using the general method, we can calculate that the editing path Path _A from t1 = [apply | in-app purchase | ZTE | mobile phone] to t2 = [how to | apply | ZTE | product | in-app purchase] is [Ins, Mat, Sub, Sub, Sub], and the corresponding editing element sequence Elements _A is [how to, apply, in-app purchase → ZTE, ZTE → product, mobile phone → in-app purchase]. Among them, no arrow indicates Mat, Ins, Del operations, and arrow indicates Sub operation.

S143, obtaining a corresponding editing operation cost vector Action _A for the editing path Path _A. Specifically, all editing operations are replaced with corresponding operation costs to form an editing operation cost vector.

For example, the editing path Path _A is [Ins, Mat, Sub, Sub, Sub], and the corresponding editing operation cost vector is [c _Ins , c _Mat , c _Sub , c _Sub , c _Sub ].

S144. Calculate the edit element distance for each element in the edit element sequence Elements _A , thereby obtaining the edit element distance vector Dis _A. Specifically, the edit element distance for Mat, Ins, and Del operations is 1, and the edit element distance for Sub operation is sim _cos (w ₁ ,w ₂ ). Wherein, sim _cos (w ₁ ,w ₂ ) is the cosine similarity between word w ₁ and word w ₂ , which can be expressed as

For example, the editing element sequence Elements _A is [how, apply, in-app purchase→ZTE, ZTE→product, mobile phone→in-app purchase], and the corresponding editing element distance vector Dis _A is [1,1,0.218,0.294,0.511].

S145, calculating the improved edit distance between the two texts according to the edit operation cost vector Action _A and the corresponding edit element distance vector Dis _A As a semantic feature based on ordered edit distance.

For example, the edit operation cost vector is [c _Ins ,c _Mat ,c _Sub ,c _Sub ,c _Sub ], and the corresponding edit element distance vector is [1,1,0.218,0.294,0.511], then:

c _A ＝1*c _Ins +1*c _Mat +1*c _Ins +0.218*c _Sub +0.294*c _Sub +0.511*c _Sub .

S15, calculating a second improved edit distance between text pairs according to the edit distance and the bag-of-words model as a semantic feature based on the unordered edit distance;

In this embodiment, the editing operations defined in the second improved edit distance c _B include: match (Mat), insert (Ins), delete (Del), and the corresponding operation costs are c _Mat , c _Ins , and c _Del , respectively. The specific calculation steps are as follows:

S151, word sequences t1 and t2 are obtained after word segmentation and stop word removal operations on texts text1 and text2.

S152, add all non-repeating words in word sequences t1 and t2 to the set to form a bag of words BOW.

For example, for t1 = [apply|in-app purchase|ZTE|mobile phone] and t2 = [how to|apply|ZTE|product|in-app purchase], the obtained bag of words BOW is [how to|apply|in-app purchase|ZTE|mobile phone|product].

S153. Calculate the edit distance from t1 to t2 based on the bag of words BOW and t1 and t2. The calculation method is as follows: for a word w in the bag of words BOW, if the word or its synonym exists in t1 and if the word or its synonym exists in t2, then record the operation Mat; if the word or its synonym exists in t1 and if the word or its synonym does not exist in t2, then record the operation Del; if the word or its synonym does not exist in t1 and if the word or its synonym exists in t2, then record the operation Ins. After performing the above operations on all words in the bag of words BOW in sequence, the edit path Path _B can be obtained, and then the corresponding edit operation cost vector Action _B can be obtained.

For example, the editing path Path _B from t1 = [apply | in-app purchase | ZTE | mobile phone] to t2 = [how to | apply | ZTE | product | in-app purchase] is [Ins, Mat, Mat, Mat, Del, Ins], then the editing operation cost vector Action _B is [c _Ins , c _Mat , c _Mat , c _Mat , c _Del , c _Ins ].

S154 , summing up all elements in the edit operation cost vector Action _B to obtain a second improved edit distance c _B between the two texts as a semantic feature based on the unordered edit distance.

For example, for the editing operation cost vector Action _B =[c _Ins ,c _Mat ,c _Mat ,c _Mat ,c _Del ,c _Ins ], c _B =c _Ins +c _Mat +c _Mat +c _Mat +c _Del +c _Ins .

S16, calculating the word meaning distance between the text pairs according to the word vector matrix as a semantic feature based on the word meaning distance;

In this step, first, the texts text1 and text2 are segmented and stop words are removed to obtain word sequences t1 and t2. Assume that the words contained in t1 are The words contained in t2 are Secondly, calculate the word With words Semantic distance Among them, For example, the subscript m represents the total number of t1 word sequences, and the superscript 1 represents that the word belongs to t1, sequence. Similarly, The subscript n represents the total number of word sequences in t2, and the superscript 2 represents that the word belongs to t2, thus defining the word in t1 The semantic distance from t2 is T2 Chinese word The semantic distance from t1 is Finally, calculate the semantic similarity between the two texts As a semantic feature based on word meaning distance.

S17. Perform dependency syntactic analysis on the text pairs and calculate the syntactic distance between the text pairs as a syntactic feature based on the dependency relationship.

In this step, first, the word sequences t1 and t2 are obtained after the word segmentation and stop word removal operations on the texts text1 and text2. Secondly, dependency syntactic analysis is performed on t1 and t2 respectively using common methods (such as StanfordNLP, FNLP and other tools), and the number of valid word collocation pairs in t1 and t2 is calculated respectively, which are recorded as _p1 and _p2 . Among them, a valid collocation pair refers to a collocation pair consisting of a core word in a sentence and a valid word that directly depends on it. The core word is the only core word in the entire sentence obtained after the sentence is analyzed by dependency syntactic analysis; the valid word is the noun, verb and adjective after the sentence is analyzed by dependency syntactic analysis.

For example, for t1 = [apply|in-app purchase|ZTE|mobile phone], after dependency syntactic analysis, the core word is "in-app purchase", and the words directly dependent on it are "apply" and "mobile phone", and these two words are both valid words, so the number of valid collocation pairs of t1 is 2. The syntactic structure distance c _D = |p ₁ -p ₂ | between the two texts is calculated for p ₁ and p ₂ as a syntactic feature based on dependency relations.

As shown in FIG. 3 , in this embodiment, step S40 includes:

S41, obtaining a target text pair, obtaining numerical features of the target text pair, and forming a feature vector of the target text pair;

S42: Substitute the feature vector of the target text pair into the prediction model to obtain a similarity score of the target text pair.

In this embodiment, a network structure for training is first built, and then the model is trained according to the sample feature matrix X and the prediction vector y obtained in the previous section, and finally the model is saved for subsequent online acquisition.

The network structure adopts a multi-layer perceptron (MLP), and uses the sample feature matrix X and the prediction vector y to perform model training on the above network structure using a general method.

After training, the obtained model parameters are recorded as W ^1* , b ^1* , W ^2* , b ^2* , where W ^1* represents the connection weight of the first layer in MLP, b ^1* is the bias of the second layer in MLP, W ^2* is the connection weight of the second layer in MLP, and b ^2* is the bias of the second layer in MLP. The prediction model can be expressed as Where x ^T is the feature vector of the sample to be predicted, g ¹ is the nonlinear activation function of the first layer in MLP, g ² is the nonlinear activation function of the second layer in MLP, and x ^T is the feature vector of the target text pair.

In this embodiment, for the target text pair t1 and t2 input into the system, according to the above numerical feature calculation steps, four numerical features c _A , c _B , c _C , c _D of the text pair are calculated in sequence to form a feature vector x ^T = [c _A , c _B , c _C , c _D ] of the target text pair.

Substituting the feature vector of the above target text pair into the prediction model, we can get the similarity score of the target text pair t1 and t2:

Embodiment 2

As shown in FIG4 , in this embodiment, a device for obtaining text similarity includes:

A training module 10, used to obtain numerical features of the text pairs according to the text pair data set;

A matrix construction module 20, used to construct a sample feature matrix through the numerical features of the text pairs;

A prediction module 30, configured to perform model training according to the sample feature matrix and the prediction vector to obtain a prediction model;

The online acquisition module 40 is used to acquire a target text pair and acquire a similarity score of the target text pair according to the sample feature matrix and the prediction model.

In this embodiment, the text similarity is judged by obtaining multiple numerical features of the text pair, taking into account the semantic and syntactic structures. The semantic similarity between the texts, including features such as word meaning, edit distance, and bag-of-words model, is considered. The grammatical similarity including syntactic structure is also considered, and the semantics and syntax are combined. A neural network is used to perform feature extraction at a higher level, which has the advantages of trainable weights, less manual intervention, simple and fast, easy implementation, and high accuracy, thereby improving the user experience.

In this embodiment, a large amount of annotated text pair data is first prepared as training corpus; each sample in the training corpus is a set of text pairs and corresponding annotated similarity scores, which can be formally expressed as [text1; text2; score], where text1 and text2 are text pairs for similarity calculation, and score is the annotated similarity score. The annotated score can come from manual annotation or other prior information, such as the user's satisfaction with the system's answer in the question-and-answer system, the user's browsing of the system's arrangement information in the retrieval system, etc. All samples are saved in the file originalData.txt, each line is a training sample, and text1, text2, and score in each training sample can be separated by a tab. Among them, the annotated similarity score score is a real number between 0 and 1, and the larger the number, the higher the similarity between the text pairs, and vice versa; in particular, a score of 0 indicates that the text pairs are completely unrelated, and a score of 1 indicates that the text pairs are exactly the same. The accuracy of the score is not fixed depending on the source. For example, if it comes from manual annotation, it may be a single-digit decimal such as 0.3 or 0.6. If it comes from other application systems, it may be a multi-digit decimal such as 0.563 or 0.8192.

In this embodiment, the format of the file originalData.txt is as follows:

I would like to ask where I can buy ZTE mobile phones, where can I buy ZTE mobile phones, 0.769

ZTE Corporation in Nanjing Yuhuatai District, Nanjing Yuhuatai District ZTE Corporation, 0.816

The intelligent question-answering system team has made new achievements, the field of intelligent question-answering is changing with each passing day, 0.324

What are the channels for applying for credit cards, how to apply for debit cards, 0.814.

In this embodiment, assuming that the training corpus file has a total of M lines of text pairs, and N numerical features are obtained for each text pair in the training corpus, the sample feature matrix extracted from the training corpus can be expressed as X∈R ^M×N . For each text pair in the training corpus, the annotation similarity score is used as the predicted value of the sample, and a prediction vector y∈R ^M×1 can be extracted from the training corpus. Therefore, the training data set can be expressed as D=[X,y].

In this embodiment, the numerical features include: semantic features based on ordered edit distance, semantic features based on unordered edit distance, semantic features based on word meaning distance, and syntactic features based on dependency relationship.

In this embodiment, in addition to the ordered edit distance, the moving distance of disordered words is also considered, which has stronger adaptability to texts with simple word order reversal and can greatly improve the system recall rate. Moreover, this embodiment also calculates the syntactic similarity based on the number of valid dependency pairs in the sentence, which can better measure the number of core words in the sentence and the words with dependency relationships therewith.

As shown in FIG5 , in this embodiment, the training module includes:

An acquisition unit 11 is used to acquire a training corpus file, wherein the training corpus file includes a plurality of text pairs and a similarity score of each text pair;

An extraction unit 12, used to obtain a training data set according to the training corpus file;

A word vector acquisition unit 13, used to obtain a word vector matrix from the training data set;

In this embodiment, the word vector training method can adopt the common method in the industry. Here, the Word2Vec method is adopted. The specific steps are as follows:

S131. Generate a new training corpus file originalDataForWord2Vec.txt from the file originalData.txt: obtain only text1 and text2 for each line of samples in the file originalData.txt, and then divide text1 and text2 into two lines for storage.

The corpus file originalDataForWord2Vec.txt is in the following format:

I would like to ask where I can buy a ZTE mobile phone

Where to buy ZTE mobile phones

ZTE Corporation in Yuhuatai District, Nanjing

ZTE Corporation in Yuhuatai District, Nanjing

The intelligent question-answering system team has made new achievements

The field of intelligent question answering is changing with each passing day

What are the channels for applying for credit cards?

How to apply for a debit card

S132. Use word2vec to perform word vector training, and the vector length is denoted as _dw (eg, _dw = 400).

S133. Record the trained wordv2ec model as a matrix Where V is the vocabulary consisting of all the words in the corpus file, |V| is the number of words in the vocabulary, represents a real matrix with |V| rows and _w columns.

S134, the word vector obtained by the model can be expressed as in, The word vector is represented as a matrix with 1 row and d _w columns, where w is a variable and can refer to any word, such as "ZTE".

An ordered edit distance acquisition unit 14, configured to acquire a first improved edit distance between text pairs according to the word vector matrix and the edit distance as a semantic feature based on the ordered edit distance;

In this embodiment, the editing operations defined in the first improved edit distance c _A include: match (Mat), insert (Ins), delete (Del), and replace (Sub), and the corresponding operation costs are c _Mat , c _Ins , c _Del , and c _Sub . The specific calculation steps are as follows:

S141. After performing word segmentation and stop word removal operations on the texts text1 and text2, word sequences t1 and t2 are obtained.

For example, text1 is "I want to apply for in-app purchase of ZTE mobile phone", after word segmentation, it is [I|want|apply|in-app purchase|ZTE|mobile phone|], after removing stop words, the word sequence t1 is [apply|in-app purchase|ZTE|mobile phone]; text2 is "How to apply for in-app purchase of ZTE products", after word segmentation, it is [how to|apply|for|ZTE|product|of|in-app purchase|], after removing stop words, the word sequence t2 is [how to|apply|ZTE|product|in-app purchase]. Among them, "I", "want", "have", "a while", "of", "are" are all stop words.

S142: Calculate the edit path Path _A from the word sequence t1 to the word sequence t2 and the corresponding edit element sequence Elements _A using a general method (such as a method based on dynamic programming).

For example, using the general method, we can calculate that the editing path Path _A from t1 = [apply | in-app purchase | ZTE | mobile phone] to t2 = [how to | apply | ZTE | product | in-app purchase] is [Ins, Mat, Sub, Sub, Sub], and the corresponding editing element sequence Elements _A is [how to, apply, in-app purchase → ZTE, ZTE → product, mobile phone → in-app purchase]. Among them, no arrow indicates Mat, Ins, Del operations, and arrow indicates Sub operation.

S143, obtaining a corresponding editing operation cost vector Action _A for the editing path Path _A. Specifically, all editing operations are replaced with corresponding operation costs to form an editing operation cost vector.

For example, the editing path Path _A is [Ins, Mat, Sub, Sub, Sub], and the corresponding editing operation cost vector is [c _Ins , c _Mat , c _Sub , c _Sub , c _Sub ].

S144. Calculate the edit element distance for each element in the edit element sequence Elements _A , thereby obtaining the edit element distance vector Dis _A. Specifically, the edit element distance for Mat, Ins, and Del operations is 1, and the edit element distance for Sub operation is sim _cos (w ₁ ,w ₂ ). Wherein, sim _cos (w ₁ ,w ₂ ) is the cosine similarity between word w ₁ and word w ₂ , which can be expressed as

For example, the editing element sequence Elements _A is [how, apply, in-app purchase→ZTE, ZTE→product, mobile phone→in-app purchase], and the corresponding editing element distance vector Dis _A is [1,1,0.218,0.294,0.511].

S145, calculating the improved edit distance between the two texts according to the edit operation cost vector Action _A and the corresponding edit element distance vector Dis _A As a semantic feature based on ordered edit distance.

For example, the edit operation cost vector is [c _Ins ,c _Mat ,c _Sub ,c _Sub ,c _Sub ], and the corresponding edit element distance vector is [1,1,0.218,0.294,0.511], then:

c _A ＝1*c _Ins +1*c _Mat +1*c _Ins +0.218*c _Sub +0.294*c _Sub +0.511*c _Sub .

An unordered edit distance acquisition unit 15, configured to acquire a second improved edit distance between text pairs according to the edit distance and the bag-of-words model as a semantic feature based on the unordered edit distance;

In this embodiment, the editing operations defined in the second improved edit distance c _B include: match (Mat), insert (Ins), delete (Del), and the corresponding operation costs are c _Mat , c _Ins , and c _Del , respectively. The specific calculation steps are as follows:

S151, word sequences t1 and t2 are obtained after word segmentation and stop word removal operations on texts text1 and text2.

S152, add all non-repeating words in word sequences t1 and t2 to the set to form a bag of words BOW.

For example, for t1 = [apply|in-app purchase|ZTE|mobile phone] and t2 = [how to|apply|ZTE|product|in-app purchase], the obtained bag of words BOW is [how to|apply|in-app purchase|ZTE|mobile phone|product].

S153. Calculate the edit distance from t1 to t2 based on the bag of words BOW and t1 and t2. The calculation method is as follows: for a word w in the bag of words BOW, if the word or its synonym exists in t1 and if the word or its synonym exists in t2, then record the operation Mat; if the word or its synonym exists in t1 and if the word or its synonym does not exist in t2, then record the operation Del; if the word or its synonym does not exist in t1 and if the word or its synonym exists in t2, then record the operation Ins. After performing the above operations on all words in the bag of words BOW in sequence, the edit path Path _B can be obtained, and then the corresponding edit operation cost vector Action _B can be obtained.

For example, the editing path Path _B from t1 = [apply | in-app purchase | ZTE | mobile phone] to t2 = [how to | apply | ZTE | product | in-app purchase] is [Ins, Mat, Mat, Mat, Del, Ins], then the editing operation cost vector Action _B is [c _Ins , c _Mat , c _Mat , c _Mat , c _Del , c _Ins ].

S154 , summing up all elements in the edit operation cost vector Action _B to obtain a second improved edit distance c _B between the two texts as a semantic feature based on the unordered edit distance.

For example, for the editing operation cost vector Action _B =[c _Ins ,c _Mat ,c _Mat ,c _Mat ,c _Del ,c _Ins ], c _B =c _Ins +c _Mat +c _Mat +c _Mat +c _Del +c _Ins .

A word meaning distance acquisition unit 16, used to acquire the word meaning distance between text pairs according to the word vector matrix as a semantic feature based on the word meaning distance;

In this embodiment, first, the word sequences t1 and t2 are obtained after the word segmentation and stop word removal operations on the texts text1 and text2. Assume that the words contained in t1 are The words contained in t2 are Secondly, calculate the word With words Semantic distance Among them, For example, the subscript m represents the total number of t1 word sequences, and the superscript 1 represents that the word belongs to t1, sequence. Similarly, The subscript n represents the total number of word sequences in t2, and the superscript 2 represents that the word belongs to t2, thus defining the word in t1 The semantic distance from t2 is T2 Chinese word The semantic distance from t1 is Finally, calculate the semantic similarity between the two texts As a semantic feature based on word meaning distance.

The syntactic distance acquisition unit 17 is used to perform dependency syntactic analysis on the text pairs and acquire the syntactic distance between the text pairs as a syntactic feature based on the dependency relationship.

In this embodiment, first, the word sequences t1 and t2 are obtained after the word segmentation and stop word removal operations on the texts text1 and text2. Secondly, a general method (such as StanfordNLP, FNLP and other tools) is used to perform dependency syntactic analysis on t1 and t2 respectively, and the number of valid word collocation pairs in t1 and t2 is calculated respectively, which are recorded as _p1 and _p2 . Among them, the valid collocation pair refers to the collocation pair consisting of the core word in the sentence and the valid words directly dependent on it. The core word is the only core vocabulary in the whole sentence obtained after the sentence is analyzed by dependency syntax; the valid word is the noun, verb and adjective after the sentence is analyzed by dependency syntax.

For example, for t1 = [apply|in-app purchase|ZTE|mobile phone], after dependency syntactic analysis, the core word is "in-app purchase", and the words directly dependent on it are "apply" and "mobile phone", and these two words are both valid words, so the number of valid collocation pairs of t1 is 2. The syntactic structure distance c _D = |p ₁ -p ₂ | between the two texts is calculated for p ₁ and p ₂ as a syntactic feature based on dependency relations.

As shown in FIG6 , in this embodiment, the online acquisition module includes:

A feature vector acquisition unit 41 is used to acquire a target text pair, calculate the numerical features of the target text pair, and form a feature vector of the target text pair;

The similarity acquisition unit 42 is used to substitute the feature vector of the target text pair into the prediction model to obtain the similarity score of the target text pair.

In this embodiment, a network structure for training is first built, and then the model is trained according to the sample feature matrix X and the prediction vector y obtained in the previous section, and finally the model is saved for subsequent online calculations.

The network structure adopts a multi-layer perceptron (MLP), and uses the sample feature matrix X and the prediction vector y to perform model training on the above network structure using a general method.

After training, the obtained model parameters are recorded as W ^1* , b ^1* , W ^2* , b ^2* , and the prediction model can be expressed as Where x ^T is the feature vector of the sample to be predicted.

In this embodiment, for the target text pair t1 and t2 input into the system, according to the above numerical feature calculation steps, four numerical features c _A , c _B , c _C , c _D of the text pair are calculated in sequence to form a feature vector x ^T = [c _A , c _B , c _C , c _D ] of the target text pair.

Substituting the feature vector of the above target text pair into the prediction model, we can get the target text pair t1 and

Similarity score of t2:

Embodiment 3

In this embodiment, an electronic device includes a memory, a processor, and at least one application stored in the memory and configured to be executed by the processor, wherein the application is configured to execute the method for obtaining text similarity described in Embodiment 1.

Embodiment 4

An embodiment of the present invention provides a readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program implements any method embodiment described in the above method embodiments for obtaining text similarity.

It should be noted that the above-mentioned device, equipment and readable storage medium embodiments belong to the same concept as the method embodiments. The specific implementation process is detailed in the method embodiments, and the technical features in the method embodiments are correspondingly applicable in the device embodiments, which will not be repeated here.

The embodiments of the present invention provide a method, device, equipment and readable storage medium for obtaining text similarity. The method comprises: obtaining numerical features of text pairs according to a text pair data set; constructing a sample feature matrix through the numerical features of the text pairs; performing model training according to the sample feature matrix and a prediction vector to obtain a prediction model; obtaining a target text pair, and obtaining a similarity score of the target text pair according to the sample feature matrix and the prediction model; and determining text similarity by obtaining multiple numerical features of the text pairs and taking into account semantic and syntactic structures. The method has the advantages of trainable weights, less manual intervention, simplicity and speed, easy implementation, high accuracy, etc., thereby improving user experience.

Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules/units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

In hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present invention shall be within the scope of the present invention.

Claims (8)

1. A method for obtaining text similarity, characterized by comprising: Obtaining numerical features of the text pairs according to the text pair data set, the numerical features including: semantic features based on ordered edit distance, semantic features based on unordered edit distance, semantic features based on word meaning distance, and syntactic features based on dependency relationship; Constructing a sample feature matrix through the numerical features of the text pairs; Performing model training according to the sample feature matrix and the prediction vector to obtain a prediction model; Obtaining a target text pair, and obtaining a similarity score of the target text pair according to the sample feature matrix and the prediction model; Among them, the semantic feature based on the ordered edit distance is the first improved edit distance between text pairs obtained according to the word vector matrix and the edit distance of the text pair data set, the semantic feature based on the unordered edit distance is the second improved edit distance between text pairs obtained according to the edit distance and the bag-of-words model, the semantic feature based on the word meaning distance is the word meaning distance between text pairs obtained according to the word vector matrix, and the syntactic feature based on the dependency relationship is the syntactic distance between the text pairs obtained by performing dependency syntactic analysis on the text pairs. 2. A method for obtaining text similarity according to claim 1, characterized in that the numerical features of the text pair obtained according to the text pair data set include: Obtaining a training corpus file, wherein the training corpus file includes a plurality of text pairs and a similarity score of each text pair; Obtaining a training data set according to the training corpus file; Obtaining a word vector matrix from the training data set; According to the word vector matrix and the edit distance, obtaining a first improved edit distance between the text pairs as a semantic feature based on the ordered edit distance; According to the edit distance and the bag-of-words model, obtaining a second improved edit distance between the text pairs as a semantic feature based on the unordered edit distance; According to the word vector matrix, obtaining the word meaning distance between the text pairs as a semantic feature based on the word meaning distance; Perform dependency syntactic analysis on text pairs and obtain the syntactic distance between text pairs as a syntactic feature based on dependency relations. 3. A method for obtaining text similarity according to claim 2, characterized in that the obtaining of the target text pair and obtaining the similarity score of the target text pair according to the sample feature matrix and the prediction model comprises: Obtain a target text pair, obtain numerical features of the target text pair, and construct a feature vector of the target text pair; Substitute the feature vector of the target text pair into the prediction model to obtain a similarity score of the target text pair. 4. A device for obtaining text similarity, characterized by comprising: A training module, used for obtaining numerical features of the text pairs according to the text pair data set, wherein the numerical features include: semantic features based on ordered edit distance, semantic features based on unordered edit distance, semantic features based on word meaning distance, and syntactic features based on dependency relationship; A matrix construction module, used for constructing a sample feature matrix through the numerical features of the text pairs; A prediction module, used for performing model training according to the sample feature matrix and the prediction vector to obtain a prediction model; An online acquisition module, used to acquire a target text pair, and acquire a similarity score of the target text pair according to the sample feature matrix and the prediction model; Among them, the semantic feature based on the ordered edit distance is the first improved edit distance between text pairs obtained according to the word vector matrix and the edit distance of the text pair data set, the semantic feature based on the unordered edit distance is the second improved edit distance between text pairs obtained according to the edit distance and the bag-of-words model, the semantic feature based on the word meaning distance is the word meaning distance between text pairs obtained according to the word vector matrix, and the syntactic feature based on the dependency relationship is the syntactic distance between the text pairs obtained by performing dependency syntactic analysis on the text pairs. 5. The device for obtaining text similarity according to claim 4, wherein the training module comprises: An acquisition unit, used for acquiring a training corpus file, wherein the training corpus file includes a plurality of groups of text pairs and a similarity score of each group of text pairs; An extraction unit, configured to obtain a training data set according to the training corpus file; A word vector acquisition unit, used to obtain a word vector matrix from the training data set; An ordered edit distance acquisition unit, used to acquire a first improved edit distance between text pairs according to the word vector matrix and the edit distance as a semantic feature based on the ordered edit distance; An unordered edit distance acquisition unit, used to acquire a second improved edit distance between text pairs according to the edit distance and the bag-of-words model as a semantic feature based on the unordered edit distance; A word meaning distance acquisition unit, used to acquire the word meaning distance between text pairs according to the word vector matrix as a semantic feature based on the word meaning distance; The syntactic distance acquisition unit is used to perform dependency syntactic analysis on the text pairs and acquire the syntactic distance between the text pairs as a syntactic feature based on the dependency relationship. 6. The device for obtaining text similarity according to claim 5, characterized in that the online obtaining module comprises: A feature vector acquisition unit, used to acquire a target text pair, acquire numerical features of the target text pair, and form a feature vector of the target text pair; The similarity acquisition unit is used to substitute the feature vector of the target text pair into the prediction model to obtain the similarity score of the target text pair. 7. An electronic device comprising a memory, a processor and at least one application stored in the memory and configured to be executed by the processor, characterized in that the application is configured to execute the method for obtaining text similarity as described in any one of claims 1-3. 8. A readable storage medium, characterized in that a computer program is stored thereon, and when the program is executed by a processor, the method for obtaining text similarity as described in any one of claims 1 to 3 is implemented.