CN104573130B - The entity resolution method and device calculated based on colony - Google Patents

Abstract

An embodiment of the present invention provides a method and device for entity resolution based on group computing, the method includes: first performing hierarchical clustering on the initial records in the database to obtain at least two cluster subsets; when an increase in the database is detected When a new record is found, at least two relevant cluster subsets most relevant to the new record are obtained from the at least two cluster subsets, and candidate candidates corresponding to the at least two relevant cluster subsets are determined. record pair; judging whether at least one of the candidate record pairs represents the same entity through crowdsourcing user annotation; if it is determined that the first candidate record pair represents the same entity, then add the new record to the first cluster to which the first record belongs In the subset; if it is determined that all the candidate record pairs do not represent the same entity, a new cluster subset is established for the new record, and a label set is created for the new cluster subset; thus static and dynamic Entity parsing is performed on the dataset, which improves parsing efficiency.

Description

Entity resolution method and device based on swarm computing

technical field

Embodiments of the present invention relate to computer technology, and in particular to a group computing-based entity resolution method and device.

Background technique

A database is a warehouse that organizes, stores and manages data according to the data structure; with the development of information technology and the market, data management is no longer just storing and managing data, but has transformed into various data management methods required by users. Entity resolution is proposed in the process of database management, wherein the purpose of entity resolution is to identify different records representing the same entity in the database. With the advent of the big data era, more and more data needs to be matched or integrated before further analysis and processing. Therefore, the demand for high-quality entity resolution is growing rapidly.

Existing entity resolution methods are mainly aimed at static data sources (that is, assuming that the data source is static), and each entity resolution process resolves the entire data source. However, in practical applications, new data will be added, deleted or modified in the database every time, that is, most data sources are dynamically changing, such as information submitted by users on social networking sites, product information on e-commerce sites, Bug resource library in the field of software engineering, etc.; if the existing entity resolution method is adopted, the entity resolution of the entire data source needs to be performed every time there is new data in the database, which costs a lot, that is, the resolution efficiency is low.

Contents of the invention

Embodiments of the present invention provide a method and device for entity analysis based on group computing, which can perform entity analysis on static and dynamic data sets, and achieve higher recall and precision with less cost, thereby improving analysis efficiency.

In the first aspect, an embodiment of the present invention provides a group computing-based entity resolution method, including:

The crowdsourcing-based hierarchical clustering method performs hierarchical clustering on the initial records in the database to obtain at least two cluster subsets;

When detecting that a new record has been added to the database, acquire feature information of the new record;

obtaining at least two relevant cluster subsets most relevant to the new record from the at least two cluster subsets according to the subset information of the at least two cluster subsets and the feature information of the new record; Wherein, the subset information of the at least two cluster subsets includes: label set information and index information of the cluster subsets;

determining candidate record pairs respectively corresponding to the at least two relevant cluster subsets according to the similarity relationship between the new record and each record in the at least two relevant cluster subsets;

Judging whether at least one of the candidate record pairs represents the same entity through crowdsourcing user labeling; if it is determined that the first candidate record pair represents the same entity, then adding the new record to the first cluster subset to which the first record belongs, And update the label set of the first clustering subset; if it is determined that all the candidate record pairs do not represent the same entity, a new clustering subset is established for the new record, and the new clustering sub-set A set of tags is created; wherein the first record and the new record form the first candidate record pair.

Optionally, the crowdsourcing-based hierarchical clustering method performs hierarchical clustering on the initial records in the database to obtain at least two cluster subsets, including:

According to the probability of representing the same entity between each pair of the initial records, the initial record pairs whose probability of representing the same entity is greater than the upper probability threshold are clustered into one class to form a corresponding primary cluster subset, and for each of the primary The clustering subset creates a label set and an index; wherein, each pair of the initial records forms the initial record pair;

The primary cluster subsets are successively merged hierarchically by means of crowdsourcing user labeling until the minimum distance between the merged cluster subsets is greater than the lower threshold, and finally at least two cluster subsets are obtained.

Optionally, according to the probability of representing the same entity between each pair of the initial records, the initial record pairs whose probability of representing the same entity is greater than the upper probability threshold are clustered into one class to form corresponding primary clustering subsets, including :

obtaining the probability that said pair of initial records represent the same entity;

The initial record pairs whose probability of representing the same entity is greater than the upper probability threshold are clustered into one class to form corresponding primary cluster subsets.

Optionally, the primary cluster subsets are successively merged hierarchically through crowdsourcing user annotation until the minimum distance between the merged cluster subsets is greater than the lower threshold, and finally at least two cluster subsets are obtained. A subset of clusters, including:

Step A, calculating the distance between each pair of primary cluster subsets in the primary cluster subsets, and selecting the pair of primary cluster subsets with the smallest distance as two candidate merging subsets;

Step B, judging whether the distance between the two candidate merging subsets is less than the lower limit threshold; if the distance between the two candidate merging subsets is less than the lower limit threshold, the two candidate merging subsets Centrally select the second record to form a second candidate record pair, and send the second candidate record pair and the label sets of the two candidate merged subsets to the crowdsourcing platform, so that the crowdsourcing platform can judge the second candidate record pair Whether the candidate record pair represents the same entity and whether the label in the label set is praised; wherein, the second candidate record pair is the record pair with the greatest probability of representing the same entity in the two candidate merged subsets;

Step C. Receive the first judgment result returned by the crowdsourcing platform, and determine whether to merge the two candidate merging subsets according to the first judgment result, and determine whether to merge the two candidate merged subsets according to the crowdsourcing platform's classification of the tags in the tag set Sorting and/or filtering the tags in the tag set according to the number of likes; if it is determined according to the first judgment result that the two candidate merging subsets represent the same entity, then merging the two candidate merging subsets into A clustering subset, updating the label set and index of the clustering subset, and using the merged clustering subset as a primary clustering subset; if the two If the candidate merging subsets do not represent the same entity, the distance between the two candidate merging subsets is set to 1;

Go back and continue to execute the step A-step C until the distance between the two candidate merging subsets is greater than the lower threshold, then at least two of the primary clustering subsets are used as the obtained at least two Clustering subsets.

Optionally, the obtaining the probability that the initial record pair represents the same entity includes:

calculating the similarity of the initial record pair according to the similarity between the corresponding attributes of the initial record pair;

A probability that the pair of initial records represent the same entity is calculated based on a machine learning model.

Optionally, the calculating the distance between each pair of primary cluster subsets in the primary cluster subsets includes:

Select the record pair (r _i , r _j ) with the highest probability representing the same entity from each pair of primary clustering subsets, where r _i ∈ C _i , r _j ∈ C _j , and C _i is the A primary cluster subset in the primary cluster subset, _Cj is another primary cluster subset in each pair of primary cluster subsets;

According to the formula Get the distance between each pair of primary clustering subsets; where, maxSimi is the probability that the record pair (r _i , r _j ) represents the same entity, and cosinSimi is the cosine similarity of each pair of primary clustering subsets Spend.

Optionally, according to the subset information of the at least two cluster subsets and the feature information of the new record, at least two most relevant to the new record are obtained from the at least two cluster subsets. A subset of related clusters, including:

Establishing an inverted index for the label set information and index information of the at least two clustering subsets obtained by performing hierarchical clustering on the initial records in the database according to the crowdsourcing-based hierarchical clustering method;

Searching is performed according to the inverted index and the feature information of the new record, and the at least two relevant cluster subsets most relevant to the new record are obtained from the at least two cluster subsets.

Optionally, the determining the candidate record pairs respectively corresponding to the at least two related cluster subsets according to the similarity relationship between the new record and each record in the at least two related cluster subsets includes :

separately calculating the similarity of the new record to each record in the at least two relevant subsets of clusters;

Respectively select a record with the greatest similarity to the new record from each of the related cluster subsets, and respectively form a candidate record pair corresponding to the related cluster subset with the new record; wherein, the The number of relevant cluster subsets is equal to the number of candidate record pairs.

Optionally, judging whether at least one of the candidate record pairs represents the same entity through crowdsourcing user annotation; if it is determined that the first candidate record pair represents the same entity, then adding the new record to the first record to which the first record belongs In a clustering subset, and update the label set of the first clustering subset; if it is determined that all the candidate similar pairs do not represent the same entity, a new clustering subset is established for the new record, and is The new subset of clusters creates a label set consisting of:

sending all the candidate record pairs to the crowdsourcing platform, so that the crowdsourcing platform determines whether the candidate record pairs represent the same entity;

receiving the second judgment result returned by the crowdsourcing platform, and determining whether at least one of the candidate record pairs represents the same entity according to the second judgment result; if it is determined according to the second judgment result that the first candidate record pair represents the same entity entity, add the new record to the first cluster subset to which the first record belongs, and update the label set of the first cluster subset; if all the candidate records are determined according to the second judgment result If the pairs do not represent the same entity, a new cluster subset is established for the new record, and a label set is created for the new cluster subset.

In the second aspect, an embodiment of the present invention provides a group computing-based entity resolution device, including:

A hierarchical clustering module for hierarchically clustering the initial records in the database based on a crowdsourcing hierarchical clustering method to obtain at least two cluster subsets;

A detection module, configured to obtain feature information of the new record when it is detected that a new record has been added to the database;

The first determination module is configured to obtain at least two most relevant to the new record from the at least two cluster subsets according to the subset information of the at least two cluster subsets and the feature information of the new record. related clustering subsets; wherein, the subset information of the at least two clustering subsets includes: label set information and index information of the clustering subsets;

A second determination module, configured to determine candidate record pairs respectively corresponding to the at least two relevant cluster subsets according to the similarity relationship between the new record and each record in the at least two relevant cluster subsets;

The division module is used to judge whether at least one of the candidate record pairs represents the same entity through crowdsourcing user labeling; if it is determined that the first candidate record pair represents the same entity, then add the new record to the first record to which the first record belongs. In the clustering subset, and update the label set of the first clustering subset; if it is determined that all the candidate record pairs do not represent the same entity, a new clustering subset is established for the new record, and for all The new cluster subset creates a label set; wherein, the first record and the new record form the first candidate record pair.

In the present invention, the hierarchical clustering method based on crowdsourcing performs hierarchical clustering on the initial records in the database to obtain at least two cluster subsets; further, when it is detected that a new record has been added to the database, Obtaining the feature information of the new record; further, obtaining the new record from the at least two cluster subsets according to the subset information of the at least two cluster subsets and the feature information of the new record The most relevant at least two relevant cluster subsets, wherein the subset information of the at least two cluster subsets includes: label set information and index information of the cluster subsets; further, according to the new The similarity relationship between the record and each record in the at least two related cluster subsets determines the candidate record pairs corresponding to the at least two related cluster subsets; further, it is judged whether At least one of the candidate record pairs represents the same entity; if it is determined that the first candidate record pair represents the same entity, the new record is added to the first cluster subset to which the first record belongs, and the first cluster is updated the label set of the subset; if it is determined that all the candidate record pairs do not represent the same entity, then a new cluster subset is established for the new record, and a label set is created for the new cluster subset; wherein, the The first record and the new record form the first candidate record pair; entity resolution can be performed on static and dynamic data sets, and a higher recall rate and precision rate can be achieved with less cost, thereby improving analysis efficiency.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 is a schematic flowchart of Embodiment 1 of the entity resolution method based on group computing in the present invention;

FIG. 2 is a schematic flowchart of Embodiment 2 of the entity resolution method based on group computing in the present invention;

FIG. 3 is a schematic flowchart of Embodiment 3 of the entity resolution method based on group computing in the present invention;

FIG. 4 is a schematic structural diagram of Embodiment 1 of an entity resolution device based on swarm computing in the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Because in some scenarios, different records representing the same entity are usually not the same; the main task of entity resolution is to identify different records representing the same entity in the database, which is especially important when cleaning or integrating data from multiple data sources. For example, a mailbox list may contain many records that actually refer to the same physical address, but there must be some differences between each record due to some different spellings or missing information. For example, a company may have several different databases for storing user profile information (each database belongs to a sub-department). Generally, the company hopes to obtain a more complete profile of each user by integrating these user information; Since each user information may appear in different forms in each different database, that is, there is no unified identifier, therefore, it is not easy to identify matching user information among multiple databases.

Machine learning is a multi-field interdisciplinary subject that has emerged in the past 20 years, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Machine learning theory is mainly to design and analyze some algorithms that allow computers to "learn" automatically, that is, machine learning algorithms are a type of algorithms that automatically analyze and obtain laws from data, and use the laws to predict unknown data. Machine learning algorithms can be roughly divided into four categories: supervised learning, semi-supervised learning, unsupervised learning and enhanced learning; 1) Supervised learning refers to learning a function from a given training data set. When new data arrives, it can be The result is predicted according to this function; among them, the training data set requirement of supervised learning is to include input and output (ie features and targets), and the targets in the training data set are marked by people; common supervised learning algorithms include regression analysis and statistical classification; 2) Compared with supervised learning, unsupervised learning has no human-labeled results in the training data set, and common unsupervised learning algorithms have clustering; 3) Semi-supervised learning is between supervised learning and unsupervised learning; 4) Enhanced learning Learning through observation, that is, every action will have an impact on the environment, and the learning object makes judgments based on the feedback of the observed surrounding environment.

With the advent of the big data era, the demand for high-quality entity resolution is growing rapidly. Traditional entity resolution solutions are based on machine learning. Although there have been a lot of research work in the field of entity resolution based on machine learning, due to the The semantic analysis, domain knowledge and related experience involved in the process, the accuracy rate of the machine in judging whether different records are the same entity is not as high as that of human judgment, that is, the machine processing is still not accurate enough. With the rapid development of the market based on the crowdsourcing model, by applying manual labeling to the process of entity resolution, that is, entity resolution based on the crowdsourcing platform, although the accuracy of manual labeling is higher than that of machine judgment, it will cost time and money. bring about greater costs.

At present, entity resolution methods based on group computing are only applicable to static databases, that is, the entire database is parsed every time. Among them, the idea of group computing is to combine crowdsourcing with machine learning or artificial intelligence and cloud computing. Fuse the efficiency of computer processing with the accuracy of crowd intelligence to solve problems. But in practical applications, databases are dynamic, such as landmark datasets in Facebook and bug resource libraries in software engineering. That is, new data will be added to the database every time, and it needs to be analyzed with the existing data in the database. Therefore, the traditional entity resolution method, which is only suitable for static data sources, cannot meet the needs of dynamic data sources.

The present invention analyzes the data by integrating the efficiency of computer processing and the accuracy of crowd intelligence, and proposes a scheme based on crowd computing that can perform entity analysis on static and dynamic data sets. , achieving high recall and precision.

Fig. 1 is a schematic flowchart of Embodiment 1 of the entity resolution method based on group computing in the present invention. As shown in Fig. 1, the method of this embodiment may include:

S101. Hierarchical clustering is performed on initial records in a database using a crowdsourcing-based hierarchical clustering method to obtain at least two cluster subsets.

Since most of the records in the actual application data source are not repeated; if all the record pairs are handed over to the crowdsourcing platform for judgment, it is not feasible in terms of economy and time. Therefore, it can be obtained based on machine learning. The repetition probability of the record pair, by setting the upper and lower probability thresholds to filter out the record pairs with a high probability or a very small probability representing the same entity, that is, the record pair with a probability greater than the upper probability threshold represents the same entity, and the probability is less than the lower limit Pairs of records with probability thresholds do not represent the same entity.

In the embodiment of the present invention, the records representing the same entity are gathered into the same subcategory through the hierarchical clustering algorithm (that is, the record pairs in different subcategories represent different entities); Hierarchical partition clusters are composed, and partitions between hierarchies have nested relationships. Specifically, by adopting a bottom-up strategy for clustering, firstly, the primary cluster subsets are obtained after the filtering step, and then according to the distance between each pair of primary cluster subsets and in a certain order through crowdsourcing user labeling Iteratively merge the primary cluster subsets into larger cluster subsets hierarchically until the minimum distance between each cluster subset after merging is greater than the lower threshold, that is, each pair of cluster subsets contains The probability that the records represent the same entity is less than the lower limit probability threshold (wherein, the smaller the distance between the two cluster subsets, it means that the record pair between the two cluster subsets represents the same entity The greater the probability of or the greater the probability of duplication between the two cluster subsets; the greater the distance between the two cluster subsets, it means that the record pair between the two cluster subsets represents The smaller the probability of the same entity or the smaller the probability of duplication between the two cluster subsets); wherein, the minimum distance between each cluster subset after merging is greater than the lower limit threshold, it represents each cluster subset The probability that the pairs of records represent the same entity is less than the lower probability threshold, that is, each cluster subset does not represent the same entity, so the clustering algorithm stops. Optionally, obtaining the primary cluster subset after passing the filtering step includes: performing preliminary clustering on the initial records in the database according to the upper limit probability threshold, for example, the probability of representing the same entity is greater than the upper limit probability threshold The initial record pairs are clustered into one class to form corresponding primary cluster subsets; wherein, the records in each primary cluster subset represent the same entity, and the records between different primary cluster subsets do not represent the same entity.

Optionally, step 101 includes:

According to the probability of representing the same entity between each pair of the initial records, the initial record pairs whose probability of representing the same entity is greater than the upper probability threshold are clustered into one class to form a corresponding primary cluster subset, and for each of the primary The clustering subset creates a label set and an index; wherein, each pair of the initial records forms the initial record pair;

The primary cluster subsets are successively merged hierarchically by means of crowdsourcing user labeling until the minimum distance between the merged cluster subsets is greater than the lower threshold, and finally at least two cluster subsets are obtained.

In the embodiment of the present invention, optionally, firstly, the probability that the initial record pair represents the same entity is acquired; secondly, the initial record pairs whose probability of representing the same entity is greater than the upper limit probability threshold are grouped into one group to form a corresponding Primary cluster subsets; further, for subsequent dynamic data query, create label sets and indexes for each of the primary cluster subsets, optionally, label each of the primary cluster sub-sets by machine or manually Set creates a label set, optionally by tokenizing, removing stopwords, stemming the attribute value of each record in the primary cluster subset, and then selecting to recur between records and across the entire dataset Inverse document frequency (inverse document frequency, abbreviated as IDF) The words (ie keywords) with larger values are added to the label set of the primary clustering subset (wherein, the IDF value is a measure of the general importance of a word, and a specific The IDF value of word is equal to: the number of total file divided by the number of the file that contains this word obtains quotient value, then described quotient value is taken logarithmic numerical value that obtains); And according to the label set information of described primary cluster subset as An index is created for each of the primary cluster subsets, wherein the index information includes keywords recorded in the primary cluster subsets; further, the primary cluster subsets are sequentially classified by crowdsourcing user annotation Merge layer by layer until the minimum distance between the merged cluster subsets is greater than the lower limit threshold, which means that the probability that the record pairs between each cluster subset represent the same entity is less than the lower limit probability threshold, that is The individual cluster subsets do not represent the same entity, so the clustering algorithm stops until at least two cluster subsets are finally obtained.

Optionally, the obtaining the probability that the initial record pair represents the same entity includes:

calculating the similarity of the initial record pair according to the similarity between the corresponding attributes of the initial record pair;

A probability that the pair of initial records represent the same entity is calculated based on a machine learning model.

In the embodiment of the present invention, the similarity of each initial record pair (the initial record pair includes two initial records) can be represented by a feature vector, and each dimension of the feature vector represents two initial records The similarity of a certain attribute between, assuming that n functions to calculate the similarity are used to measure m attributes, then the dimension of the feature vector is n*m dimensions, that is, according to the relationship between the corresponding attributes of the initial record pair The similarity calculates the similarity of the initial record pair. Furthermore, the entity resolution based on the machine learning model can be regarded as a classification problem, for example, it means that two records represent the same entity, otherwise it means that the two entities represent different entities, that is, the input of a general classifier represents a pair of records The eigenvector of the similarity between them, the output is the classification result of the two types of problems, but the probability that a pair of records that this application needs to obtain represents the same entity, therefore, in the embodiment of the present invention, it is proposed to calculate the initial record pair based on a machine learning model The probability of representing the same entity, optionally, the classifier is trained through a training set, wherein the training set contains records representing duplicate records (i.e., records representing the same entity) and non-repeating records (i.e., records representing different entities) The feature vector, the trained classifier can represent the probability that each of the initial record pairs represent the same entity.

Optionally, the upper limit probability threshold and the lower limit probability threshold can be determined according to the recall rate and precision rate of the set target; if the upper limit probability threshold value is set too low, the precision rate will be reduced; If the lower limit probability threshold is too high, the recall rate will be reduced; if the upper limit probability threshold is too high or the lower limit probability threshold is too low, filtering efficiency will be affected.

S102. When it is detected that a new record is added to the database, acquire feature information of the new record.

In the embodiment of the present invention, when it is detected that a new record R has been added to the database, the feature information of the new record is obtained, and optionally, the attribute value of the new record is segmented, stop words removed, and stem, and extract keywords (i.e. feature information) according to the term frequency-inverse document frequency (term frequency-inverse document frequency, referred to as TF-IDF) value, so that according to the feature information of the new record and the existing Subset information of the cluster subset, reasonably classifying the new record, if it is determined that the new record and a record in a certain cluster subset represent the same entity, then merge the new record If it is determined that the new record does not represent the same entity as the records in any of the cluster subsets, a new cluster subset will be established for the new record.

S103. According to the subset information of the at least two cluster subsets and the feature information of the new record, obtain at least two related clusters most relevant to the new record from the at least two cluster subsets set.

In the embodiment of the present invention, according to the subset information of the at least two cluster subsets and the feature information of the new record, the most relevant information for the new record is obtained from the at least two cluster subsets by means of information retrieval. at least two relevant cluster subsets; optionally, the subset information of the at least two cluster subsets includes: label set information and index information of the cluster subsets.

Optionally, step S103 includes:

Establishing an inverted index for the label set information and index information of the at least two clustering subsets obtained by performing hierarchical clustering on the initial records in the database according to the crowdsourcing-based hierarchical clustering method;

Searching is performed according to the inverted index and the feature information of the new record, and the at least two relevant cluster subsets most relevant to the new record are obtained from the at least two cluster subsets.

In the embodiment of the present invention, the tag set information and index information of the at least two cluster subsets obtained by performing hierarchical clustering on the initial records in the database according to the crowdsourcing-based hierarchical clustering method are set up and inverted Index, optionally, according to the label set information and index information of each of the cluster subsets, all the labels in the label set of each of the cluster subsets are used as keys, and the storage of each of the cluster subsets The address is used as a value, that is, each item in the inverted index includes an attribute value and the storage address of the corresponding record with the attribute value (because the attribute value is not determined by the record, but the position of the record is determined by the attribute value , so it is called an inverted index); further, according to the inverted index and the feature information of the new record, the information retrieval method is used to retrieve, and from the at least two clustering subsets, the The at least two related cluster subsets that are related, that is, the at least two related cluster subsets that most likely represent the same entity as the new record.

S104. Determine candidate record pairs respectively corresponding to the at least two relevant cluster subsets according to the similarity relationship between the new record and each record in the at least two relevant cluster subsets.

In the embodiment of the present invention, according to the similarity relationship between the new record R and each record in the at least two related cluster subsets, one of the related cluster subsets is selected to be related to the new record The record R' with the largest similarity, so as to determine the candidate record pairs (R, R') respectively corresponding to the at least two related cluster subsets.

Optionally, the step S104 includes:

separately calculating the similarity of the new record to each record in the at least two relevant subsets of clusters;

Respectively select a record with the greatest similarity to the new record from each of the related cluster subsets, and respectively form a candidate record pair corresponding to the related cluster subset with the new record; wherein, the The number of relevant cluster subsets is equal to the number of candidate record pairs.

In the embodiment of the present invention, the similarity between the new record R and each record in the at least two related cluster subsets is calculated respectively, and further, one of the related cluster subsets is selected from each of the related cluster subsets respectively The record R' with the highest similarity of the new record R, and respectively form a candidate record pair (R, R') corresponding to the related cluster subset with the new record R; wherein, the related cluster subset The number is equal to the number of the candidate record pairs. If five relevant clustering subsets most relevant to the new record R are obtained in step S103, then in step S104, it is determined that the five relevant clustering subsets are respectively Corresponding five candidate record pairs. Optionally, the similarity between the new record R and each record in the at least two relevant cluster subsets may be calculated through a text similarity algorithm.

S105. Determine whether at least one of the candidate record pairs represents the same entity through crowdsourcing user annotation; if it is determined that the first candidate record pair represents the same entity, add the new record to the first cluster to which the first record belongs Centralize, and update the label set of the first clustering subset; if it is determined that all the candidate record pairs do not represent the same entity, then create a new clustering subset for the new record, and set up a new clustering subset for the new clustering A class subset creates a label set; wherein said first record and said new record form said first candidate record pair.

In the embodiment of the present invention, after the candidate record pairs corresponding to the at least two related cluster subsets are determined according to the similarity relationship between the new record and each record in the at least two related cluster subsets , send the candidate record pair (R, R') to the crowdsourcing platform, and judge whether at least one of the candidate record pairs (R, R') represents the same entity through crowdsourcing user annotation; if the first candidate record is determined A pair (R, R1') represents the same entity, then the new record R is added to the first cluster subset to which the first record R1' belongs (wherein, the first cluster subset is the at least two A cluster subset in the related cluster subset), and update the label set of the first cluster subset; if it is determined that all the candidate similar pairs (R, R') do not represent the same entity, then all The new record R establishes a new clustering subset, and creates a label set for the new clustering subset, optionally, creates a label set of the new clustering subset by extracting keywords of the new record; Wherein, the first candidate record pair (R, R1') is a candidate record pair among all the candidate record pairs (R, R').

Optionally, step S105 includes:

sending all the candidate record pairs to the crowdsourcing platform, so that the crowdsourcing platform determines whether the candidate record pairs represent the same entity;

receiving the second judgment result returned by the crowdsourcing platform, and determining whether at least one of the candidate record pairs represents the same entity according to the second judgment result; if it is determined according to the second judgment result that the first candidate record pair represents the same entity entity, add the new record to the first cluster subset to which the first record belongs, and update the label set of the first cluster subset; if all the candidate records are determined according to the second judgment result If the pairs do not represent the same entity, a new cluster subset is established for the new record, and a label set is created for the new cluster subset.

In the embodiment of the present invention, all the candidate record pairs (R, R') determined in step S104 are sent to the crowdsourcing platform, so that the crowdsourcing platform can judge whether the candidate record pairs (R, R') are represent the same entity, such as the crowdsourcing platform indicates whether the candidate record pair (R, R') represents the same entity by marking each candidate record pair (R, R'); further, receiving The second judgment result returned by the crowdsourcing platform (such as whether the crowdsourcing platform marks each of the candidate record pairs represents the same entity), and determines whether at least one of the Candidate record pairs represent the same entity, since the crowdsourcing platform will include multiple crowdsourcing users, that is, multiple crowdsourcing users will mark whether each candidate record pair represents the same entity, that is, the second judgment result is A plurality of crowdsourcing users mark whether each candidate record represents the result of the same entity. Optionally, a voting algorithm is used to aggregate crowdsourcing results, that is, the answer with more than half of the votes is selected as the result. If more than half of the crowdsourcing users agree If one of the candidate record pairs is marked as representing the same entity, it is determined that the candidate record pair represents the same entity, and if individual or less than half of the crowdsourcing users mark a certain candidate record pair as representing a different entity, it is determined The candidate record pair does not represent the same entity; if it is determined according to the second judgment result that the first candidate record pair represents the same entity, then add the new record to the first cluster subset to which the first record belongs, and update The label set of the first clustering subset; if it is determined according to the second judgment result that all the candidate record pairs do not represent the same entity, a new clustering subset is established for the new record, and for all Create a label set for the new subset of clusters described above.

In the embodiment of the present invention, the hierarchical clustering method based on crowdsourcing performs hierarchical clustering on the initial records in the database to obtain at least two cluster subsets; further, when it is detected that a new record has been added to the database , obtain the feature information of the new record; further, obtain the same information from the at least two cluster subsets according to the subset information of the at least two cluster subsets and the feature information of the new record. Newly record the most relevant at least two relevant cluster subsets, wherein the subset information of the at least two cluster subsets includes: label set information and index information of the cluster subsets; further, according to the The similarity relationship between the new record and each record in the at least two relevant cluster subsets determines the candidate record pairs corresponding to the at least two relevant cluster subsets; further, through crowdsourcing user labeling judging whether at least one of the candidate record pairs represents the same entity; if it is determined that the first candidate record pair represents the same entity, then adding the new record to the first cluster subset to which the first record belongs, and updating the first A label set of a cluster subset; if it is determined that all of the candidate record pairs do not represent the same entity, a new cluster subset is established for the new record, and a label set is created for the new cluster subset; wherein , the first record and the new record form the first candidate record pair; that is, entity resolution can be performed on static and dynamic data sets, and a higher recall rate and precision rate can be achieved with less cost, Thereby improving the analysis efficiency.

Optionally, the primary cluster subsets are successively merged hierarchically through crowdsourcing user annotation until the minimum distance between the merged cluster subsets is greater than the lower threshold, and finally at least two cluster subsets are obtained. A subset of clusters, including:

Step A, calculating the distance between each pair of primary cluster subsets in the primary cluster subsets, and selecting the pair of primary cluster subsets with the smallest distance as two candidate merging subsets;

Step B, judging whether the distance between the two candidate merging subsets is less than the lower limit threshold; if the distance between the two candidate merging subsets is less than the lower limit threshold, the two candidate merging subsets Centrally select the second record to form a second candidate record pair, and send the second candidate record pair and the label sets of the two candidate merged subsets to the crowdsourcing platform, so that the crowdsourcing platform can judge the second candidate record pair Whether the candidate record pair represents the same entity and whether the label in the label set is praised; wherein, the second candidate record pair is the record pair with the greatest probability of representing the same entity in the two candidate merged subsets;

Step C. Receive the first judgment result returned by the crowdsourcing platform, and determine whether to merge the two candidate merging subsets according to the first judgment result, and determine whether to merge the two candidate merged subsets according to the crowdsourcing platform's classification of the tags in the tag set Sorting and/or filtering the tags in the tag set according to the number of likes; if it is determined according to the first judgment result that the two candidate merging subsets represent the same entity, then merging the two candidate merging subsets into A clustering subset, updating the label set and index of the clustering subset, and using the merged clustering subset as a primary clustering subset; if the two If the candidate merging subsets do not represent the same entity, the distance between the two candidate merging subsets is set to 1;

Go back and continue to execute the step A-step C until the distance between the two candidate merging subsets is greater than the lower threshold, then at least two of the primary clustering subsets are used as the obtained at least two Clustering subsets.

In the embodiment of the present invention, in step A, the distance between each pair of primary cluster subsets in the primary cluster subsets is calculated, and the pair of primary cluster subsets with the smallest distance is selected as two candidate merging subsets ; Among them, the pair of primary cluster subsets with the smallest distance represents that the record pair between the primary cluster subsets has the highest probability of representing the same entity. Optionally, the calculating the distance between each pair of primary cluster subsets in the primary cluster subsets includes: selecting respectively from each pair of primary cluster subsets the record pair ( r _i , r _j ), where, r _i ∈ C _i , r _j ∈ C _j , C _i is a primary cluster subset of each pair of primary cluster subsets, and C _j is each pair of primary cluster subsets Another primary cluster subset in the class subset; further, according to the formula Get the distance between each pair of primary clustering subsets; where, maxSimi is the probability that the record pair (r _i , r _j ) represents the same entity, and cosinSimi is the cosine similarity of each pair of primary clustering subsets Spend. Optionally, other methods may also be used to calculate the distance between each pair of primary cluster subsets in the primary cluster subsets, which will not be repeated here.

Further, in step B, by judging whether the distance between the two candidate merging subsets is less than the lower limit threshold (that is, judging whether the probability that the records of the two candidate merging subsets represent the same entity is greater than the lower limit probability Threshold); if the distance between the two candidate merging subsets is less than the lower limit threshold (that is, the probability that the record pair between the two candidate merging subsets represents the same entity is greater than the lower limit probability threshold), Then select the second record from the two candidate merging subsets to form a second candidate record pair, and send the second candidate record pair and the label sets of the two candidate merging subsets to the crowdsourcing platform, so that The crowdsourcing platform judges whether the second candidate record pair represents the same entity and whether it likes the tags in the tag set, as indicated by the crowdsourcing platform by marking the second candidate record pair. Whether the second candidate record pair represents the same entity; wherein, the second candidate record pair is the record pair with the highest probability of representing the same entity in the two candidate merging subsets, such as selecting the representative in the two candidate merging subsets The record pair (r ₁ , r ₂ ) with the highest probability of the same entity (that is, the second candidate record pair), the two candidate merging subsets are C ₁ and C ₂ respectively, and r ₁ is the first record pair in C ₁ Second record, _r2 is the _second record in C2.

Further, in step C, the first judgment result returned by the crowdsourcing platform is received (such as the result of whether the crowdsourcing platform marks the second candidate record pair as representing the same entity and whether the two candidates are merged The results of the number of likes of tags in the tag set of the subset), and determine whether to merge the two candidate merging subsets according to the first judgment result and according to the number of likes of the tags in the tag set by the crowdsourcing platform Sorting and/or filtering the tags in the tag set, because the crowdsourcing platform will include multiple crowdsourcing users, that is, multiple crowdsourcing users will mark whether the second candidate record pair represents the same entity, that is, The second judgment result is the result of multiple crowdsourcing users marking whether the second candidate record pair represents the same entity and the number of likes for the tags in the tag set of the two candidate merged subsets, optional Specifically, the voting algorithm can be used to aggregate crowdsourcing results, that is, to select the answer with more than half of the votes as the result. If more than half of the crowdsourcing users mark the second candidate record pair as representing the same entity, then determine that the second candidate record pair represents the same entity. For the same entity, if individual or less than half of the crowdsourcing users mark the second candidate record pair as representing a different entity, it is determined that the second candidate record pair does not represent the same entity; if it is determined according to the first judgment result The two candidate merging subsets represent the same entity, then merge the two candidate merging subsets into one clustering subset, update the label set and index of the clustering subset, and merge the obtained The clustering subset is used as the primary clustering subset; if it is determined according to the first judgment result that the two candidate merging subsets do not represent the same entity, the distance between the two candidate merging subsets is set to 1 ; Further, return and continue to execute the steps A-Step C until the distance between the two candidate merging subsets is greater than the lower threshold, which means that the record pair representative between the two candidate merging subsets The probability of the same entity is less than the lower limit probability threshold, and since the pair of primary clustering subsets with the smallest distance is used as the candidate merging subset, if the clustering between the two candidate merging subsets is greater than the lower limit threshold , then the distances between other primary clustering subsets must also be greater than the lower limit threshold, that is, each of the initial clustering subsets does not represent the same entity, therefore, the clustering algorithm stops, and the at least two The primary clustering subsets are obtained as the at least two clustering subsets, so that iteratively merging the primary clustering subsets into a larger clustering subset in a certain order is achieved. .

Fig. 2 is a schematic flow chart of the second embodiment of the entity resolution method based on group computing in the present invention. On the basis of the above embodiment, the hierarchical clustering method based on crowdsourcing is used to perform hierarchical clustering on the initial records in the database The steps of obtaining at least two clustering subsets are described in detail, as shown in Figure 2, the method of this embodiment may include:

S201. Obtain the probability that the initial record pair represents the same entity, and execute step S202;

S202. Cluster the initial record pairs whose probability of representing the same entity is greater than the upper probability threshold into one class to form corresponding primary cluster subsets, and create a label set and index for each primary cluster subset, and execute Step S203;

S203. Calculate the distance between each pair of primary cluster subsets in the primary cluster subsets, select a pair of primary cluster subsets with the smallest distance as two candidate merging subsets, and perform step S204;

S204. Determine whether the distance between the two candidate merging subsets is smaller than the lower threshold; if yes, execute step S205; if not (that is, the distance between the two candidate merging subsets is greater than the lower threshold), Then execute step S206;

S205. Select second records from the two candidate merging subsets respectively to form a second candidate record pair, and send the second candidate record pair and the label sets of the two candidate merging subsets to the crowdsourcing platform to Make the crowdsourcing platform judge whether the second candidate record pair represents the same entity and whether it likes the tags in the tag set; further, step S207 is executed; wherein, the second candidate record pair is the two The record pair with the highest probability representing the same entity in candidate merged subsets;

S206. The clustering is terminated, and at least two of the primary cluster subsets are used as the obtained at least two cluster subsets;

S207. Receive the first judgment result returned by the crowdsourcing platform, and determine whether to merge the two candidate merging subsets according to the first judgment result, and set the tag points according to the crowdsourcing platform for the tags The number of likes sorts and/or filters the tags in the tag set; if it is determined according to the first judgment result that the two candidate merging subsets represent the same entity, then perform step S208; if according to the first judgment result If it is determined that the two candidate merging subsets do not represent the same entity, step S209 is performed;

S208. Merge the two candidate merging subsets into one clustering subset, update the label set and index of the clustering subset, use the merged clustering subset as a primary clustering subset, and Return to continue to execute step S203;

S209. Set the distance between the two candidate merging subsets as 1, and return to step S203.

Fig. 3 is a schematic flowchart of Embodiment 3 of the entity resolution method based on group computing in the present invention. On the basis of the above embodiments, as shown in Fig. 3 , the method of this embodiment may include:

S301. When it is detected that a new record has been added to the database, acquire feature information of the new record;

S302. According to the subset information of the at least two cluster subsets and the feature information of the new record, at least two related clusters most related to the new record are obtained from the at least two cluster subsets. class subset;

S303. Determine the candidate record pairs respectively corresponding to the at least two related cluster subsets, and send all the candidate record pairs to the crowdsourcing platform, so that the crowdsourcing platform can judge whether the candidate record pairs represent the same entity;

S304. Receive the second judgment result returned by the crowdsourcing platform, and determine whether at least one of the candidate record pairs represents the same entity according to the second judgment result; if the first candidate record pair is determined according to the second judgment result represent the same entity, execute step S305; if it is determined according to the second judgment result that all the candidate record pairs do not represent the same entity, execute step S306;

S305. Add the new record to the first cluster subset to which the first record belongs, and update the label set of the first cluster subset;

S306. Establish a new cluster subset for the new record, and create a label set for the new cluster subset.

In the embodiment of the present invention, the at least two cluster subsets in step S302 are existing cluster subsets in the database when it is detected that a new record is added in the database.

FIG. 4 is a schematic structural diagram of Embodiment 1 of an entity resolution device based on group computing in the present invention. As shown in FIG. 402 , a first determination module 403 , a second determination module 404 , and a division module 405 .

Wherein, the hierarchical clustering module 401 is used for hierarchically clustering the initial records in the database based on the crowdsourcing hierarchical clustering method to obtain at least two clustering subsets;

The detection module 402 is configured to obtain feature information of the new record when it is detected that a new record has been added in the database;

The first determining module 403 is configured to obtain at least two most relevant information of the new record from the at least two cluster subsets according to the subset information of the at least two cluster subsets and the feature information of the new record. related clustering subsets; wherein, the subset information of the at least two clustering subsets includes: label set information and index information of the clustering subsets;

The second determination module 404 is used to determine the candidate record pairs respectively corresponding to the at least two related cluster subsets according to the similarity relationship between the new record and each record in the at least two related cluster subsets;

The division module 405 is used to judge whether at least one of the candidate record pairs represents the same entity through crowdsourcing user annotation; if it is determined that the first candidate record pair represents the same entity, then add the new record to the first record to which the first record belongs. In the clustering subset, and update the label set of the first clustering subset; if it is determined that all the candidate record pairs do not represent the same entity, a new clustering subset is established for the new record, and for all The new cluster subset creates a label set; wherein, the first record and the new record form the first candidate record pair.

Optionally, the hierarchical clustering module includes:

The primary clustering unit is used to cluster the pairs of initial records whose probability of representing the same entity is greater than the upper limit probability threshold according to the probability of representing the same entity between each pair of the initial records to form corresponding primary clustering subsets, And create a tag set and an index for each of the primary clustering subsets; wherein, each pair of the initial records forms the initial record pair;

The hierarchical clustering unit is configured to sequentially merge the primary cluster subsets hierarchically through crowdsourcing user labeling, until the minimum distance between the merged cluster subsets is greater than the lower threshold, and finally obtain at least Two cluster subsets.

Optionally, the primary clustering unit is specifically used for:

obtaining the probability that said pair of initial records represent the same entity;

The initial record pairs whose probability of representing the same entity is greater than the upper probability threshold are clustered into one class to form corresponding primary cluster subsets.

Optionally, the hierarchical clustering unit is specifically used for:

Step A, calculating the distance between each pair of primary cluster subsets in the primary cluster subsets, and selecting the pair of primary cluster subsets with the smallest distance as two candidate merging subsets;

Step B, judging whether the distance between the two candidate merging subsets is less than the lower limit threshold; if the distance between the two candidate merging subsets is less than the lower limit threshold, the two candidate merging subsets Centrally select the second record to form a second candidate record pair, and send the second candidate record pair and the label sets of the two candidate merged subsets to the crowdsourcing platform, so that the crowdsourcing platform can judge the second candidate record pair Whether the candidate record pair represents the same entity and whether the label in the label set is praised; wherein, the second candidate record pair is the record pair with the greatest probability of representing the same entity in the two candidate merged subsets;

Step C. Receive the first judgment result returned by the crowdsourcing platform, and determine whether to merge the two candidate merging subsets according to the first judgment result, and determine whether to merge the two candidate merged subsets according to the crowdsourcing platform's classification of the tags in the tag set Sorting and/or filtering the tags in the tag set according to the number of likes; if it is determined according to the first judgment result that the two candidate merging subsets represent the same entity, then merging the two candidate merging subsets into A clustering subset, updating the label set and index of the clustering subset, and using the merged clustering subset as a primary clustering subset; if the two If the candidate merging subsets do not represent the same entity, the distance between the two candidate merging subsets is set to 1;

Go back and continue to execute the step A-step C until the distance between the two candidate merging subsets is greater than the lower threshold, then at least two of the primary clustering subsets are used as the obtained at least two Clustering subsets.

Optionally, the primary clustering unit is also specifically used for:

calculating the similarity of the initial record pair according to the similarity between the corresponding attributes of the initial record pair;

A probability that the pair of initial records represent the same entity is calculated based on a machine learning model.

Optionally, the hierarchical clustering unit is also specifically used for:

Select the record pair (r _i , r _j ) with the highest probability representing the same entity from each pair of primary clustering subsets, where r _i ∈ C _i , r _j ∈ C _j , and C _i is the A primary cluster subset in the primary cluster subset, _Cj is another primary cluster subset in each pair of primary cluster subsets;

According to the formula Get the distance between each pair of primary clustering subsets; where, maxSimi is the probability that the record pair (r _i , r _j ) represents the same entity, and cosinSimi is the cosine similarity of each pair of primary clustering subsets Spend.

Optionally, the first determining module is specifically configured to:

Establishing an inverted index for the label set information and index information of the at least two clustering subsets obtained by performing hierarchical clustering on the initial records in the database according to the crowdsourcing-based hierarchical clustering method;

Searching is performed according to the inverted index and the feature information of the new record, and the at least two relevant cluster subsets most relevant to the new record are obtained from the at least two cluster subsets.

Optionally, the second determination module is specifically configured to:

separately calculating the similarity of the new record to each record in the at least two relevant subsets of clusters;

Respectively select a record with the greatest similarity to the new record from each of the related cluster subsets, and respectively form a candidate record pair corresponding to the related cluster subset with the new record; wherein, the The number of relevant cluster subsets is equal to the number of candidate record pairs.

Optionally, the division module is specifically used for:

sending all the candidate record pairs to the crowdsourcing platform, so that the crowdsourcing platform determines whether the candidate record pairs represent the same entity;

receiving the second judgment result returned by the crowdsourcing platform, and determining whether at least one of the candidate record pairs represents the same entity according to the second judgment result; if it is determined according to the second judgment result that the first candidate record pair represents the same entity entity, add the new record to the first cluster subset to which the first record belongs, and update the label set of the first cluster subset; if all the candidate records are determined according to the second judgment result If the pairs do not represent the same entity, a new cluster subset is established for the new record, and a label set is created for the new cluster subset.

The entity resolution device based on swarm computing in this embodiment can be used to execute the technical solution in any embodiment of the above-mentioned entity resolution method based on swarm computing in the present invention, and its implementation principle and technical effect are similar, and will not be repeated here.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by program instructions and related hardware. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other various media that can store program codes.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims (10)

1. A method for entity resolution based on group computing, characterized in that, comprising: The crowdsourcing-based hierarchical clustering method performs hierarchical clustering on the initial records in the database to obtain at least two cluster subsets; When detecting that a new record has been added to the database, acquire feature information of the new record; obtaining at least two relevant cluster subsets most relevant to the new record from the at least two cluster subsets according to the subset information of the at least two cluster subsets and the feature information of the new record; Wherein, the subset information of the at least two cluster subsets includes: label set information and index information of the cluster subsets; determining candidate record pairs respectively corresponding to the at least two relevant cluster subsets according to the similarity relationship between the new record and each record in the at least two relevant cluster subsets; Judging whether at least one of the candidate record pairs represents the same entity through crowdsourcing user labeling; if it is determined that the first candidate record pair represents the same entity, then adding the new record to the first cluster subset to which the first record belongs, And update the label set of the first clustering subset; if it is determined that all the candidate record pairs do not represent the same entity, a new clustering subset is established for the new record, and the new clustering sub-set A set of tags is created; wherein the first record and the new record form the first candidate record pair. 2. The method according to claim 1, wherein the hierarchical clustering method based on crowdsourcing carries out hierarchical clustering to the initial records in the database to obtain at least two clustering subsets, comprising: According to the probability of representing the same entity between each pair of the initial records, the initial record pairs whose probability of representing the same entity is greater than the upper probability threshold are clustered into one class to form a corresponding primary cluster subset, and for each of the primary The clustering subset creates a label set and an index; wherein, each pair of the initial records forms the initial record pair; The primary cluster subsets are successively merged hierarchically by means of crowdsourcing user labeling until the minimum distance between the merged cluster subsets is greater than the lower threshold, and finally at least two cluster subsets are obtained. 3. The method according to claim 2, wherein, according to the probability of representing the same entity between each pair of the initial records, the initial records whose probability of representing the same entity is greater than the upper limit probability threshold are clustered into one class , forming the corresponding primary cluster subsets, including: obtaining the probability that said pair of initial records represent the same entity; The initial record pairs whose probability of representing the same entity is greater than the upper probability threshold are clustered into one class to form corresponding primary cluster subsets. 4. The method according to claim 2, wherein said primary clustering subsets are successively merged hierarchically by means of crowdsourcing user labeling, until the merged clustering subsets The minimum distance is greater than the lower threshold, and you end up with at least two subsets of clusters, including: Step A, calculating the distance between each pair of primary cluster subsets in the primary cluster subsets, and selecting the pair of primary cluster subsets with the smallest distance as two candidate merging subsets; Step B, judging whether the distance between the two candidate merging subsets is less than the lower limit threshold; if the distance between the two candidate merging subsets is less than the lower limit threshold, the two candidate merging subsets Centrally select the second record to form a second candidate record pair, and send the second candidate record pair and the label sets of the two candidate merged subsets to the crowdsourcing platform, so that the crowdsourcing platform can judge the second candidate record pair Whether the candidate record pair represents the same entity and whether the label in the label set is liked, the result of judging whether the second candidate record pair represents the same entity is the first judgment result; wherein, the second candidate record pair is the first judgment result; The record pair with the highest probability representing the same entity in the two candidate merged subsets; Step C. Receive the first judgment result returned by the crowdsourcing platform, and determine whether to merge the two candidate merging subsets according to the first judgment result, and determine whether to merge the two candidate merged subsets according to the crowdsourcing platform's classification of the tags in the tag set Sorting and/or filtering the tags in the tag set according to the number of likes; if it is determined according to the first judgment result that the two candidate merging subsets represent the same entity, then merging the two candidate merging subsets into A clustering subset, updating the label set and index of the clustering subset, and using the merged clustering subset as a primary clustering subset; if the two If the candidate merging subsets do not represent the same entity, the distance between the two candidate merging subsets is set to 1; Go back and continue to execute the step A-step C until the distance between the two candidate merging subsets is greater than the lower threshold, then at least two of the primary clustering subsets are used as the obtained at least two Clustering subsets. 5. The method according to claim 3, wherein the obtaining the probability that the initial record pair represents the same entity comprises: calculating the similarity of the initial record pair according to the similarity between the corresponding attributes of the initial record pair; A probability that the pair of initial records represent the same entity is calculated based on a machine learning model. 6. The method according to claim 4, wherein the calculating the distance between each pair of primary cluster subsets in the primary cluster subsets comprises: Select the record pair (r _i , r _j ) with the highest probability representing the same entity from each pair of primary clustering subsets, where r _i ∈ C _i , r _j ∈ C _j , and C _i is the A primary cluster subset in the primary cluster subset, _Cj is another primary cluster subset in each pair of primary cluster subsets; According to the formula Get the distance between each pair of primary clustering subsets; where, maxSimi is the probability that the record pair (r _i , r _j ) represents the same entity, and cosinSimi is the cosine similarity of each pair of primary clustering subsets Spend. 7. The method according to any one of claims 1-6, characterized in that, according to the subset information of the at least two clustering subsets and the feature information of the new record from the at least two At least two relevant cluster subsets most relevant to the new record are obtained from the cluster subsets, including: Establishing an inverted index for the label set information and index information of the at least two clustering subsets obtained by performing hierarchical clustering on the initial records in the database according to the crowdsourcing-based hierarchical clustering method; Searching is performed according to the inverted index and the feature information of the new record, and the at least two relevant cluster subsets most relevant to the new record are obtained from the at least two cluster subsets. 8. The method according to any one of claims 1-6, characterized in that, determining the relationship between the new record and each record in the at least two relevant cluster subsets according to the similarity relationship Candidate record pairs corresponding to the at least two related cluster subsets, including: separately calculating the similarity of the new record to each record in the at least two relevant subsets of clusters; Respectively select a record with the greatest similarity to the new record from each of the related cluster subsets, and respectively form a candidate record pair corresponding to the related cluster subset with the new record; wherein, the The number of relevant cluster subsets is equal to the number of candidate record pairs. 9. The method according to any one of claims 1-6, characterized in that, judging whether at least one of the candidate record pairs represents the same entity through crowdsourcing user labeling; if it is determined that the first candidate record pair represents same entity, then add the new record to the first cluster subset to which the first record belongs, and update the label set of the first cluster subset; if it is determined that all the candidate similar pairs do not represent the same entity , then establish a new cluster subset for the new record, and create a label set for the new cluster subset, including: Sending all the candidate record pairs to the crowdsourcing platform, so that the crowdsourcing platform judges whether the candidate record pairs represent the same entity, and the result of judging whether the candidate record pairs represent the same entity is the second judgment result; receiving the second judgment result returned by the crowdsourcing platform, and determining whether at least one of the candidate record pairs represents the same entity according to the second judgment result; if it is determined according to the second judgment result that the first candidate record pair represents the same entity entity, add the new record to the first cluster subset to which the first record belongs, and update the label set of the first cluster subset; if all the candidate records are determined according to the second judgment result If the pairs do not represent the same entity, a new cluster subset is established for the new record, and a label set is created for the new cluster subset. 10. An entity resolution device based on swarm computing, characterized in that it comprises: A hierarchical clustering module for hierarchically clustering the initial records in the database based on a crowdsourcing hierarchical clustering method to obtain at least two cluster subsets; A detection module, configured to obtain feature information of the new record when it is detected that a new record has been added to the database; The first determination module is configured to obtain at least two most relevant to the new record from the at least two cluster subsets according to the subset information of the at least two cluster subsets and the feature information of the new record. related clustering subsets; wherein, the subset information of the at least two clustering subsets includes: label set information and index information of the clustering subsets; A second determination module, configured to determine candidate record pairs respectively corresponding to the at least two relevant cluster subsets according to the similarity relationship between the new record and each record in the at least two relevant cluster subsets; The division module is used to judge whether at least one of the candidate record pairs represents the same entity through crowdsourcing user labeling; if it is determined that the first candidate record pair represents the same entity, then add the new record to the first record to which the first record belongs. In the clustering subset, and update the label set of the first clustering subset; if it is determined that all the candidate record pairs do not represent the same entity, a new clustering subset is established for the new record, and for all The new cluster subset creates a label set; wherein, the first record and the new record form the first candidate record pair.