CN108959413A - A kind of topical webpage clawing method and Theme Crawler of Content system - Google Patents

Abstract

This application provides a kind of topical webpage clawing method and Theme Crawler of Content system, method includes: to obtain the link not crawled from first including kind of sublink wait crawl in link set；Determine that corresponding first degree of correlation of the corresponding target webpage of link and second degree of correlation obtained, first degree of correlation and second degree of correlation are respectively the degree of correlation of target text content and Object linking and designated key in target webpage；The temperature value of target webpage is determined according to first degree of correlation and second degree of correlation, and stores the content to be presented of target webpage；If the temperature value of target webpage is greater than or equal to preset temperature value, Object linking is put into second wait crawl link set；If first wait crawl in link set there is no the link that had not been obtained, from second wait crawl obtained in link set with the designated key degree of correlation it is highest do not crawl link and continue to crawl.The application makes user that can obtain largely webpage relevant to designated key from network.

Description

A theme webpage crawling method and theme crawler system

technical field

The present invention relates to the technical field of webpage crawling, in particular to a method for crawling theme webpages and a theme crawler system.

Background technique

With the rapid development of the Internet, people have ushered in an era of information explosion, and all kinds of information are flooding all aspects of life. In order to facilitate the acquisition of information, search engines have emerged. People can quickly retrieve information on many web pages through search engines, and search engines have improved the efficiency of people's access to information. At present, commonly used search engines such as Google, Baidu, etc. are all general search engines. This type of search engine tries to obtain all the resources on the Internet. However, people's needs are various. The content of webpages with specified topics can be obtained, but general search engines cannot meet the personalized needs of users.

Contents of the invention

In view of this, the present invention provides a method and a theme crawler system for crawling theme webpages, so that users can obtain webpages related to a specified theme based on the method and system. The technical solution is as follows:

A method for crawling a theme webpage, comprising:

Obtain uncrawled links from the first link set to be crawled, wherein the first link set to be crawled includes pre-acquired seed links;

Determine the first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link, the first correlation degree is the correlation degree between the target link in the target webpage and the specified topic, and the second correlation degree is Relevance between the target text content in the target webpage and the specified topic;

Determine the temperature value of the target webpage according to the first correlation degree and the second correlation degree, and store the content to be displayed of the target webpage, wherein the temperature value can represent the relationship between the target webpage and the specified the relevance of the topic;

If the temperature value of the target webpage is greater than or equal to the first preset temperature value, then put the target link into the second set of links to be crawled;

If there is no unobtained link in the first link set to be crawled, then obtain the uncrawled link with the highest correlation with the specified topic from the second set of links to be crawled, and then execute the The first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link are determined as described above.

Wherein, said determining the first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link includes:

Crawling the target webpage from the Internet according to the obtained link;

Extracting the target text content, the target link, and the anchor text corresponding to the target link from the target webpage;

Determining, based on the anchor text, a degree of correlation between the corresponding target link and the specified topic as the first correlation degree, and determining a degree of correlation between the target text content and the specified topic as the second degree of correlation.

Wherein, the determination of the relevance between the target text content and the specified topic includes:

Segmenting the target text content by using a two-way long-short-term memory conditional random field model to obtain multiple words;

Determine the theme of the target text content through the theme determination module established by the plurality of words and prediction;

Determining the degree of relevance between the subject of the target text content and the specified theme as the degree of relevance between the target text content and the specified theme.

Wherein, the determination of the relevance between the corresponding target link and the specified topic based on the anchor text includes:

converting words in the anchor text into word vectors;

Inputting the word vector into a pre-established topic prediction model to obtain a prediction result output by the topic prediction model, wherein the prediction result is used to indicate the topic of the anchor text, and the topic prediction model is marked with a topic The word vector corresponding to the training anchor text is obtained by training the training sample;

Determining the relevance between the topic of the anchor text and the specified topic as the correlation between the target link corresponding to the anchor text and the specified topic.

Wherein, the process of pre-establishing the topic prediction model includes:

Obtain multiple anchor texts marked with topics to form a training anchor text set;

Each word in the training anchor text in the training anchor text set is converted into a word vector in turn to obtain a word vector set corresponding to the training anchor text, wherein the distance between different word vectors represents its corresponding text the connection between

The word vector set corresponding to the training text is used as an input to train a bidirectional recurrent neural network, and the trained bidirectional recurrent neural network is used as the topic prediction model.

Preferably, the subject webpage crawling method also includes:

In the second set of links to be crawled, links whose relevance degree to the specified topic is less than a preset relevance degree and whose corresponding temperature value is less than a second preset temperature value are deleted.

A theme crawler system, comprising: a link acquisition module, a correlation degree determination module, a temperature value determination module and a link processing module;

The link acquisition module is configured to acquire uncrawled links from the first link set to be crawled, the first link set to be crawled includes pre-acquired seed links;

The correlation degree determining module is used to determine the first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link, and the first correlation degree is the relationship between the target link in the target webpage and the specified topic degree of relevance, the second degree of relevance is the degree of relevance between the target text content in the target webpage and the specified topic;

The temperature value determination module is configured to determine the temperature value of the target webpage according to the first correlation degree and the second correlation degree, and store the content to be displayed of the target webpage, wherein the temperature value can characterizing the relevance of the target webpage to the specified topic;

The link processing module is configured to put the target link into the second set of links to be crawled when the temperature value of the target webpage is greater than or equal to the first preset temperature value;

The link obtaining module is further configured to obtain, from the second set of links to be crawled, the most relevant link to the specified topic when there is no unobtained link in the first set of links to be crawled. The uncrawled link then triggers the correlation determination module to determine the first correlation and the second correlation corresponding to the target webpage corresponding to the obtained link.

Wherein, the correlation determination module includes: a web page crawling sub-module, a data extraction sub-module, a first correlation determination sub-module and a second correlation determination sub-module;

The webpage crawling submodule is used to crawl the target webpage from the network according to the obtained link;

The data extraction submodule is used to extract the target text content, the target link and the anchor text corresponding to the target link from the target webpage;

The first correlation determination submodule is configured to determine the correlation between the corresponding target link and the specified topic based on the anchor text as the first correlation;

The second correlation determination submodule is configured to determine the correlation between the target text content and the specified topic as the second correlation.

Wherein, the first correlation determination submodule includes: a conversion submodule, a prediction submodule and a determination submodule;

The conversion submodule is used to convert the words in the anchor text into word vectors;

The prediction sub-module is configured to input the word vector into a pre-established topic prediction model, and obtain a prediction result output by the topic prediction model, wherein the prediction result is used to indicate the topic of the anchor text, and the The topic prediction model is obtained by training the word vector corresponding to the training anchor text marked with the topic as the training sample;

The determination sub-module is configured to determine the degree of relevance between the subject of the anchor text and the specified subject, as the degree of relevance between the target link corresponding to the anchor text and the specified subject.

Preferably, the theme crawler system also includes: a link deletion module;

The link deletion module is configured to delete links in the second set of links to be crawled whose relevance to the specified topic is less than a preset relevance and whose corresponding temperature value is less than a second preset temperature value.

The above technical scheme has the following beneficial effects:

The theme webpage crawling method and theme crawler system provided by the present invention firstly obtain uncrawled links from the first set of links to be crawled, and then respectively determine the target text content in the target webpage corresponding to the acquired links and the content of the specified theme The degree of relevance and the degree of relevance between the target link and the specified topic, and then determine the temperature value of the target webpage according to the determined degree of relevance, and store the content to be displayed on the marked webpage, if the temperature value of the target webpage is greater than or equal to the first preset temperature value, put the target link into the second set of links to be crawled, and if there is no unobtained link in the first set of links to be crawled, then obtain the link related to the specified topic from the second set of links to be crawled Continue to crawl the uncrawled link with the highest degree. The topic web page crawling method and topic crawler system provided by the present invention enable users to obtain a large number of web pages related to specified topics from the Internet, and the user experience is better.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

Fig. 1 is the subject web page crawling method that the embodiment of the present invention provides;

FIG. 2 is a schematic flow diagram of the implementation process of determining the first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link in the subject webpage crawling method provided by the embodiment of the present invention;

FIG. 3 is a schematic flow diagram of an implementation process of determining the correlation between a corresponding target link and a specified topic based on an anchor text in a method for crawling a topical webpage provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of a word vector provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of a topic prediction model provided by an embodiment of the present invention;

6 is a schematic diagram of a vector P with a dimension of 2m obtained by pooling an nx2m matrix Z provided by an embodiment of the present invention;

7 is a schematic diagram of word segmentation of a sentence using the two-way long-short memory conditional random field model provided by the embodiment of the present invention;

Fig. 8 is a schematic structural diagram of a theme crawler system provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

An embodiment of the present invention provides a method for crawling a subject webpage, which is used to crawl webpages related to a specified topic. Please refer to FIG. 1 , which shows a schematic flow diagram of the method for crawling a webpage, which may include:

Step S101: Obtain uncrawled links from the first set of links to be crawled.

Wherein, the first set of links to be crawled includes pre-acquired seed links.

The seed link is the starting position for web crawling, and a good seed link can quickly find web pages related to the topic.

Step S102: Determine the first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link. The first correlation degree is the correlation degree between the target link in the target webpage and the specified topic, and the second correlation degree is the correlation degree between the target link in the target webpage. The relevance of the target text content to the specified topic.

Wherein, the target text content in the target web page refers to the content of the text contained in the target web page, such as an article, a piece of news, etc., and the target link in the target web page refers to the URL existing in the target web page, through which URLs can be jumped to the corresponding page.

Step S103: Determine the temperature value of the target webpage according to the first correlation degree and the second correlation degree, and store the content to be displayed on the target webpage.

Wherein, the temperature value of the target webpage can represent the degree of relevance between the target webpage and the specified topic. The higher the temperature value of the target web page, the higher the relevance of the target web page to the specified topic, and the lower the temperature value of the target web page, the lower the relevance of the target web page to the specified topic.

Wherein, the content to be displayed of the target webpage may include the title of the target webpage, the text content of the target webpage, and the link of the target webpage.

Considering that the target web page contains a lot of HTML tags and other codes, however, this information is meaningless for display, because users do not pay attention to this information. In addition, storing such information that is meaningless for display will waste storage space , therefore, when storing the information of the target webpage, it is only necessary to store the title of the target webpage, the text content of the target webpage, and the link of the target webpage.

The title of a possible landing page, the text content of the landing page, and the link to the landing page can be stored in a file in the following format:

<title>The title of the page</title>

<body>Text content of the page</body>

<url>link to web page</url>

When storing in the database, 3 fields can be set: title, body, url.

Step S104: If the temperature value of the target web page is greater than or equal to the first preset temperature value, put the target link into the second set of links to be crawled.

It should be noted that if the temperature value of the target webpage is greater than or equal to the first preset temperature value, it indicates that the target webpage has a high degree of relevance to the specified topic, and accordingly, the target link contained in the target webpage may have a high degree of relevance to the specified topic. is also higher, therefore, put the target link contained in the target webpage into the second set of links to be crawled.

Wherein, the links in the second set of links to be crawled are non-seed links, but links extracted from webpages obtained based on seed links. It should be noted that webpages obtained based on seed links may include direct or indirect fetched webpage.

It should be noted that if the temperature value of the target webpage is less than the first preset temperature value, it indicates that the target webpage is less relevant to the specified topic, and correspondingly, the links contained in the target webpage are also more relevant to the specified topic. Small, so stop crawling the links in the target webpage.

Step S104: If there is no unacquired link in the first link set to be crawled, obtain an uncrawled link most relevant to the specified topic from the second set of links to be crawled, and then perform step S102.

In a possible implementation manner, the process of obtaining the uncrawled link with the highest correlation with the specified topic from the second set of links to be crawled may include: based on the correlation between the uncrawled link and the specified topic Second, the links in the link set to be crawled are sorted, and the link with the highest correlation with the specified topic is determined by sorting, and the link with the highest correlation with the specified topic is obtained from the second link set to be crawled.

It should be noted that, if there are unacquired links in the first link set to be crawled, the uncrawled links are preferentially obtained in the first link set to be crawled, and step S102 and subsequent steps are executed, and the first link to be crawled is After all the links in the link collection are crawled, the uncrawled links that are most relevant to the specified topic are obtained from the second link collection to be crawled for crawling. When there is no unobtained link in the second set of links to be crawled, the webpage crawling process based on the specified topic is ended.

The method for crawling Korean-oriented webpages provided by the embodiments of the present invention first obtains uncrawled links from the first set of links to be crawled, and then respectively determines the target text content and the content of the specified topic in the target webpage corresponding to the obtained links. The degree of relevance and the degree of relevance between the target link and the specified topic, and then determine the temperature value of the target webpage according to the determined degree of relevance, and store the content to be displayed on the marked webpage, if the temperature value of the target webpage is greater than or equal to the first preset temperature value, put the target link into the second set of links to be crawled, and if there is no unobtained link in the first set of links to be crawled, then obtain the link related to the specified topic from the second set of links to be crawled Continue to crawl the uncrawled link with the highest degree. The webpage crawling method provided by the embodiment of the present invention enables the user to obtain a large number of webpages related to a specified topic from the Internet, and the user experience is better.

It should be noted that the method for crawling themed webpages provided in the above embodiments is applicable to crawling themed webpages in multiple languages, such as Chinese, Korean, Japanese, and so on.

As the main language of the Korean ethnic group in my country, Korean is also the official language of South Korea and North Korea. It has great research significance, which is embodied in the following aspects: First, the Korean people in our country use Korean as their mother tongue, and they are searching for information. They may use Korean from time to time, so it is necessary to provide them with a search engine for specific content to facilitate their lives; secondly, South Korea and North Korea are both in the vicinity of our country, and have many connections with our country since ancient times. Today, when the international community is constantly changing, people In addition to caring about the development of your own country, you should also pay attention to the development and situation of neighboring countries. In view of this, when this embodiment introduces the steps in the method provided by the above embodiments of the present invention, Korean language is used as an example for description. It should be noted that, when the above method is for Korean, multiple (for example, 163) links can be selected in advance from the NAVER website (https://www.naver.com/) as seed links.

Step S102 in the above embodiment: determining the first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link will be introduced below.

Please refer to FIG. 2 , which shows a schematic flowchart of the implementation process of determining the first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link, which may include:

Step S201: Crawling the corresponding target webpage from the Internet according to the obtained link.

Step S202: Extracting the target text content, the target link and the anchor text corresponding to the target link from the target webpage.

Among them, the anchor text refers to the text content corresponding to the link URL in the web page, which is generally included in an a tag and organized as follows:

<a href="URL">Anchor Text</a>

The anchor text corresponding to the target link contains a brief summary of the target webpage, which can be used to predict the subject of the target link. For the process of predicting the subject of the corresponding target link based on the anchor text, please refer to the subsequent description.

Step S203: Based on the anchor text, determine the degree of relevance between the corresponding target link and the specified topic as the first degree of relevance, and determine the degree of relevance between the target text content and the designated topic as the second degree of relevance.

The process of determining the correlation between the corresponding target link and the specified topic based on the anchor text will be described below. Please refer to FIG. 3 , which shows a schematic flowchart of an implementation process for determining the relevance between the corresponding target link and the specified topic based on the anchor text, which may include:

Step S301: convert the words in the anchor text into word vectors.

It should be noted that for the word vectorization method, there is a way to vectorize words using one-hot encoding in the prior art, that is, to vectorize words with 0 and 1, but the vector obtained by this vectorization method Sparse and relatively high dimensionality, in view of this, the embodiment of the present invention can use the word-embedding word embedding method to vectorize the representation of the word, and use the word2vec model to extend the word vector method to the representation of the word, and obtain a certain dimension Words represent vectors.

The word2vec model uses the relationship between the word and the context to generate a representation of the word, so that the obtained representation can contain certain semantic information, and the relationship between the word and the context is used to train the representation of the word in the model. The Korean article information in the wiki can be used during training, and the obtained vectorized representation is similar to Figure 4.

Step S302: input the word vector into the pre-established topic prediction model, and obtain the prediction result output by the topic prediction model.

The prediction result is used to indicate the topic of the anchor text, and the topic prediction model is obtained by training the word vector corresponding to the training anchor text labeled with the topic as a training sample.

The process of pre-establishing the topic prediction model may include: obtaining multiple anchor texts labeled with topics to form a training anchor text set; converting each word in the training anchor text in the training anchor text set into a word vector in turn, and obtaining and training The set of word vectors corresponding to the text, where the distance between different word vectors represents the relevance between the corresponding words; the set of word vectors corresponding to the training text is used as input to train the two-way cyclic neural network, and the two-way cyclic neural network obtained from the training is Neural Networks as Topic Prediction Models.

Please refer to Figure 5, which shows a schematic diagram of a topic prediction model. In this embodiment, the bidirectional cyclic neural network can be a bidirectional LSTM network. Direction output h _f and h _b are concatenated to obtain h _i , if the dimension of h _f is the number m of hidden units, then the dimension of h _i is 2m:

h _i =[h _f , h _b ] (1)

For an anchor text with a length of n, the output obtained after calculation and splicing by the bidirectional LSTM network becomes an nx2m matrix Z:

Z＝{h ₀ ,h ₁ ,h ₂ ,...,h _n-1 ,h _n-1 } (2)

For the obtained matrix Z, use the pooling method in the convolutional network to extract and compress the features in Z, and obtain a vector P with a dimension of 2m, as shown in Figure 6.

The pooling method processes a part of the matrix through specific operations, compresses and transforms the matrix, and extracts useful features from it for subsequent analysis. In one possible implementation, each row in the matrix can be pooled using the max pooling method, namely:

k _x =max(h _ij )0≤j<n (3)

After pooling, there is a fully connected network with h units. Through network calculation, the result output is converted into the probability value of the corresponding topic.

Step S303: Determine the correlation between the topic of the anchor text and the specified topic, and use it as the correlation between the target link corresponding to the anchor text and the specified topic.

This embodiment uses all the content in the anchor text when predicting the subject of the anchor text, eliminating the interference of other information, and using the word vector representation method, which can contain more semantic information, which is more conducive to the anchor text The topic of the text is predicted. The cyclic neural network used in the topic prediction model can better simulate the character sequence information, and since the cyclic neural network used is a bidirectional cyclic neural network, it can obtain information in both positive and negative directions of an anchor text, More information is used to judge the topic.

The following describes the process of determining the relevance of the target text to the specified topic.

In a possible implementation, the process of determining the relevance between the target text content and the specified topic may include: using a two-way long-short memory conditional random field model to segment the target text content to obtain multiple words; The established topic judgment model judges the topic of the target text content; determines the correlation degree between the topic of the target text content and the specified topic as the correlation degree between the target text content and the specified topic.

After obtaining the above-mentioned representation method for characters, the word vector can be used for word segmentation of Korean. In this embodiment, when the target text content is segmented, the word segmentation process is regarded as a process of forming words by characters, and each word in the word is assigned a different label form: B, indicating that the word is at the beginning of a word; M, indicates that the character is inside a word; S, indicates that the character is a separate word; E, indicates that the character is at the end of the word. This embodiment uses the method that the long-short memory unit (LSTM) in the recurrent neural network combines with the conditional random field (CRF), constitutes a two-way long-short memory conditional random field model to complete the word segmentation of Korean, please refer to Fig. 7, showing A schematic diagram of word segmentation for a Korean sentence using the two-way long-short memory conditional random field model is shown. It can be seen from Figure 7 that a sentence or character sequence is used as input, and the word vector is obtained by searching in the word vector table, and then the word The vector is input into the bidirectional LSTM network. After the output of the bidirectional LSTM network is obtained, CRF is used to decode the relationship between the word and the label in the model to obtain the final word segmentation result.

In order to determine the subject of the target text content, in addition to word segmentation, a subject determination model must also be constructed. In a possible implementation, support vector machine (support vector machine, SVM) can be used to construct a topic determination model, the feature extraction method can be used to obtain the feature of the topic, and the feature vector can be obtained by using the TF-IDF method to weight the feature, and the training A topic judgment model is obtained. When judging the topic of the target text content, the topic judgment model is directly used to judge the topic of the target text content.

In another embodiment of the present invention, the step S103 in the foregoing embodiments: determining the temperature value of the target web page according to the first correlation degree and the second correlation degree is introduced.

It should be noted that not all relevant subject pages on the Internet are directly connected, but indirectly linked together through some other web pages, and these links form a "tunnel". Tunneling is to make topic crawlers try to cross these links and discover more topical web pages. Tunneling is a future-oriented method for topic crawlers.

In this embodiment, different temperature values are calculated for each webpage, and the temperature value is dynamically adjusted using Newton's law of cooling during the webpage crawling process, so that the subject crawler has a certain ability of tunneling.

In this embodiment, the temperature value of the webpage is calculated by the correlation between the text content in the webpage and the specified topic and the correlation between the links in the webpage and the specified topic. Specifically, the temperature value T of the _webpage is calculated according to the formula (4) calculate:

T _i-1 in formula (4) represents the temperature of the parent webpage, δ represents the decay rate of the temperature, t _i represents the temperature of the webpage itself, and the temperature value of the webpage is described as the temperature transmitted from the parent webpage and the temperature of the current webpage itself. combined. It should be noted that it can be seen from formula (4) that the calculation of the temperature value of a webpage is an iterative process, and the initial parent webpage is the webpage corresponding to the seed link, and the temperature of the webpage can be set to a fixed temperature value.

The original temperature t _i of the webpage itself is calculated according to formula (5):

w(content) in the above formula (5) represents the correlation degree between the text content of the web page and the specified topic, and w(l _k ) represents the correlation degree between the links in the web page and the specified topic.

The temperature decay rate δ is calculated according to formula (6):

δ＝e ^-u*τ (6)

u in the above formula (6) represents the cooling coefficient, which can be set as a fixed value, and τ represents the time interval, and can also be set as the current depth value of the web page.

During tunnel traversal, due to the different temperature values of web pages, the crawling capabilities of different web pages are different. The links corresponding to web pages with high temperatures may be more relevant to the specified topic, and there are more opportunities to collect more web page information. , the opposite is true for low temperature. In order to allow the theme crawler to stop continuing to crawl the links, when the temperature of the webpage is lower than the first preset temperature value, the link corresponding to the webpage is given up and continues to crawl.

In order to improve the efficiency of web page crawling and avoid crawling links with low relevance to the specified topic, the method provided by the embodiment of the present invention may further include: in the second set of links to be crawled, the links with a specified topic whose relevance is less than the preset The correlation degree is set, and links whose corresponding temperature values are less than the second preset temperature value are deleted.

It should be noted that the temperature value corresponding to a link refers to the temperature value of the web page containing the link, that is, when a web page contains multiple links, the temperature values corresponding to these links are the same, and all of them contain these links. The temperature value of the web page.

After crawling all the links in the first set of links to be crawled and the second set of links to be crawled, display the stored content to be displayed of each target webpage, specifically, according to the target text content in the target webpage and the specified Display the content to be displayed on the target webpage in descending order of the relevance of the topic, that is, among the webpage content displayed to the user, the webpage whose text content is highly relevant to the specified topic ranks first content.

The embodiment of the present invention also provides a theme crawler system, please refer to Figure 8, which shows a schematic structural diagram of the system, which may include: a link acquisition module 801, a correlation degree determination module 802, a temperature value determination module 803 and a link processing module 804.

A link acquisition module 801, configured to acquire uncrawled links from the first set of links to be crawled, the first set of links to be crawled includes pre-acquired seed links;

The relevance determination module 802 is configured to determine the first relevance and the second relevance corresponding to the target webpage corresponding to the obtained link.

Wherein, the first correlation degree is the correlation degree between the target link in the target webpage and the designated topic, and the second correlation degree is the correlation degree between the target text content in the target webpage and the designated topic.

The temperature value determination module 803 is configured to determine the temperature value of the target webpage according to the first correlation degree and the second correlation degree, and store the content to be displayed of the target webpage.

Wherein, the temperature value can represent the degree of relevance between the target webpage and the specified topic.

The link processing module 804 is configured to put the target link into the second set of links to be crawled when the temperature value of the target web page is greater than or equal to the first preset temperature value.

The link obtaining module 801 is further configured to obtain, from the second set of links to be crawled, an unacquired link with the highest correlation with the specified topic when there is no unacquired link in the first set of links to be crawled. The crawled link then triggers the relevance determination module 802 to determine the first relevance and the second relevance corresponding to the target webpage corresponding to the obtained link.

The theme crawler system provided by the embodiment of the present invention first obtains uncrawled links from the first set of links to be crawled, and then respectively determines the relevance and target text content of the target text content in the target webpage corresponding to the obtained links and the specified topic. The degree of relevance between the link and the specified topic, and then determine the temperature value of the target webpage according to the determined degree of relevance, and store the content to be displayed on the marked webpage. If the temperature value of the target webpage is greater than or equal to the first preset temperature value, the The target link is put into the second link set to be crawled, and when there is no unobtained link in the first link set to be crawled, the unobtained link with the highest correlation with the specified topic is obtained from the second link set to be crawled. The crawled link continues to crawl. The topic crawler system provided by the embodiment of the present invention enables users to obtain a large number of web pages related to a specified topic from the Internet, and the user experience is better.

In the topic crawler system provided in the above embodiments, the relevance determination module 802 may include: a webpage crawling submodule, a data extraction submodule, a first relevance determination submodule and a second relevance determination submodule.

The webpage crawling submodule is configured to crawl the target webpage from the Internet according to the obtained link.

The data extraction submodule is configured to extract the target text content, the target link, and the anchor text corresponding to the target link from the target webpage.

The first correlation determination submodule is configured to determine, based on the anchor text, the correlation between the corresponding target link and the specified topic as the first correlation.

The second correlation determination submodule is configured to determine the correlation between the target text content and the specified topic as the second correlation.

Further, the first correlation determination submodule may include: a conversion submodule, a prediction submodule and a determination submodule.

The conversion submodule is used to convert the words in the anchor text into word vectors.

The prediction sub-module is configured to input the word vector into a pre-established topic prediction model, and obtain a prediction result output by the topic prediction model, wherein the prediction result is used to indicate the topic of the anchor text, and the The topic prediction model is obtained by training the word vectors corresponding to the training anchor text marked with topics as training samples.

The determination sub-module is configured to determine the degree of relevance between the subject of the anchor text and the specified subject, as the degree of relevance between the target link corresponding to the anchor text and the specified subject.

The topic crawler system provided by the above embodiments may further include: a link deletion module.

The link deletion module is configured to delete links in the second set of links to be crawled whose relevance to the specified topic is less than a preset relevance and whose corresponding temperature value is less than a second preset temperature value.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other.

In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and equipment may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes. .

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A subject webpage crawling method is characterized in that, comprising: Obtain uncrawled links from the first link set to be crawled, wherein the first link set to be crawled includes pre-acquired seed links; Determine the first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link, the first correlation degree is the correlation degree between the target link in the target webpage and the specified topic, and the second correlation degree is Relevance between the target text content in the target webpage and the specified topic; Determine the temperature value of the target webpage according to the first correlation degree and the second correlation degree, and store the content to be displayed of the target webpage, wherein the temperature value can represent the relationship between the target webpage and the specified the relevance of the topic; If the temperature value of the target webpage is greater than or equal to the first preset temperature value, then put the target link into the second set of links to be crawled; If there is no unobtained link in the first link set to be crawled, then obtain the uncrawled link with the highest correlation with the specified topic from the second set of links to be crawled, and then execute the The first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link are determined as described above. 2. The subject web page crawling method according to claim 1, wherein said determination of the first degree of relevance and the second degree of relevance corresponding to the target web page corresponding to the obtained link comprises: Crawling the target webpage from the Internet according to the obtained link; Extracting the target text content, the target link, and the anchor text corresponding to the target link from the target webpage; Determining, based on the anchor text, a degree of correlation between the corresponding target link and the specified topic as the first correlation degree, and determining a degree of correlation between the target text content and the specified topic as the second degree of correlation. 3. The subject web page crawling method according to claim 2, wherein said determination of the degree of relevance between said target text content and a designated subject comprises: Segmenting the target text content by using a two-way long-short-term memory conditional random field model to obtain multiple words; Determine the theme of the target text content through the theme determination module established by the plurality of words and prediction; Determining the degree of relevance between the subject of the target text content and the specified theme as the degree of relevance between the target text content and the specified theme. 4. The subject webpage crawling method according to claim 2, wherein said determining the degree of relevance between the corresponding target link and the specified subject based on said anchor text comprises: converting words in the anchor text into word vectors; Inputting the word vector into a pre-established topic prediction model to obtain a prediction result output by the topic prediction model, wherein the prediction result is used to indicate the topic of the anchor text, and the topic prediction model is marked with a topic The word vector corresponding to the training anchor text is obtained by training the training sample; Determining the relevance between the topic of the anchor text and the specified topic as the correlation between the target link corresponding to the anchor text and the specified topic. 5. The subject web page crawling method according to claim 3, wherein the process of setting up the subject prediction model in advance comprises: Obtain multiple anchor texts marked with topics to form a training anchor text set; Each word in the training anchor text in the training anchor text set is converted into a word vector in turn to obtain a word vector set corresponding to the training anchor text, wherein the distance between different word vectors represents its corresponding text the connection between The word vector set corresponding to the training text is used as an input to train a bidirectional recurrent neural network, and the trained bidirectional recurrent neural network is used as the topic prediction model. 6. The subject web page crawling method according to claim 1, further comprising: In the second set of links to be crawled, links whose relevance degree to the specified topic is less than a preset relevance degree and whose corresponding temperature value is less than a second preset temperature value are deleted. 7. A theme crawler system, comprising: a link acquisition module, a correlation degree determination module, a temperature value determination module and a link processing module; The link acquisition module is configured to acquire uncrawled links from the first link set to be crawled, the first link set to be crawled includes pre-acquired seed links; The correlation degree determining module is used to determine the first correlation degree and the second correlation degree corresponding to the target webpage corresponding to the obtained link, and the first correlation degree is the relationship between the target link in the target webpage and the specified topic degree of relevance, the second degree of relevance is the degree of relevance between the target text content in the target webpage and the specified topic; The temperature value determination module is configured to determine the temperature value of the target webpage according to the first correlation degree and the second correlation degree, and store the content to be displayed of the target webpage, wherein the temperature value can characterizing the relevance of the target webpage to the specified topic; The link processing module is configured to put the target link into the second set of links to be crawled when the temperature value of the target webpage is greater than or equal to a first preset temperature value; The link obtaining module is further configured to obtain, from the second set of links to be crawled, the most relevant link to the specified topic when there is no unobtained link in the first set of links to be crawled. The uncrawled link then triggers the correlation determination module to determine the first correlation and the second correlation corresponding to the target webpage corresponding to the obtained link. 8. theme crawler system according to claim 7, is characterized in that, described correlation determination module comprises: web page crawling sub-module, data extraction sub-module, first correlation determination sub-module and second correlation determination sub-module module; The webpage crawling submodule is used to crawl the target webpage from the network according to the obtained link; The data extraction submodule is used to extract the target text content, the target link and the anchor text corresponding to the target link from the target webpage; The first correlation determination submodule is configured to determine the correlation between the corresponding target link and the specified topic based on the anchor text as the first correlation; The second correlation determination submodule is configured to determine the correlation between the target text content and the specified topic as the second correlation. 9. The theme crawler system according to claim 8, characterized in that, said first correlation determination submodule comprises: conversion submodule, prediction submodule and determination submodule; The conversion submodule is used to convert the words in the anchor text into word vectors; The prediction sub-module is configured to input the word vector into a pre-established topic prediction model, and obtain a prediction result output by the topic prediction model, wherein the prediction result is used to indicate the topic of the anchor text, and the The topic prediction model is obtained by training the word vector corresponding to the training anchor text marked with the topic as the training sample; The determination sub-module is configured to determine the degree of relevance between the subject of the anchor text and the specified subject, as the degree of relevance between the target link corresponding to the anchor text and the specified subject. 10. The subject crawler system according to claim 7, further comprising: a link deletion module; The link deletion module is configured to delete links in the second set of links to be crawled whose relevance to the specified topic is less than a preset relevance and whose corresponding temperature value is less than a second preset temperature value.