CN102622451A - System for automatically generating television program labels - Google Patents

Abstract

The present invention provides a system for automatically generating TV program labels, including grabbing pages related to the program, pruning and filtering the pages, and obtaining a program information acquisition module describing the main content of the program information; summarizing the main content, and extracting key words from the information keywords The extraction module; the knowledge base module that establishes the network relationship between the entries and expands the acquired keywords; the keyword expansion module that expands the keywords to obtain a larger entry set by using the network provided by the knowledge base module; will get The associated term set of the keyword is processed, the noise is filtered out, the score is calculated, and the label generation module of the program label set is generated. The invention fills the blank of the system for automatically generating TV program tags, and the introduction of the knowledge base also makes the system not restricted by web pages, has better expansibility, and has better discoverability for tags. The knowledge base can be established offline, and the label generation algorithm is simple, so the system efficiency is also very high.

Description

Automatic Generation System of TV Program Labels

technical field

The invention relates to a system in the field of computer application technology, in particular to a system for automatically generating TV program labels.

Background technique

For a long time, how to help people make better choices has always been a meaningful and interesting question. People make choices based on certain information. Combining the collected information with personal ideas and hobbies creates choice behavior. However, access to information is not simple. In the past when the Internet was not yet developed and information exchange was inconvenient, the lack of information and the difficulty of comparison became obstacles for people to make choices. In the information age, information acquisition can be completed with just a click of the mouse, but this brings another problem, the flood of information. In the face of massive amounts of information, it will take a lot of time for people to just identify and screen the information, which also creates barriers to choice. In order to solve this problem, automatic label generation system came into being. By extracting the main body of the information, summarizing the content, and analyzing keywords, a tag set corresponding to the information is generated. Using the label set, people can quickly grasp the general idea of information, and at the same time provide a basis for information classification, which can help people make choices.

At present, there are many researches on automatic tag generation system, but they mainly focus on text processing, that is, automatically generate tags related to a document. Jialie Shen[1] studied the automatic generation of music tags. The method used is mainly to extract the features of the audio, and then use the manually tagged music as the training material to generate a classifier through machine learning, and use this classifier to add tags to the music. . Stefan Siersdorfer[2] proposed a supplementary scheme for video tags, using existing video comparison technology to merge existing tags of similar videos, but this is not the real automatic generation of tags. Therefore, at this stage, adding tags to video mainly depends on manual processing, and the research on the automatic generation system of TV program tags is still a vacancy.

[1] Jialie Shen, Meng Wang, Shuicheng Yan, HweeHwa Pang, Xiansheng HuaEffective Music Tagging through Advanced Statistical Modeling SIGIR 2010;

[2] Stefan Siersdorfer, Jose San Pedro, Mark Sanderson Automatic Video Tagging using Content Redundancy SIGIR 2009.

Contents of the invention

Aiming at the above-mentioned deficiencies in the prior art, the present invention provides a system for automatically generating TV program labels. It only needs to provide the system with the name of the TV program, and the system can automatically obtain information related to the program from the Internet. The obtained information is summarized and expanded, and the system will return a tag set related to the program.

The present invention is achieved through the following technical solutions.

An automatic generation system for TV program labels, comprising a sequentially connected program information acquisition module, information keyword extraction module, keyword expansion module and label generation module, and also includes a knowledge base module connected with the keyword expansion module, wherein:

- The program information acquisition module is used to grab the pages related to the program from the Internet, and obtain the main content describing the program information by pruning and filtering the pages;

- information keyword extraction module, used to summarize the main content obtained by the program information acquisition module, and extract keywords from the main content;

- a knowledge base module, used to establish the network relationship between entries, so as to expand the acquired keywords;

-keyword extension module, used to utilize the network provided by the knowledge base module to expand the keywords obtained by the information keyword extraction module to obtain a larger entry set;

- A tag generation module, used to process the associated entry sets of all keywords obtained, filter out noise, calculate scores, and finally generate a tag set for the program.

Described program information acquisition module comprises HTML parser, receives the target television program set that needs to generate label, under the assistance of search engine, obtains network page for each program, and described page obtains main content through the processing of HTML parser, The main body content is passed to the information keyword extraction module for further processing.

The information keyword extraction module includes a word segmentation and part-of-speech tagger. After obtaining the main content describing each program information, the content is divided by the word segmentation and part-of-speech tagging device, and only the noun part-of-speech words are reserved.

The words of the noun part of speech identify keywords through statistical methods.

Described statistical method comprises the following steps:

In the first step, for a specific program, the words are divided into two groups, one group comes from web pages related to the program, and one group comes from other web pages in the program collection;

The second step is to calculate the word frequency for these two groups of words, and count the mean and standard deviation. In this way, each word uses 4 statistics to describe its characteristics. The mean and standard deviation of the word frequency of , and the mean and standard deviation of the word frequency of this word on pages not related to the program;

In the third step, according to the relationship among the four statistics, the keywords that can best represent the characteristics of the program are identified.

The knowledge base module uses Baidu Encyclopedia as a data source and stores it in the form of a graph.

The organization method of Baidu Encyclopedia includes the following steps:

In the first step, for each entry, there is a page describing the entry. In addition to the plain text, the page will also refer to other existing entries in Baidu Encyclopedia;

In the second step, in the graph of the knowledge base, there will be a directed edge between each such described entry and the referenced entry, and the PageRank algorithm is applied to this graph to obtain the importance of each entry;

In the third step, the weight of entries and the mutual reference relationship between entries constitute the entire knowledge base.

The keyword expansion module finds other entries that have a path with it in the figure of the knowledge base module for the keywords obtained by each information keyword extraction module, and according to the importance of the entry itself and the entry and keyword The distance of the term is calculated to calculate the weight of the entry.

The label generating module combines all the associated entries of the obtained keywords together, and when an entry is associated with multiple keywords at the same time, the weights of this entry in various keywords are added. All the entries are sorted according to the sum of the weights, and the previous ones are returned as needed, so as to obtain the label set describing the characteristics of the program.

When the present invention works, the target television program collection that needs to generate tags is first provided for the system. With the assistance of the search engine, the program information acquisition module acquires a certain amount of web pages for each program. These pages are processed by the HTML parser in the module to obtain the main content, and these main content will be passed to the information keyword extraction module for further processing. After the information keyword extraction module obtains the main content describing each program information, it divides the content through the word segmentation and part-of-speech tagger in the module, and only retains the noun part-of-speech words. These words will statistically identify keywords. The statistical method is as follows: for a specific program, the words are divided into two groups. One set comes from web pages related to the show, and one set comes from other web pages in the show collection. The word frequency is calculated for both groups of words, and the mean and standard deviation are calculated. In this way, each word is characterized by 4 statistics. They are the mean and standard deviation of the word frequency of the word on the page related to the program, and the mean and standard deviation of the word frequency of the word on the page not related to the program. According to the relationship among the four statistics, the keywords that can best represent the characteristics of the program can be identified. The keywords extracted through the web pages can already reflect the characteristics of the program to a certain extent, but the disadvantage is that the range of the obtained keywords is limited, that is, they must appear on the web pages. Aiming at this limitation, a very important point of the present invention is to introduce a knowledge base module. The knowledge base module uses Baidu Encyclopedia as the data source and stores it in the form of a graph. Baidu Encyclopedia is organized in such a way that for each entry, there is a page describing the entry. In addition to the plain text, the page will also refer to other existing entries in Baidu Encyclopedia. In the knowledge base graph, there will be a directed edge between each such described term and the referenced term. Applying the PageRank algorithm to this graph, we get the importance of each term. The weight of entries and the mutual reference relationship between entries constitute the entire knowledge base. In this way, the task of the keyword expansion module is very simple. For each keyword obtained by the information keyword extraction module, other entries that have a path with it can be found in the graph of the knowledge base. According to the importance of the entry itself and the distance between the entry and the keyword to calculate the weight of the entry. The tag generation module is the last link of the system. In the information keyword extraction module, we get the keyword set that can reflect the characteristics of the program. In the keyword expansion module, we get the entry set associated with each keyword, and each Each entry has weight. The tag generation module is responsible for integrating the two parts of the results, that is, merging all associated entries of the obtained keywords together. When an entry is associated with multiple keywords at the same time, the weights of this entry in various keywords are added. Sort all the entries according to the sum of weights, and return the previous ones as needed, and we get the label set describing the characteristics of the program.

Compared with the prior art, the present invention fills the gap in the automatic generation of TV program labeling system, and the introduction of the knowledge base also makes the system not restricted by web pages, has better expansibility, and has better discoverability for labels . The knowledge base can be established offline, and the label generation algorithm is simple, so the system efficiency is also very high.

Description of drawings

Fig. 1 shows a system block diagram of the present invention;

Fig. 2 shows the implementation details of the program information acquisition module of the present invention;

Fig. 3 shows the generation details of the entry list in the information keyword extraction module of the present invention;

Fig. 4 shows the details of keyword generation in the information keyword extraction module of the present invention.

Detailed ways

The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. This embodiment is implemented on the premise of the technical solution of the invention, and detailed implementation methods and specific operating procedures are provided, but the protection scope of the present invention is not limited to the following the embodiment.

The task of this embodiment is to automatically generate tags for a group of TV programs, which are program 1, program 2, program 3, program 4, program 5, program 6, program 7, program 8, program 9, and program 10.

As shown in Figure 1, this embodiment includes 5 modules: program information acquisition module, information keyword extraction module, knowledge base module, keyword expansion module, label generation module, wherein, program information acquisition module, information keyword extraction module The keyword expansion module and the tag generation module are connected in sequence, and the knowledge base module is connected with the keyword expansion module. The program information acquisition module is responsible for grabbing pages related to these 10 programs from the Internet, and obtaining the main content describing the program information by pruning and filtering the pages. The information keyword extraction module is responsible for summarizing the main content obtained by the program information acquisition module, and extracting keywords from the main content. The knowledge base module is responsible for establishing the network relationship between entries, so as to expand the acquired keywords. The keyword expansion module is responsible for using the network provided by the knowledge base module to expand the keywords obtained by the information keyword extraction module to obtain a larger entry set. The label generation module is responsible for processing the entry set, filtering out noise, calculating scores, and finally generating the label set of the program.

As shown in Figure 2, the program information acquisition module includes an HTML parser to receive the set of target TV programs that need to generate tags, and with the assistance of a search engine, obtain a web page for each program, and the pages are processed by the HTML parser. The main content is obtained, and the main content is passed to the information keyword extraction module for further processing. Specifically, the program information acquisition module uses a search engine to obtain 10 pages related to the target program, that is, HTML files. By removing useless tags such as advertisements, pictures, titles, scripts, etc. in the obtained HTML files, we have obtained 10 files describing program information.

As shown in Figure 3, the information keyword extraction module includes word segmentation and part-of-speech tagging. After obtaining the main content describing each program information, the content is divided by word segmentation and part-of-speech tagging, and only noun words are reserved. Specifically, the documents returned by the program information acquisition module will first be processed by word segmentation and part-of-speech tagging through the information keyword extraction module, and only the noun part-of-speech words will be retained, so that each document is converted into a word set. The 10 documents corresponding to a program will have repeated words, so the words in the 10 documents will be hashed, and the word frequency of each word in each document will be counted. Finally, we will get a list of entries for each program, and each item in the list is a data structure, including the text content of the entry and the word frequency of the entry in 10 documents.

It should be noted that the words of the noun part of speech identify keywords through statistical methods.

The statistical method includes the following steps:

In the first step, for a specific program, the words are divided into two groups, one group comes from web pages related to the program, and one group comes from other web pages in the program collection;

The second step is to calculate the word frequency for these two groups of words, and count the mean and standard deviation. In this way, each word uses 4 statistics to describe its characteristics. The mean and standard deviation of the word frequency of , and the mean and standard deviation of the word frequency of this word on pages not related to the program;

In the third step, according to the relationship among the four statistics, the keywords that can best represent the characteristics of the program are identified.

As shown in Fig. 4, the obtained word list is further processed to obtain the final keyword list. Here, for each word in the target program entry list, four statistics are calculated, namely: the word frequency mean and standard deviation of the word in the target program, the word frequency mean and standard deviation of the word in other programs . After obtaining the 4 statistics, first classify the words according to the following rules:

The first category: the mean and standard deviation of word frequency in other programs are both 0;

The second category: the word frequency mean and standard deviation are both 0 in other programs, and the mean value in the target program is larger than the mean value of other programs and the standard deviation is smaller than that of other programs;

The third category: the situation other than the first and second categories.

Each category then calculates the score according to the following rules:

The first category: the mean in the target program divided by the standard deviation;

The second type: the mean value in the target program multiplied by the standard deviation of other programs divided by the standard deviation in the target program and then divided by the mean value of other programs.

The third category: directly set to 0.

Next, the words are sorted, the first category is better than the second category, the second category is better than the third category, and the same category is re-sorted according to the score, and finally the top 20 words are output to form a keyword list.

The knowledge base module uses Baidu Encyclopedia as the data source and stores it in the form of a graph.

It should be noted that the organization of Baidu Encyclopedia includes the following steps:

In the first step, for each entry, there is a page describing the entry. In addition to the plain text, the page will also refer to other existing entries in Baidu Encyclopedia;

In the second step, in the graph of the knowledge base, there will be a directed edge between each such described entry and the referenced entry, and the PageRank algorithm is applied to this graph to obtain the importance of each entry;

In the third step, the weight of entries and the mutual reference relationship between entries constitute the entire knowledge base.

Each keyword in the keyword list will get an associated entry set through the keyword expansion module, and each entry has a weight. The tag generation module will integrate the two parts of the results, that is, merge all associated entries of the obtained keywords together. When an entry is associated with multiple keywords at the same time, the weights of this entry in various keywords are added. Sort all entries according to the sum of weights, and return the top 20 entries, we get the label set describing the characteristics of the program.

Repeat the above process for the 10 programs in the experimental example, and we have completed the task of automatically generating labels for these programs.

Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims (9)

1. TV programme label automatic creation system; It is characterized in that; Comprise the programme information acquisition module, information keyword extracting module, keyword expansion module and the label generation module that connect successively, also comprise the base module that is connected with the keyword expansion module, wherein:

-programme information acquisition module is used for grasping from network the page relevant with program, through pruning and the filtration to the page, obtains describing the body matter of programme information;

-information keyword extracting module is used to gather the body matter that the programme information acquisition module obtains, and from body matter, extracts keyword;

-base module is used to set up the cyberrelationship between entry, for use in the keyword that obtains is expanded;

-keyword expansion module is used to the network that utilizes base module to provide, and the keyword that the information keyword extracting module obtains is expanded, and obtains a bigger entry collection;

-label generation module is used for the related entry collection of all keywords that obtain is handled, and filtering noise calculates mark, and finally generates the tally set of program.

2. TV programme label automatic creation system according to claim 1 is characterized in that said programme information acquisition module comprises html parser; Reception needs to generate the target set of TV shows of label; Assisting down of search engine, be each program acquisition Webpage, the said page is through the processing of html parser; Obtain body matter, said body matter passes to the information keyword extracting module and is for further processing.

3. TV programme label automatic creation system according to claim 1; It is characterized in that; Said information keyword extracting module comprises participle and part-of-speech tagging device; After obtaining describing the body matter of each programme information, content is divided, and only keep the word of noun part of speech through participle and part-of-speech tagging device.

4. TV programme label automatic creation system according to claim 1 is characterized in that, the word of said noun part of speech is through statistical method identification keyword.

5. according to claim 4 TV programme label automatic creation system, it is characterized in that said statistical method may further comprise the steps:

The first step for certain specific program, is divided into two groups with word, and one group derives from the Webpage relevant with this program, one group of other Webpage that derive from the program set;

Second step; These two groups of words are calculated word frequency; And count average and standard deviation; Like this, each word all uses 4 statistics to describe its characteristic, said 4 statistics be respectively this word with word frequency average, standard deviation and this word of program related pages with the word frequency average and the standard deviation of the uncorrelated page of program;

In the 3rd step,, the key word recognition that can show programs feature is come out according to the relation between 4 statistics.

6. TV programme label automatic creation system according to claim 1 is characterized in that said base module, is stored with the form of scheming as data source with Baidu's encyclopaedia.

7. TV programme label automatic creation system according to claim 6 is characterized in that, the organizational form of said Baidu encyclopaedia may further comprise the steps:

The first step for each entry, all has a page that this entry is described, and except plain text, also can existing other entries in Baidu's encyclopaedia be quoted in the page;

In the figure of knowledge base, all can there be a directed edge in second step between entry that is described that each is such and the entry of quoting, and to this figure Using P ageRank algorithm, obtained the importance of each entry;

In the 3rd step, the mutual adduction relationship between the weight of entry and entry has constituted whole knowledge base.

8. TV programme label automatic creation system according to claim 1; It is characterized in that; The keyword that said keyword expansion module obtains each information keyword extracting module; In the figure of base module, find other entries that have a paths with it,, calculate the weight of entry according to the importance of entry itself and the distance of entry and keyword.

9. TV programme label automatic creation system according to claim 1; It is characterized in that the related entry of all keywords that said label generation module will obtain combines, when the simultaneously related a plurality of keyword of entry; With the weight addition of this entry in various keywords; The summation of all entries according to weight sorted, and return the front several as required, thereby obtained describing the tally set of programs feature.

Images