CN101901249A - A Text-Based Query Expansion and Ranking Method in Image Retrieval - Google Patents

Abstract

The invention belongs to the field of multimedia information retrieval, and relates to a method for realizing query expansion and sorting based on a semantic dictionary in image retrieval. The invention includes: a WordNet-based semantic similarity measurement algorithm for English words, a HowNet-based semantic similarity measurement algorithm for Chinese words, a query expansion word selection and optimization algorithm based on extension rules, and a scoring and optimization algorithm for retrieval results. The method of the invention uses a related text processing method and a semantic network dictionary to improve the image search engine, interacts with users through semantic expansion and sorts the retrieval results through the improved similarity measure. Compared with the traditional method, the present invention has the advantages of high accuracy, strong integrity and low time and space cost. It is of great significance for efficient image retrieval based on large-scale image data sets, considering the high-level semantic information of images, and has extensive application value in the field of cross-language and cross-media retrieval.

Description

A Text-Based Query Expansion and Ranking Method in Image Retrieval

technical field

The invention belongs to the field of multimedia information retrieval, relates to a specific media-image retrieval method, in particular to a method for realizing query expansion and sorting based on a semantic dictionary in image retrieval. The method can be used to cooperate with the content-based image retrieval method to improve image search quality and user search experience.

Background technique

In recent years, with the development of the Internet and social informatization, the capacity of digital images is increasing rapidly, and a large amount of image data is generated every day. How to quickly and accurately find, access images, and effectively use these images has become an urgent problem to be solved, which is the so-called image retrieval technology. The earliest text-based image retrieval technology (Text-based Image Retrieval, TBIR) was mainly used; since the 1990s, the research and application of content-based image retrieval (Content-based Image Retrieval, CBIR) has been greatly developed, and then Semantic-based image retrieval, feedback-based image retrieval, and knowledge-based image retrieval ^[1] have been developed. As an early technology, TBIR relies heavily on image annotation results, which is a limitation. However, text-based retrieval is a relatively mature technology, and its fast and reliable features are still very prominent. Therefore, TBIR is still an aspect worthy of research. If it can absorb some characteristics of other methods or use it interactively with several other methods, it can have good results.

From the perspective of image data itself, it can be divided into two categories. One type has related text description information, such as news images, and reporters usually attach a short text description when sending back the image; while the other type has no text description. For the image library "with text description information", the general retrieval system extracts keywords from the text description of the image as an index to realize image retrieval. Since many years of research on text retrieval have achieved a lot of results, this type of image retrieval system can often better support retrieval based on advanced semantics than the retrieval system based solely on the underlying features of images. However, the text description of the image is a brief description of the image from the perspective of the image author's own understanding and preferences, which is different from the annotation information for the image for retrieval purposes, and the content described by the two must also be different. . Therefore, the keywords extracted from the text description are different from the manually marked keywords specially used for retrieval purposes, which not only leads to a decrease in precision, but also may return many irrelevant query results to users, making users at a loss ^[2] . At the same time, in image retrieval applications, there is a certain deviation between the user's real information needs and the query request submitted by the user, and between the query request and the query request understood by the system. These deviations lead to a mismatch between the relevant images queried and the user's query, and even the information that the user wants to get. It has become an important research hotspot in the field of information retrieval, especially in the field of image retrieval, to optimize user queries through query expansion, express user query needs more accurately and objectively, and help users quickly and accurately obtain the required information ^[3,4] .

The current query expansion methods can be roughly divided into three categories, namely, based on semantic lexicon, based on global analysis and based on local analysis ^{[5, 6, 7]} . The method based on the semantic lexicon generally uses the semantic knowledge dictionary ^[8,9,10] to select words that have a certain semantic relevance with the query word for expansion. Synonymous relations, etc. ^{[11, 12 , 13, 14, 15, 16]} . This method relies on a complete semantic system and is independent of the object set to be retrieved ^{[17, 18, 19]} . The basic idea of the method based on global analysis is to conduct correlation analysis on all the words or phrases in the search objects, and add the words or phrases with the highest degree of correlation with the query words to the initial query to generate a new query ^[20] . Although this method can maximize the search for the relationship between words, the update cost after the retrieval object set changes is huge, and it will also be infeasible in terms of time and space as the retrieval object collection scale increases. The method based on local analysis is a two-stage query, that is, the first search is performed on the user's initial query, and the first N search objects are selected for analysis according to the search results, and the words with higher importance are found out, which are compared with the initial query. The query is composed of a new query, and then the second retrieval is performed using the new query ^[20] . This method is prone to the problem of "query drift". When the first retrieval result is not good, words that are not related to the query topic may be selected and added to the initial query, which will seriously reduce the query accuracy, even lower than that without query expansion. situation.

On the other hand, how to present the desired retrieval results to users and help users quickly locate the resources they need has always been the goal of image retrieval. When the user enters a query, he hopes to retrieve the most desired results in time, and these results can be ranked at the top of the retrieval results ^[21] . Especially when a large number of search results are returned, from the perspective of users' browsing habits, they basically only care about the results of the first few items, and it is impossible and unwilling to go through the very late search results one by one, and even be read by users. The probability is almost zero ^[22] . Therefore, the sorting effect of retrieval results directly affects whether users can easily obtain the required resources, and also determines the user's satisfaction with the retrieval system ^[23] . Especially the image search engine, which organizes a large number of various image resources, is an information query tool for specific media resources. Users use this type of search engine with a stronger purpose, and pay more attention to whether they can find the required resources as soon as possible in the search results, which puts forward higher requirements for the sorting process of the image search engine. The important factor that determines the ranking result is the ranking strategy of the image search engine, and the ranking strategy is one of the core parts of the image search engine, and it is also the key to the success of the image search engine.

Existing general search engine sorting algorithms can be divided into five types in principle, namely word frequency and position weighted sorting algorithm, Direct Hit algorithm, Alexa website ranking algorithm, Google sorting algorithm and similarity algorithm ^{[24, 25, 26]} . The use of word frequency and position weighting algorithm is the main idea of the early sorting of search engines, and its technology is the most mature, and it is still the core sorting technology of many search engines. The advantage of this algorithm is that it is simple and easy to implement, and it is more suitable for structured document data. Direct Hit is a sorting algorithm that focuses on information quality and user behavior feedback. To a certain extent, it can meet the "user guarantee principle" and also consider the quality of information. However, due to the random behavior of users, it is difficult to guarantee the accuracy of the sorting results. Alexa focuses on publishing world website rankings, mainly considering two aspects: comprehensive ranking and category ranking. Google's ranking algorithm is the decisive factor for its excellent search results, using a sophisticated way of sorting web file levels - PageRank. The similarity between query and retrieval result records is also an important basis for search engine ranking. Currently, the more common method is to treat query strings and documents as vectors, and the length of query and retrieval results needs to be considered.

Through the above comprehensive analysis, it can be found that in an image search engine based on keywords input by users, the keywords input by users are the only query information obtained by the engine. Under the condition that there is no change in the image database and image text description information, the relevant information corresponding to the keyword sequence must be the only retrieval result. Therefore, mining as much information as possible from the keyword sequence to assist the query will help the engine better understand the user's intention. Query expansion is one such way of expanding information. If the explicit information can be expanded through the query or the uncertainty of the user's intention can be narrowed implicitly, it will bring better retrieval results to a certain extent. In addition, image search engines often have a lot of query results, and users often only have the patience to view the first few dozen results. In other words, how to place the image retrieval results that are closer to the user's search intent at a higher position in the returned results is very important. For example, 50 correct results out of 100 returned results are not necessarily more effective than only 20 correct results, so precision and recall are both very important. Practice shows that no user will use all the search results, and the user only selects the useful ones. This is the convenience that sorting provides to users.

At the same time, the above analysis also shows that the existing query expansion and retrieval result ranking algorithms usually originate from text retrieval and are aimed at large-scale text information processing ^{[27, 28]} . Although the text-based image retrieval technology is born out of the more mature text retrieval technology, there are some unsuitable technologies in it, which will have a negative impact on image retrieval. The general query expansion model or ranking model cannot be effective for image retrieval, and the existing research on query expansion and retrieval document ranking mode for text retrieval needs to be strengthened and deepened.

References relevant to the present invention are:

[1] Oilscoil Chathair, Bhaile Cliath, A.F.Sineaton, I.Quigley, Alan F.Smeaton, Ian Quigleyand Glasnevin Dublin. "Experiments on Using Semantic Distances between Words in ImageCaption Retrieval". In Research and Development in Information Retrieval. 17, valpp -180, 1996.

[2] Yang Linpeng, Ji Donghong, Tang Li and Niu Zhenyu. "Chinese Information Retrieval based on Terms and Relevant Terms". In ACM Transactions on Asian Language Information Processing; Vol.4(3): 357-374, September, 2005.

[3] Xiqing Lin and Ximing Chen. New Methods for Query Expansion and Query Re-weighting for Document Retrieval. Master Thesis, Department of Information Engineering, National Science and Technology University, Taiwan, 2005.

[4] YiXuan Hong. Ontological Inference for User Intention Extraction, Query Expansion and Concept-based Retrieval. Master Thesis, Department of Information Engineering, National Dong-hua University, Taiwan, 2004.

[5] C.Ch.Latiri, S.Ben Yahin, J.P.ChevaVet and A.Jaouaa. "Query Expansion using FuzzyAssociation Rules between Terms". In Proceedings of the 4th JIM International Conference on Knowledge Discovery and Discrete Mathematics, Mets, 03France, .

[6] Hsi-Ching Lin, Li-Hui Wang and Shyi-Ming Chen. "Query Expansion for Document Retrievalbased on Fuzzy Rules and User Relevance Feedback Techniques". In Expert Systems with Applications, Vol.31(2): 397-405, August 2006.

[7] Hang Cui, Ji-Rong Wen, Jian-Yun Nie and Wei-Ying Ma. "Query Expansion by Mining User Logs". In IEEE Transactions on Knowledge and Data Engineering, Vol.15(4): 829-839, July/August 2003.

[8] Christiane Fellbaum (ed.). WordNet: An Electronic Lexical Database. The MIT Press, Cambridge, MA, 1998.

[9] George A. Miller and Florentina Hristea. "WordNet Nouns: Classes and Instances". In Computational Linguistics, Vol.2(1): 1-3, 2006.

[10] Dong Zhendong, Dong Qiang and Hao Changling, "Theoretical Discovery of HowNet", "Journal of Chinese Information Science", Vol.21(4): 3-9, 2007.

[11] Zhiguo Gong, Chan Wa Cheang, and Leong Hou U. "Web Query Expansion by WordNet". In LectureNotes in Computer Science Volume 4080/2006, pp.379-388, 2005.

[12] Ming-Hung Hsu, Ming-Feng Tsai and Hsin-Hsi Chen. "Query Expansion with ConceptNet and WordNet: An Intrinsic Comparison". In Proceedings of AIRS 2006, pp.1-13, 2006.

[13] Alexander Budanitsky and Graeme Hirst. "Evaluating WordNet-based Measures of Lexical Semantic Relatedness". In Computational Linguistics, Vol.32(1): 13-47, 2006.

[14] Liu Qun and Li Sujian, "Calculation of Lexical Semantic Similarity Based on HowNet", Computational Linguistics and Chinese Language Processing, Vol.7(2):59-76, 2002.

[15] Li Feng and Li Fang, "Computation of Semantic Similarity of Chinese Words—Based on HowNet", Journal of Chinese Information Science, Vol.21(3):99-105, 2007.

[16] Jiang Min, Xiao Shibin, Wang Hongwei, Shi Shuicai, "An Improved Word Semantic Similarity Calculation Based on HowNet", "Chinese Journal of Information", Vol.22(5): 84-89, 2008.

[17] Diana Inkpen and Graeme Hirst. "Building and Using a Lexical Knowledge Base of Near-Synonym Differences". In Computational Linguistics, Vol.32(2): 223-262, 2006.

[18] Ted Pedersen, Satanjeev Banerjee and Siddharth Patwardhan. "Maximizing Semantic Relatedness to Perform Word Sense Disambiguation". In University of Minnesota Supercomputing Institute Research Report UMSI 2005/25, March, 2005.

[19] Budanitsky, Alexander and Graeme Hirst. "Semantic Distance in WordNet: An Experimental, Application-Oriented Evaluation of Five Measures. In Proceedings of the Workshop on WordNet and Other Lexical Resources, The Second Meeting of the North American Computing Association for the North American Chap Linguistics, Pittsburgh, PA, pp. 29-34, 2001.

[20] Yuen-Hsien Tseng, Da-Wei Juang and Shiu-Han Chen. "Global and Local Expansion Term Expansion for Text Retrieval". In Proceedings of the Fourth NTCIR Workshop on Evaluation of Information Retrieval, Automatic Text Summarization and Question, Tokyo ok Answer, Japan, June 2-4, 2004.

[21] H. Vernon Leighton, Jaideep Srivastava. Precision among World Wide Web Search Services (Search Engines): AltaVista, Excite, Hotbot, Infoseek, Lycos.September, 2006. http://www.winona.msus.edu/library /webind2/webind2.htm.

[22] Claudio Carpineto, Giovanni Romano and Vittorio Giannini. "Improving Retrieval Feedback with Multiple Term-Ranking". In ACM Transactions on Information Systems, Vol.20(3): 259-290, July, 2002.

[23] Kemafor Anyanwu, Angela Maduko and Amit Sheth. "SemRank: Ranking Complex Relationship Search Results on the Semantic Web". In Proceedings of WWW 2005, pp.117-127, Chiba, Japan, May 10-14, 2005.

[24] Shengli Wu and Fabio Crestani. "Methods for Ranking Information Retrieval Systems without Relevance Judgements". In Proceedings of the 2003 ACM Symposium on Applied Computing, pp.811-816, Melbourne, Florida, USA, 2003.

[25]Boanerges Aleman-Meza, Chris Halaschek, I. Budak Arpinar and Amit Sheth. "Context-Aware Semantic Association Ranking". In Technical Report 03-010, LSDIS Lab, Computer Science, University of Georgia, August, 2003.

[26] Taher H. Haveliwala. "Topic-Sensitive PageRank: A Context-Sensitive Ranking Algorithm for Web Search". In IEEE Transactions on Knowledge and Data Engineering, Vol.15(4): 784-796, July/August, 2003.

[27] Jinan Fiaidhi, Sabah Mohammed, Jihad Jaam and Ahmad Hasnah. "A Standard Framework for Personalization via Ontology-based Query Expansion". In Pakistan Journal of Information and Technology, Vol.2(2): 96-103, 2003.

[28]Chris Buckley and Ellen M.Voorhees. "Retrieval Evaluation with Incomplete Information". In Proceedings of SIGIR 2004, pp.25-32, Sheffield, UK, 2004.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art, and propose a method for effectively performing query expansion and sorting retrieval results in text-based image retrieval.

The invention uses the text retrieval technology for reference to establish a "special query expansion and retrieval result sorting model" suitable for the characteristics of the image retrieval itself. Using the design concept of "from general to specific", a technical framework (including four main algorithms) is disclosed, that is, starting from the "general query expansion and retrieval result ranking model", using related text processing methods and semantic web dictionaries to image Improve the search engine, interact with users through semantic expansion, use the semantic network dictionary and the established expansion rules; and sort the retrieval results through the improved similarity measure, which focuses on the construction and optimization of the scoring algorithm, and finally establishes "Specialized Query Expansion and Result Ranking Model" for Text-Based Image Retrieval.

The method for query expansion and retrieval result sorting in the image retrieval proposed by the present invention is to utilize relevant text processing methods and semantic web dictionaries to improve the image search engine, which includes the following aspects: (1) preprocessing and preanalysis (Pre- Processing and Pre-Analysis)——for the initial query, the word segmentation and punctuation marks of the query are completed through pre-processing, and based on the pre-processed initial query, stop word marking, part of speech analysis and keyword extraction are completed through pre-analysis; (2) Word Lexical Semantic Similarity Measurement (Word Lexical Semantic Similarity Measurement)——For the English word semantic similarity measurement, the semantic distance is calculated based on the length and depth of the network path, while for the Chinese word semantic similarity measurement, based on comprehensive consideration of the main category Sememe similarity, semantic expression similarity and main class sememe frame similarity are calculated, and the maximum matching rule and sememe depth information are incorporated at the same time; (3) Query Expansion based on Fusion of Expansion Rules ——Based on the semantic network knowledge, while integrating the established specific extension rules, the semantic extension is carried out for the keyword sequence from the initial query; (4) Retrieval Results Ranking based on Scoring——based on the search The retrieval results returned by the engine are used as the processing object, and the "similarity" between the query keyword sequence and the image description is evaluated based on the word semantic similarity measure, so as to obtain the score, and optimize through the improved scoring algorithm, so that the final score is used as The order by which the search engine returns images.

Compared with the prior art, the above method of the present invention finally obtains the search results based on the extended query in the image search engine, and has three advantages, namely, high accuracy, strong integrity and low space-time cost. Its high accuracy rate is reflected in the fact that there are very few expanded words derived from unused semantic words in query expansion, which ensures that the expanded words obtained after expansion have a high degree of commonality with the initial query keywords; the search results are sorted by the search engine Images with poor correlation or "wrong" images are ranked as far back as possible in the search result list, which ensures that the "better" results in the same result set are sorted higher, so that users can see them more easily. Its integrity is reflected in the fact that in the query expansion, the expansion words added to the initial query keyword sequence are very complete, and are highly relevant and consistent with the image descriptions in the image collection; the search behavior of the search engine in the ranking of retrieval results is not affected Any interference, which ensures that the returned result set has not been affected in any way after sorting. Its low space-time cost is reflected in query expansion and retrieval result sorting. Regardless of secondary factors such as network transmission speed and server processing speed, it has better time-space efficiency. However, the transmission time in the actual environment will be longer than the actual calculation time. If the time difference is much higher, the user will not feel such a time gap.

The outstanding contribution of the present invention is to provide (1) English word semantic similarity measurement algorithm based on WordNet; (2) Chinese word semantic similarity measurement algorithm based on HowNet; (3) query expansion word selection and optimization algorithm based on expansion rules ; (4) Scoring and optimization algorithm of retrieval results. Using the above four core algorithms, a technical framework of query expansion and retrieval result ranking in image retrieval is designed.

The above-mentioned advantages of the present invention can meet the application requirements of efficient image retrieval in consideration of high-level semantic information of images for large-scale image data sets, and cross-language cross-media retrieval is its main application.

Description of drawings

Fig. 1 is the flow chart of the inventive method, label among the figure: (one) is preprocessing and pre-analysis functional module; (two) is the word semantic similarity measurement functional module; (three) is the query expansion function of fusion expansion rule (4) a functional module for sorting retrieval results based on scores.

Figure 2 demonstrates the specific steps of the above-mentioned algorithm flow framework through specific examples. By giving the intermediate output of each algorithm module and the final retrieval result of the framework, it gives people an intuitive understanding.

Among them, the labels (1) and (2) are the original English query and Chinese query input by the user respectively; the labels (3) and (4) are respectively using the English semantic dictionary WordNet and the Chinese semantic dictionary HowNet, using the "based on extension rules The expanded word sets corresponding to the original English query and Chinese query obtained by "Query expansion word selection and optimization algorithm"; labels (5) and (6) are the expanded word sets based on the original English query and Chinese query respectively, using image The corresponding retrieval results obtained by the search engine; labels (7) and (8) are the initial retrieval lists based on the original English query and Chinese query respectively, using the fusion of "WordNet-based semantic similarity measurement algorithm for English words" and "HowNet-based The final retrieval results obtained by the "Scoring and Optimization Algorithm for Search Results" of "Measurement Algorithm for Semantic Similarity of Chinese Words".

Detailed ways

The following describes in detail the process framework of the present invention for query expansion and retrieval result sorting in image retrieval and four core algorithms that form the framework in conjunction with the accompanying drawings:

Example 1

1. Algorithm process framework

Accompanying drawing 1 is the flow chart of this application framework, and the numerals 1-4 respectively represent the above-mentioned four main functional modules.

The framework is divided into four main modules: Pre-Processing and Pre-Analysis, Concept Semantic Relativity Measurement, Query Expansion based on Fusion of Expansion Rules ) and Retrieval ResultsRanking based on Scoring. Three of the four core algorithms, the English word semantic similarity measurement algorithm based on WordNet, the Chinese word semantic similarity measurement algorithm based on HowNet, and the retrieval result scoring and optimization algorithm will be used in the scoring-based retrieval result sorting module, The query expansion word selection and optimization algorithm based on the expansion rules will be used in the query expansion module that fuses the expansion rules. In the first two modules of the application framework, some existing technologies that are relatively mature at present will also be used.

(1) Pre-Processing and Pre-Analysis (Pre-Processing and Pre-Analysis): For the initial query, the main task of its pre-processing is to complete the word segmentation and punctuation marks of the query. Among them, the maximum probability word segmentation strategy is adopted for Chinese queries, while an additional case conversion process for English queries is required. Based on the preprocessed initial query, preanalysis mainly accomplishes three tasks. One is to mark the stop words in the query; the other is to analyze the part of speech of each word to determine its correct part of speech. Among them, words with changing forms in the English query need to be restored; and the third is In order to extract query terms as keywords based on the results of the part of speech analysis.

(2) Word Lexical Semantic Similarity Measurement (Word Lexical Semantic Similarity Measurement): The word semantic similarity reflects the aggregation characteristics between words, which can be measured by the degree of replaceability between two words. Word semantic distance can be regarded as the opposite of word semantic similarity. The greater the semantic distance between two words, the lower their similarity; conversely, the smaller the semantic distance between two words, the greater their similarity. For the semantic similarity measurement of English words, based on the organizational structure of the WordNet semantic dictionary, the calculation of the synonym set corresponding to the concept is mainly considered, and a pair of synonym sets with the largest similarity (or the smallest semantic distance) are taken to represent two synonym sets respectively. The final result of , which also needs to consider the specific requirements of space and time costs for image search engines. For the measurement of semantic similarity of Chinese words, based on the organizational structure of the HowNet semantic dictionary, according to the structural characteristics of the knowledge system description language, the semantic similarity is divided into three parts to calculate. The first part considers the similarity of the main class sememe alone; the second part considers the similarity of the entire semantic expression, and divides the sememe into layers according to the hierarchical characteristics described by the knowledge description language, and then adopts the method of maximum matching for each layer. Computation of similarity; the third part considers the similarity of the main class semantic original frame. At the same time, the sememe depth information is added to the calculation of sememe similarity to treat sememes with different amounts of information differently.

(3) Query Expansion based on Fusion of Expansion Rules: For a given query keyword, determine another word as its expansion word according to the expansion rules, that is, the semantic expansion rules of a single word. After the rule is determined, a semantic expansion can be understood as expanding each word of the keyword sequence based on the knowledge of the semantic network and then merging the results. Considering the arbitrariness of user input, the sequence of words in the keyword sequence is not considered and treated equally. At the same time, based on the dual considerations of image description information in the image data set and the scale of the expanded word set, the keyword set after query expansion is optimized.

(4) Retrieval Results Ranking based on Scoring: Based on the retrieval results returned by the search engine, the images are scored according to the keyword sequence and image description, and the image scores will be used as the sorting basis for the returned images . In fact, the scoring object is the description of the image without any knowledge of the image itself. This is a scoring scheme that relies on pure trust and description of the image. That is, based on the calculation support of "semantic similarity of words", the degree of "similarity" between the query keyword sequence and the image caption description keyword sequence is scored. Among them, corresponding weights are assigned to the calculation results of image caption description keywords, which are used to highlight possible "prominent" objects in the image. At the same time, by setting an appropriate size of the correlation calculation result cache, a multi-layer cache mechanism is established, and the scoring strategy is improved, thereby simplifying the complexity of the calculation, saving a lot of time, and improving the processing speed.

2. WordNet-based semantic similarity measurement algorithm for English words

The idea of the WordNet-based semantic similarity measurement algorithm for English words is based on the following idea: It is hoped that the system can dynamically calculate the semantic similarity between the English query keyword sequence and the retrieval result keyword sequence during the online processing of image retrieval result ranking, but On the basis of ensuring that it is suitable for the magnitude of the space-time cost, the statistical information of the "general English word semantic similarity measurement model" is not completely lost. That is to say, it is hoped that the synset obtained through proper processing from a specific English semantic network knowledge source can be used to modify the semantic similarity measurement model of the original words. However, the “Measurement Model for Semantic Similarity of General English Words” has the problems of data sparsity and part-of-speech limitations of the processed words, so it is impossible to integrate synsets and semantic computing models to accurately measure the semantic similarity of words during online processing. Therefore, the present invention provides a new algorithm that can use the synonym set obtained after special expansion processing to modify the original general English word semantic similarity measurement model. New algorithm of the present invention satisfies the following three conditions simultaneously:

(1) Avoid the problem of data sparsity—the number of words that make up the semantic definition of words is often not enough, which leads to the problem of data sparsity in the process of semantic calculation. Therefore, the problem can only be solved by modifying the original general English word semantic similarity measurement model by using the word set obtained after special expansion processing.

(2) The part of speech of words is not limited-it cannot only be applied to the measurement of semantic similarity between noun words, but should have the ability to measure the semantic similarity between words across parts of speech.

(3) Low spatio-temporal complexity—algorithms with high spatio-temporal complexity cannot be used in the processing process. For example, the method of probability calculation needs to read a large amount of statistical data for processing. Algorithm improvement and optimization need to be considered to meet the requirements of space-time cost .

The present invention designs the new algorithm that meets above three conditions by following steps,

Regarding the measurement of semantic similarity of English words, the classic Lesk algorithm regards the semantic definitions of words as unordered word bags, and uses the intersection of words between definitions to measure their similarity. Lesk believes that the words used in the semantic definitions of words with similar semantics are also similar, but the number of words that make up the definition is often not enough, which leads to the problem of data sparseness. Therefore, to solve this problem, the present invention proposes several extended algorithms.

The Lesk extension algorithm can overcome the data sparsity problem in the classic Lesk algorithm to a certain extent by extending the semantic definition of words. EKEDAHL and GOLUB use WordNet to adjust the Lesk algorithm, and expand the semantic definition of words used to calculate the number of overlaps by looking for the nearest two hypernyms of a concept. Pedersen et al. adopted another extension method, considering all semantic definitions directly connected to a word in the WordNet structure, including hypernyms and hyponyms, etc., and at the same time giving greater weight to the phrase. The authors claim that under the same conditions, the algorithm has a significant performance improvement over the traditional Lesk algorithm. The most commonly used information for the Lesk extension algorithm is hypernym information, that is, the parent node in WordNet, which is a further abstraction of word semantics.

The existing Lesk extension algorithm mainly considers the semantic information directly connected to a word in the WordNet hierarchy, especially the hypernym, to extend the semantic definition of words, which can overcome the problem of data sparsity to a certain extent. Although these methods can effectively use the direct information in the WordNet structure, they ignore some very useful indirect information. Thus, the present invention establishes a Lesk extension algorithm (Lesk-C for short) based on equivalent words (Coordinate Terms), which can further expand the semantic definition of words, wherein the equivalent words are defined as the set of synonyms to which a certain word belongs in the WordNet hierarchy (e.g. equivalents of "basketball" include "football", "volleyball", etc.). Obviously, a synonym set and its corresponding equivalent words must have a common parent node.

The Lesk-C algorithm extends the semantic definition of a concept by introducing all (or some) equivalent definitions of a word meaning. Its idea is based on the assumption that any concept and its equivalents play the same role in determining the context. According to the above assumptions, considering that "basketball", "football" and "volleyball" are a group of equivalent words, the established Lesk-C algorithm is used to expand "volleyball" by using the definitions of the equivalent words "basketball" and "football". "The definition will undoubtedly increase the possibility of word intersection. Equivalent words are sibling nodes in WordNet. Although they are not abstract or even directly related to the original word meaning, under the assumption that the semantic definition of any concept and its equivalent words have the same role in determining the concept in the context, Equivalents and hypernyms are equally meaningful.

For the complete semantic definition of words obtained after being extended based on Lesk-C, since each word belongs to multiple synsets, the semantic correlation measure (or semantic distance calculation) between two concepts is actually two Computation of synsets. Generally speaking, a pair of synsets with the largest similarity (or the smallest semantic distance) is used to represent two synsets respectively to calculate the final result.

The following description conventions of the present invention and the symbols to be used are defined as follows:

(1) The path distance between two synsets S1 and S2 on the WordNet semantic network is the number of edges passed by the path from S ₁ to S ₂ , expressed by the Len(S ₁ , S ₂ ) function.

(2) When only the upper and lower type edges of the WordNet semantic network are considered, the semantic network degenerates into a forest. After adding a virtual root node, the forest is transformed into a tree. The lowest common parent node (Lowest Super-Ordinate) of two synsets S ₁ and S ₂ in the upper and lower semantic trees is represented by the Lcs(S ₁ , S ₂ ) function, and its depth on the tree is represented by the Depth() function express.

(3) The relationship between concept semantic relevance Sim(C ₁ , C ₂ ) and semantic distance Dist(C ₁ , C ₂ ) is:

Sim(C ₁ , C ₂ )+Dist(C ₁ , C ₂ )=1 (1)

In concept semantic similarity measurement, the WordNet hierarchy is regarded as a graph, and then the path information is used to calculate the correlation. The more direct idea is: the closer the distance between two nodes, the greater the correlation between them. That is to say, if the common hypernyms of concepts represented by two nodes are closer to them, the similarity between the two is greater. Here, the similarity formula used is as follows:

$\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="S"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">S</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; Depth</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Lcs</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="S"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">S</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; Depth</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; Depth</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Among them, Depth() is the depth of concept C or synonym set S in the WordNet hierarchy, and LCS(S1, S2) is the largest depth among all public hypernyms of concepts C ₁ and C ₂ or synonym sets S ₁ and S ₂ the hypernym of .

This formula can be transformed into the following formula through deformation:

$\mathrm{<span\; class="notranslate">\; Dist</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; Len</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Lso</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; Len</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="2"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Lcs</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; Len</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Lcs</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; Len</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="2"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Lcs</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; Depth</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Lcs</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="C"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">C</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

There is a polysemy phenomenon in English, and the semantic similarity of words should calculate the similarity between concepts (or word meanings, semantic definitions). The semantic similarity of two isolated words is the maximum similarity between all concepts.

Sim(W ₁ , W ₂ )=maxSim(C _1i , C _2j )i=1Λn, j=1Λm (4)

Among them, W1 represents word 1 with n concepts, W ₂ represents word 2 with m concepts, C _1i is the i-th concept of W1, and C _2j is the j-th concept of W ₂ .

The steps of the above algorithm are described in pseudocode as follows:

(1) Get input: two words W ₁ and W ₂ .

(2) Select its two concepts C _1i and C _2j .

(3) Search the WordNet semantic network file to obtain two synonym sets S _1i and S _2j respectively representing C _1i and C _2j .

(4) According to formulas (1)-(3), input S _1i and S _2j into Dist(C _1i , C _2j ) to calculate the semantic distance result.

(5) Repeat steps (2)-(4) to obtain the similarity (semantic distance) value between each pair of concepts of two words. According to formula (4), the maximum value is selected as the final word similarity value.

Wherein, when calculating Dist(C _1i , C _2j ), only the hyponymy relation is used.

3. Chinese word semantic similarity measurement algorithm based on HowNet

The idea of the HowNet-based semantic similarity measurement algorithm for Chinese words is based on the following idea: It is hoped that the system can dynamically calculate the semantic similarity between the Chinese query keyword sequence and the retrieval result keyword sequence during the online processing of image retrieval result ranking, but On the basis of ensuring that it is suitable for the magnitude of time and space costs, it can fully consider the many difficulties and complexities that exist in Chinese. That is, it is hoped that by using the multi-level description of the corresponding Chinese word concept in the specific semantic network knowledge source, rich semantic information can be extracted, and a measurement mechanism that is more in line with human subjective perception can be established. However, the "general Chinese word semantic similarity measurement model" has the problems of not being able to fully capture the inherent correlation between word concepts, domain imbalance, and data sparseness, so the online processing process is more inclined to calculate the similarity of word concepts themselves, rather than Too much focus on its different semantics. Therefore, the present invention provides a new algorithm that can use the word concept semantic description with "correctness, non-prejudice and completeness" to modify the general Chinese word semantic similarity measurement model. New algorithm of the present invention satisfies three conditions simultaneously:

(1) Avoid the data sparsity problem—the number of words that make up the semantic definition of a concept is often not enough, which leads to the sparse data problem in the process of semantic computation. Therefore, we can only use the multi-level description of word concept semantics and add auxiliary information to modify the original general Chinese word semantic similarity measurement model to solve this problem.

(2) Highly distinguishable—it should be able to effectively use the knowledge structure of the Chinese semantic network to distinguish different word groups at different levels of similarity.

(3) Low spatio-temporal complexity—algorithms with high spatio-temporal complexity cannot be used in the processing process. For example, the method of probability calculation needs to read a large amount of statistical data for processing. Algorithm improvement and optimization need to be considered to meet the requirements of space-time cost .

Now describe how to design a new algorithm that meets the above three conditions.

In order to ensure the complexity, consistency and accuracy of the semantic description of words and concepts, HowNet adopts a knowledge description specification system - Knowledge Database Mark-up Language (KDML), which has the following four important forms.

(1) Sememes - words used in KDML are called Sememes, such as "exercise|exercise" and "sport|sports", and are organized according to KDML grammatical rules. Sememe has no ambiguity, and is extracted from Chinese characters (including simple words), "the most basic and not easy to divide the smallest unit of meaning", that is, the smallest unit of description.

(2) Main class sememe—the first sememe in the semantic expression is also called the main class sememe. In the above example, “exercise|exercise” is the main class sememe. The main class semantics must point out the most basic meaning of the concept, and it can be considered that it has the strongest ability to describe the concept.

(3) Semantic expression——"DEF={...}" is the core of the whole record, which is the definition and description of concepts, called semantic expression. In order to ensure the complexity, consistency and accuracy of concept description, KDML is used for specification.

(4) The main class sememe frame—simply put, most sememes are also defined as semantic expressions like words, as shown in the figure below. Among them, for the sememe "thing|everything", its main class sememe frame is "{entity|entity: {ExistAppear|existence: existent={～}}}", and the description syntax strictly follows the KDML description language.

In the semantic description of word concepts established based on KDML, sememes in different bracket levels have different ability to describe the semantic definition of words, and the sememes in the outer brackets have stronger ability to describe concepts; The sememe in the inner brackets is a specific explanation of the previous layer of sememe, and it is an indirect description of the concept, and its descriptive ability is relatively weak. Therefore, it is necessary to treat them differently when measuring the semantic similarity of words.

As an important basis for word similarity measurement, the calculation of sememe similarity is carried out according to the hierarchical system of sememe (namely, the hyponym relationship). Based on the tree hierarchy, the calculation formula of sememe similarity is established by considering the path length between nodes and introducing the hierarchical depth of nodes, as shown below.

$\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="S"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">S</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Depth</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Depth</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Depth</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Depth</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; Dist</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="1"\; label="S"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">S</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Among them, S ₁ and S ₂ represent two sememes respectively; Dist(S ₁ , S ₂ ) represents the path length between sememe S ₁ and S ₂ ; α is an adjustment parameter; Depth(S ₁ ) and Depth(S 2 ₂ ) represent the hierarchical depths of sememes S ₁ and S ₂ respectively; min(Depth(S ₁ ), Depth(S ₂ )) represents the smaller of the hierarchical depths of sememes S ₁ and S ₂ . The semantic information carried by sememes can be divided into different sizes. The lower the nodes are, the richer the semantic information is, and the higher the nodes are, the more abstract the semantic information is. Therefore, sememes at different levels should be treated differently.

There is a polysemy phenomenon in Chinese, and the semantic similarity of words should calculate the similarity between word concepts. The semantic similarity of two isolated words (not in a certain context) is the sum of the similarities between all their concepts. maximum value.

Sim(W ₁ , W ₂ )=maxSim(C _1i , C _2j )i=1Λn, j=1Λm (2)

Among them, word W ₁ has n concepts, word W2 has m concepts, C _1i is the i-th concept of W ₁ , and C _2j is the j-th concept of W ₂ . According to the structural characteristics of KDML, the conceptual semantic similarity is divided into three parts for calculation:

Sim(C ₁ , C ₂ )=w ₁ ^* P ₁ +w ₂ ^* P ₂ +w ₃ ^* P ₃ (3)

Among them, P ₁ is the similarity between two conceptual main class sememes; P ₂ is the similarity between the entire semantic expressions; P ₃ is the calculation of the similarity between two DEF main class sememes; w ₁ , w _{2 ,} and w ₃ are the weights corresponding to the three partial similarities respectively, and should satisfy the constraint condition w ₁ +w ₂ +w ₃ =1 and w ₂ >w ₁ , w ₂ >w ₃ .

For P ₁ , it is calculated according to formula (1). It has been shown above that the main sememe has the most direct semantic description ability for the concept, so it is meaningful to consider it as a part separately.

For P ₂ , since the semantic expression is a complete entity and has its own grammatical rules, it is necessary to calculate its semantic similarity by taking it as a whole and referring to KDML rules. This part is the most complex part with the largest weight ratio in the whole semantic similarity measure. Because the entire semantic expression needs to be considered. The calculation process can be divided into two stages. According to the hierarchical characteristics described by KDML, sememes are divided into layers, and then each layer adopts the method of maximum matching to calculate the semantic similarity. First, calculate the semantic similarity of each group of sememes, and select the group with the largest value. If there are multiple groups of sememes with the same semantic similarity, one group can be selected. Second, select the one with the highest semantic similarity among the remaining sememe groups, and so on. When the number of sememes in the same layer of two concepts is not equal, there will be a situation where sememes and empty elements are paired. In this case, the smaller value r (the set parameter) can be uniformly taken. Finally, the value of _P2 can be obtained by summing the semantic similarity of the selected sememe groups and taking the average value.

For P ₃ , its calculation method is the same as that of P ₂ . The semantic similarity measurement for the main class sememe framework is actually another method to calculate the main class sememe semantic similarity, which once again emphasizes the direct description ability of the main class sememe for concepts.

Finally, based on the above three-part similarity calculation, the semantic similarity between each pair of concepts can be calculated according to formula (3), and then the maximum value is taken as the semantic similarity between words according to formula (2).

Some special cases that need to be noted are that when a word can be fully explained with only one sememe, it means that the sememe contains a relatively large amount of information and is at the bottom of the sememe tree. At this time, if the sememe depth information is added, the descriptive ability of the single sememe can be improved, and the semantic similarity of words can be closer to the expected value. In addition, the special meaning sememes enclosed in quotation marks can also be called concrete words, which contain rich and specific semantic information, and have a direct decisive effect and influence on the nature of the concepts they describe. Therefore, it should be distinguished from common sememes, and an adjustment parameter should be given to the semantic similarity between specific words.

In the above-mentioned semantic similarity measurement model, based on the whole semantic expression, sememes are divided according to levels, and the method of maximum matching is adopted, and the direct description ability of the main class sememe to the concept is considered separately. This mechanism for measuring semantic similarity can make more effective use of the knowledge structure of HowNet, making the results more discriminative. At the same time, due to the proper consideration of sememe depth information in the measurement process, the result is more accurate, especially when the number of sememes in the semantic expression is not large.

The steps of the algorithm are described in pseudocode as follows:

(1) Get input: two words W ₁ and W ₂ .

(2) Select its two concepts C _1i and C _2j .

(3) Search HowNet's semantic network files to obtain the main class sememe, semantic expression, semantic expression framework and other related information of concepts C _1i and C _2j .

(4) Based on the calculation formula (1) of the sememe similarity, obtain the similarity information P ₁ between the two conceptual main class sememes.

(5) Based on the two-stage solution process, calculate the similarities P ₂ and P ₃ between the semantic expressions of the two concepts and the original frame of the main class respectively.

(6) Synthesize the similarity information of the three parts, and obtain the similarity value between two concepts according to the formula (3).

(7) Repeat steps (2) to (6) to obtain the similarity value between each pair of concepts of two words. According to formula (2), the maximum value is selected as the final word similarity value.

4. Query expansion word selection and optimization algorithm based on expansion rules

There are two ways to learn from the query semantic extension based on the semantic network of semantic lexicons. One is to automatically add the search results based on the original query to the original query keyword sequence. This method generally requires manual participation and a certain scale of machine learning and accumulation. Otherwise, a large number of irrelevant words will be introduced, making the expansion results very bad. The other way is to give the choice to the user, and only provide the expanded results, and it is up to the user to decide whether it is applicable or which expanded results to use. Although using the latter method requires the active participation of the user, which increases the complexity of the user's use of the search engine to a certain extent, the resulting expanded words are actually a kind of user input and have high use value.

Automatic query expansion in the field of text retrieval is a relatively mature technology, such as [3, 4, 7]. This type of algorithm considers more about merging relevant information in retrieved documents, but many retrieved documents have nothing to do with queries . The research on semi-automatic query expansion combined with user interaction is relatively mature, such as [5, 6, 20], this type of algorithm usually provides all relevant lexical information that can be extracted from the retrieved documents to the user, causing the user to face a wide range of There are many choices, and it is easy to cause inappropriate selection or introduce unnecessary noise information. At the same time, the above two query expansion methods are both established for text retrieval and combined with the characteristics of text information, and are not fully applicable to text-based image retrieval. In view of this, a new algorithm that combines two classic query expansion techniques and is suitable for image retrieval came into being. It is easier to implement and use, and it is a lightweight and more direct effect method.

After determining the query expansion mode, in realizing the query expansion function, the construction of query expansion rules suitable for the characteristics of image retrieval must be considered first. That is, for a given query key term, what specific rules are used to determine that another term is its appropriate expansion word, that is, the semantic expansion rules of a single term. Based on the determination of expansion rules, query semantic expansion can be understood as expanding each term of the query keyword sequence separately and then merging them. Considering the arbitrariness of user input, there is no difference in the order of each term in the keyword sequence, and they are treated equally.

In order to satisfy the user's desire to make the search intention more obvious by selecting expanded terms, the present invention considers the following two situations to establish query expansion rules.

(1) The user cannot find a good vocabulary to describe the image as the search object. On the other hand, the annotator of the image often uses direct and commonly used words, which makes the user's word input more difficult. Therefore, according to the labeling information of the image itself and the retrieval requirements, some terms that have commonality with the query keywords should be expanded. For example, if a user wants to search for images of big cats and enters "big_cat", the results will be terrible. Because most of the images are tagged with specific terms such as "tiger" and "lion", few results will be returned. In view of this situation, the best solution is to input the names of big cats such as "tiger" and "lion" into the search box, but this method is obviously very cumbersome for users. So, if it were possible to just type "tiger" and expand to some other big cat names, the user would just have to select the expanded term into the search box without having to think about what other names need to be typed in via the keyboard.

(2) When the query keyword entered by the user has multiple meanings (this is often the case), by selecting extended terms to add to the keyword sequence, it can be said to provide a certain disambiguation basis for the search engine.

For example, for the keyword "bank", if it can be together with "water" or "coast", the image search engine has a basis to avoid returning or scoring too high an image about "bank".

Based on the above expansion rules, in query semantic expansion, by searching the semantic network of semantic dictionaries (including WordNet for English queries and HowNet for Chinese queries), there will be partial relationships (Part), siblings, and original English query keywords. Related terms of the relationship (Sibling) and child relationship (Child) are returned as expanded terms, and the original Chinese query is expanded directly using DEF matching. Among them, the child relation expanded for English query only includes direct children, that is, direct child nodes in the semantic hierarchy relation.

In addition to the above-mentioned extension rules, the present invention also considers that the extended term is finally added to the keyword sequence, and the term of the keyword sequence is used for matching processing with the image annotation information in the image database during the search process. Therefore, it is meaningless to expand the terms that have not appeared in the image annotation information because they are useless to the search results. The number of terms in the lexicon is usually tens of thousands, or even more than one hundred thousand, which may be selected as extended terms through the above-mentioned expansion rules. However, image annotation information generally only contains common words, and the set of common words is much smaller. Therefore, in the query semantic expansion based on expansion rules, the last step is to use tag words to filter the extended word set, and the extended words that do not appear in the tag word set will be discarded.

In addition, a problem that needs to be solved in the semantic expansion of English queries based on WordNet is that because each term in the keyword sequence is independent of each other, the final expansion result is the union of the expanded word sets of each term. Similarly, the extended word set of each term can also be obtained by the union of the extended word sets including the synsets of the term. If a certain Synset of a term is located in a "dense" position in the semantic network, and the semantic relationship is relatively complex, the scale of the expanded vocabulary obtained from this Synset is much larger than that of other Synsets. While all synsets of a term have the same status, it is possible to introduce a lot of extended term information that becomes noise through the aforementioned synset that brings a large-scale expanded set. For example, when expanding the keyword term "tiger", a large number of expanded terms are actually related to "humanities". This unexpected result comes from a Synset of "tiger", whose semantics is "a fierce or audacious person", and the semantics of this "person" is not the common semantics of "tiger", so it cannot be passed without disambiguation processing. Specific rules for finding such Synsets. Therefore, when performing semantic expansion, limit the size of each Synset's expanded word set (such as limiting each Synset to expand at most 15 expanded terms), so as to avoid expanding a large number of words due to a relatively remote Synset. Unqualified expansion term.

The steps of the algorithm are described in pseudocode as follows:

(1) Obtain input: the original query keyword sequence.

(2) Select one of its keywords.

(3) If it is an English keyword item, search the semantic network file of WordNet to obtain its synset. If it is a Chinese keyword item, search HowNet's semantic network file to obtain its semantic definition DEF.

(4) Based on the expansion rules, for each Synset of English keyword items, according to the partial relationship (Part), brother relationship (Sibling) and child relationship (Child) in the semantic network hierarchy, find the corresponding synonym word set as the expansion word set; for each DEF of the Chinese keyword item, it is directly matched and expanded.

(5) Based on the extended post-processing strategy, the extended word set is filtered and screened according to the image database annotation set information, so as to obtain the final optimized extended word set.

(6) Repeat (2)-(5) to obtain and merge the expanded word sets of each keyword item in the original query, and use it as an expanded query expression corresponding to the original query.

5. Scoring and optimization algorithm for search results

The basic unit of sorting for image search engines is images, and the basic basis for sorting is image features. In image retrieval based on image content, the underlying features of an image are the features of an image; while in text-based image retrieval, the annotation information of an image is the image feature. For the latter, the query keyword input by the user is added as the sorting standard of the images, and the sorting is to arrange the images whose tag word sequence is closer to the query keyword sequence in a higher position in the retrieval result list. Therefore, a scoring and optimization algorithm for the retrieval results is needed to determine which image is "better" for the user's query keyword sequence among the images of the retrieval results. However, the "good" standard does not actually exist, and even if different users enter the same query, they are likely to make very different evaluations on the same returned results. Therefore, the scoring and optimization algorithm of retrieval results only defines a scoring rule, and sorts the images by adjusting parameters to achieve a "better" effect.

In view of the fact that the sorting process in the field of text retrieval disclosed in the prior art is a relatively mature technology, such as [24, 25, 26], this type of algorithm considers more about the direct matching of query keywords and retrieved documents, and it is possible As a result, retrieved documents that do not contain the user's query keywords but are indeed relevant cannot be returned. The invention establishes a sorting strategy suitable for text-based image retrieval and provides a retrieval result scoring and optimization algorithm. The results show that it does not interfere with the search behavior of the image search engine and has no impact on the returned retrieval results. The main function of this algorithm is to rank the more "better" results in the same retrieval result set higher, so that users can observe them more easily.

There are often many retrieval results of image search engines, and users often only have the patience to look at some of the previous results. In other words, how to place the search results that are closer to the user's search intent at the front of the returned results is very important. Therefore, a scoring algorithm based on the semantic similarity of words is designed to sort the returned retrieval results, and to score according to the query keyword sequence and the annotation information of the image (that is, the tagged word sequence), so that the scores of the returned images are used as the sorting basis . In fact, the scoring object of the algorithm is the label set of the image without any knowledge of the image itself. This is a scoring scheme that purely trusts and relies on image labeling information. The semantic similarity measure of words discussed above is for the sorting algorithm here to be able to get the calculation support of "semantic similarity".

Since the input of the query keyword by the user is done in a random manner, each term in the query keyword sequence is treated equally. However, for annotated word sequences for images, it is assumed that the top-ranked terms are more trustworthy. This assumption is based on the fact that annotators tend to input the most prominent objects in the image first. Admittedly, for the same image, different annotators have different judgments, and there may not necessarily be the most salient object. But for most images, the in-focus objects in the image are still very obvious. It is for this consideration that the calculation results of tag words in the scoring algorithm are weighted to highlight possible "prominent" objects in the image. From this, the ranking score of the image is calculated using the following formula:

$\mathrm{<span\; class="notranslate">\; Score</span>}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; k</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

其所占比例为：Among them, _ki is the i-th keyword of the keyword sequence; t _j is the j-th tagged word _of the image tagged word sequence; Sim( _ki , t _j ) is used _to calculate the The semantic similarity between them; w(j, m) is the relevant weight, w(j, m)=(m+1-j) ² , which is used to highlight the contextual relationship of tagged items in the tagged sequence; while n and m are are the number of terms contained in the query keyword sequence and the image tagging word sequence, respectively. Considering that the weight of the first tagged word in the image tagged word sequence is m ² , then relative to the total weight

Its proportion is:

$\frac{{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="2"\; label="m"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">m</figure-callout></span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="2"\; label="m"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">m</figure-callout></span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="2"\; label="j"\; filenames="HSA00000118885700021.png"\; state="\{\{state\}\}">j</figure-callout></span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; 6</span>}\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 6</span>}\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

This function is a decreasing function. As the sequence of image tagging words increases, the influence of the weight of the top words decreases linearly. If an image contains too many objects, each object will not be particularly prominent.

One situation to be aware of is when there is a lot of double counting in the score calculation. The query keyword participates in all word semantic similarity calculations, and the annotation sequence of each image will contain at least one query keyword (otherwise the image will not be returned as a retrieval result), and there will be a large number of identical annotations in the retrieval result images terms, so the semantic computation actually necessary is much less often than the semantic computation is invoked. By setting a similarity calculation result cache of an appropriate size and recording some semantic calculation results, it is of great help to improve the processing speed. On the other hand, when the image search engine is dealing with paged display caused by too many retrieval results, it is recommended to re-search each time a paged result is accessed. If a cache can be used to cache the scoring results of some images, then a large number of calculations will be avoided when the user switches between different pages of image retrieval results. From similarity (semantic distance) result cache, result document cache, to the underlying synset read module cache, the original scoring algorithm is optimized based on the multi-layer cache mechanism, which can save processing time to a large extent.

The calculation method of the scoring algorithm gives a score that is precisely integrated according to the results of the semantic similarity function, but in actual calculations, some approximate results can also complete the task. After all, all you need is a good sorting effect, and the exact score value is not important. In the score calculation, the tag word sequence of each image has tag words that match the query keywords, and also has tag words that have a relatively small semantic similarity (or a considerable semantic distance) with the query keywords. Each tagged word in the tagged word sequence has a corresponding order weight. A tagged word that ranks last and has a small semantic similarity has an impact on the final search list. It is a tag that ranks first and matches the query keyword. Whether it's one-twentieth or one-fiftieth of a word obviously doesn't matter. Therefore, for these calculation results that have little influence on the final search list, uniformly adopt a same result representation instead of step-by-step calculation, which will greatly simplify the calculation complexity.

The steps of the algorithm are described in pseudocode as follows:

(1) Obtain input: the original query keyword sequence.

(2) Use the query expansion function to obtain the expanded word set of the original query keyword sequence.

(3) Calculate the semantic similarity for each pair of query key terms and their extended terms, and store all the results in the cache.

(4) Obtain the retrieval results corresponding to the original query based on the image search engine.

(5) Calculate the score for each image in the retrieval result. If the relevant information can be obtained from the cache, use the existing results; otherwise, consider the result of the semantic calculation to be very large (that is, the semantic distance is far away), and use a unified constant result instead of performing semantic calculation.

(6) Reorder each image according to the score of each image in the retrieval result to return the final retrieval list. Among them, in the scoring process of step (5), all semantic calculations will be used as preprocessing, and no semantic calculations are required for actual scoring, so the total number of semantic calculations will be reduced by an order of magnitude compared with the scoring algorithm before optimization, so as to improve Processing efficiency.

Embodiment 2 application example

Attached Figure 2 demonstrates the specific steps of the above-mentioned algorithm flow framework through a specific example, and gives people an intuitive understanding by giving the intermediate output of each algorithm module and the final retrieval result of the framework.

Labels (1) and (2) are the original English query and Chinese query input by the user respectively; labels (3) and (4) respectively use the English semantic dictionary WordNet and the Chinese semantic dictionary HowNet, using the "extension rule-based query The expanded word sets corresponding to the original English query and Chinese query obtained by the "Extended Word Selection and Optimization Algorithm"; labels (5) and (6) are the expanded word sets based on the original English query and Chinese query, respectively, using the image search engine The corresponding retrieval results obtained; labels (7) and (8) are the initial retrieval lists based on the original English query and Chinese query respectively, using the fusion of "WordNet-based English word semantic similarity measurement algorithm" and "HowNet-based Chinese word The final retrieval results obtained by the "Scoring and Optimization Algorithm of Retrieval Results" of "Semantic Similarity Measurement Algorithm".

Claims (5)

1. text based query expansion and sort method in the image retrieval is characterized in that comprising the steps:

(1) pre-service and preanalysis

At initial query, finish the participle and the punctuation mark mark-on of inquiry by pre-service, and, finish stop word mark-on, part of speech analysis and keyword extraction by preanalysis based on through pretreated initial query;

(2) phrase semantic measuring similarity

Measure at the english vocabulary semantic similarity, the path Network Based and the degree of depth are come the computing semantic distance, measure at the Chinese terms semantic similarity, calculate based on taking all factors into consideration the former similarity of main classes justice, semantic formula similarity and the former framework similarity of main classes justice, incorporate maximum match rule and adopted former depth information simultaneously;

(3) query expansion of fusion extension rule

Based on semantic network identity, merge the particular extension rule of being set up simultaneously, carry out semantic extension at the keyword sequence that comes from initial query;

(4) result for retrieval based on scoring sorts

The result for retrieval that returns with search engine is as process object, based on " the close degree " between phrase semantic measuring similarity assessment searching keyword sequence and the iamge description explanation, obtain scoring, and be optimized by scoring algorithm, final score is returned the sort by of image as search engine.

2. method according to claim 1, it is characterized in that, in the prototype of described english vocabulary semantic similarity metric algorithm, set up a kind of Lesk expansion algorithm based on equal speech, further expand the phrase semantic definition, wherein equal speech is defined as synset under certain word and is combined in sibling in the WordNet hierarchical structure, wherein, synonym set is public father node of pairing equal speech existence with it.

3. method according to claim 1, it is characterized in that, in the prototype of described Chinese terms semantic similarity metric algorithm, based on whole semantic formula, divide justice is former by level, adopt the method for maximum match, consider the former direct descriptive power of main classes justice separately for notion; Simultaneously, in metrics process, add the consideration of adopted former depth information, notion semantic similarity wherein is divided into following three parts and calculates:

Sim(C ₁，C ₂)＝w ₁*P ₁+w ₂*P ₂+w ₃*P ₃

Wherein, P ₁Be the similarity of two notion main classes justice between former; P ₂Be the similarity between the whole semantic formula; P ₃Be at calculation of similarity degree between two former frameworks of DEF main classes justice; w ₁, w ₂With w ₃Be respectively three pairing weights of part similarity, should satisfy constraint condition w ₁+ w ₂+ w ₃=1 and w ₂＞w ₁, w ₂＞w ₃

4. method according to claim 1 is characterized in that its algorithm steps of query expansion of described fusion extension rule adopts following false code to describe:

(1) obtains input: the original query keyword sequence;

(2) select its certain key word item;

(3) if be the English keywords item, search the semantic network file of WordNet, obtain its synset Synset;

If be the Chinese key word item, search the semantic network file of HowNet, obtain its semantical definition DEF;

(4) based on extension rule, at each Synset of English keywords item, according to the part relations in the semantic network hierarchical structure, brotherhood, and children relation are sought corresponding near synonym word set as the expansion word set; At each DEF of Chinese key word item, do with direct coupling expansion;

(5) based on expansion aftertreatment strategy, according to image library mark collection information, the expansion word set is carried out filtering screening, obtain the final expansion word set after the optimization;

(6) repeat (2)～(5), the expansion word set that obtains each key word item in the original query merges, with its as with the back query express of the corresponding expansion of original query.

5. method according to claim 1 is characterized in that, in the described prototype based on the result for retrieval sort algorithm of marking, the result of calculation additional weight of mark speech in the scoring algorithm is used for outstanding possible " giving prominence to " object of image; Adopt the ranking score of following formula computed image:

$\mathrm{Score}=\frac{\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}\underset{j=1}{\overset{m}{\mathrm{\&Sigma;}}}w(j,m)\mathrm{Sim}(k_i,t_j)}{\underset{j=1}{\overset{m}{\mathrm{\&Sigma;}}}w(j,m)}$

Wherein, k _iI keyword for keyword sequence; t _jJ mark speech for the image labeling word sequence; Sim (k _i, t _j) be used to calculate two lexical item k _iWith t _jBetween semantic similarity; W (j m) is associated weight, and w (j, m)=(m+1-j) ², be used for the context that outstanding mark sequence marks lexical item; N and m are respectively the lexical item numbers that searching keyword sequence and image labeling word sequence are comprised; First mark speech weight in the image labeling word sequence is m ², then with respect to total weight

Its proportion is:

$\frac{m^2}{\underset{j=1}{\overset{m}{\mathrm{\&Sigma;}}}w(j,m)}=\frac{m^2}{\underset{j=1}{\overset{m}{\mathrm{\&Sigma;}}}{j}^2}=\frac{6m^2}{m(m+1)(2m+1)}=\frac{6m}{(m+1)(2m+1)}$

This function is a decreasing function, and along with the increase of image labeling word sequence, the weights influence of file leader's speech is linear successively decreases.

Images