JP2019535082A - System and method for language detection - Google Patents

Abstract

Implementations of the present disclosure relate to methods, systems, and computer program storage devices for identifying languages in messages. Non-language characters are removed from the text message to generate a sanitized text message. By performing at least one of (i) an alphabet-based language detection test that determines a first set of scores and (ii) a script-based language detection test that determines a second set of scores In the sanitized text message, alphabets and / or scripts are detected. Each score in the first set of scores represents the likelihood that the sanitized text message contains an alphabet for one of a plurality of different languages. Each score in the second set of scores represents the likelihood that the sanitized text message contains a script for one of a plurality of different languages. A language in the sanitized text message is identified based on at least one of the first set of scores, the second set of scores, and the combination of the first and second sets of scores.

Description

This application claims priority and benefit of US patent application Ser. No. 15 / 283,646 filed Oct. 3, 2016, and is hereby incorporated by reference. No. is incorporated herein by reference in its entirety.

BACKGROUND This disclosure relates to language detection, and in particular, to a system and method for detecting a language in a short text message.

  In general, language detection or language identification is a process in which the language present in the body of the text is automatically detected based on the content of the text. Language detection is useful in the context of automatic language translation, and in general the language of a text message must be known before the message can be accurately translated into a different language.

  Traditional language detection is usually performed on many word and sentence sets (ie document level), but especially difficult domains are those where messages often contain only a few words (eg 4 words or less) A chat text domain where some or all of the words may be informal and / or misspelled. In the chat text domain, the lack of information and the presence of such informal messages have proven that existing language detection approaches are inaccurate and / or slow. .

  Overview

  Embodiments of the systems and methods described herein can include, for example, information about the content of a message, information about the keyboard used to generate the message, and / or information about the language preference of the user who generated the message. Used to detect language in text message based. Compared to previous language detection techniques, the systems and methods described herein are generally faster and more accurate, especially for short text messages (eg, 4 words or less).

  In various examples, the systems and methods use multiple language detection tests and classifiers to determine probabilities associated with possible languages in text messages. Each language detection test may output a set or vector of probabilities associated with a potential language. The classifier may combine the outputs from the language detection test to determine the most likely language for the message. The particular language detection test and classifier selected for the message may depend on the prediction accuracy, confidence score, and / or language domain for the message.

  Certain examples of systems and methods described herein perform initial classification of languages in text messages so that more focused language detection techniques can be performed to make a final language determination. For example, the system and method may perform an initial language detection test on the text message to identify a group or category (eg, Cyrillic or Latin language) for the language in the text message. Once a language category is identified, language detection techniques designed for that language category can be used to identify a particular language in the message. In a preferred example, unwanted elements (eg, pictograms or number digits or letters) are removed from the text message prior to language detection, which makes language detection faster and more accurate. The systems and methods described herein are generally more accurate and efficient than previous language detection approaches. The systems and methods may be configured to use any one or more of the language detection methods described herein.

  In one aspect, the subject matter of this disclosure relates to a computer-implemented method for identifying a language in a message. The method includes obtaining a text message, removing non-language characters from the text message to generate a sanitized text message, and at least one of an alphabet and a script present in the sanitized text message. Detecting, determining (i) a first set of scores each representing a likelihood that the sanitized text message includes an alphabet for one of a plurality of different languages. Performing an alphabet-based language detection test to: (ii) a second set of scores each representing a likelihood that the sanitized text message includes a script for one of a plurality of different languages. Script-based language detection to determine Comprising at least one of the performing the strike. The method further includes determining a language in the sanitized text message based on at least one of a first set of scores, a second set of scores, and a combination of the first and second sets of scores. Including identifying.

  In some implementations, non-language characters include pictographs and / or numbers. The combination may include interpolation between the first and second sets of scores. In some examples, identifying languages in the sanitized text message generates a third set of scores each representing the likelihood that the sanitized text message includes one of a plurality of different languages. To perform a language detection test on the sanitized text message. The language detection test may be selected from a plurality of language detection tests based on at least one of the first set of scores, the second set of scores, and the combination of the first and second sets of scores. .

  In some cases, the language detection test includes a language detection method and one or more classifiers. Language detection methods may include, for example, a dictionary-based language detection test, an n-gram language detection test, an alphabet-based language detection test, a script-based language detection test, a user language profile language detection test, or any combination thereof . The one or more classifiers may include, for example, a supervised learning model, a partially supervised learning model, an unsupervised learning model, interpolation, or any combination thereof. In various implementations, the method includes processing a third set of scores using one or more classifiers to identify languages in the sanitized text message. The method may include outputting an indication from one or more classifiers that the sanitized text message is in the identified language. The indication may include a confidence score.

  In another aspect, the subject matter of this disclosure relates to a computer-implemented system for identifying a language in a message. The system includes a sanitizer module, a grouper module, and a language detector module. The sanitizer module obtains the text message and removes non-language characters from the text message so as to generate a sanitized text message. The grouper module detects at least one of alphabets and scripts that are present in the sanitized text message, and the grouper module detects that each sanitized text message is for one of a plurality of different languages. Performing an alphabet-based language detection test to determine a first set of scores representing likelihoods including alphabets, each of which a sanitized text message is scripted for one of a plurality of different languages. Operable to perform an operation including at least one of performing a script-based language detection test to determine a second set of scores representing the likelihood of including. The language detector module is configured to determine a language in the sanitized text message based on at least one of a first set of scores, a second set of scores, and a combination of the first and second sets of scores. Identify

  In various examples, non-language characters include pictographs and / or numbers. The combination may include interpolation between the first and second sets of scores. The grouper module is configured to generate language from a plurality of language detector modules based on at least one of a first set of scores, a second set of scores, and a combination of the first and second sets of scores. It may be operable to perform an operation that includes selecting a detector module. The language detector module may include a language detection method module. The language detection method module is a language detection test for the sanitized text message to generate a third set of scores each representing a likelihood that the sanitized text message includes one of a plurality of different languages. May be operable to perform operations including performing. The language detection test can include, for example, a dictionary-based language detection test, an n-gram language detection test, an alphabet-based language detection test, a script-based language detection test, a user language profile language detection test, or any combination thereof. .

  In some implementations, the language detector module includes an operation that includes processing a third set of scores using one or more classifiers to identify a language in the sanitized text message. A classifier module operable to execute is included. The one or more classifiers may include, for example, a supervised learning model, a partially supervised learning model, an unsupervised learning model, interpolation, or any combination thereof. The classifier module may be operable to perform operations that include outputting an indication that the sanitized text message is in the identified language. The indication may include a confidence score.

  In another aspect, the subject matter of this disclosure relates to an article. The article includes a non-transitory computer readable medium storing instructions, wherein the instructions, when executed by one or more computers, obtain a text message and generate a sanitized text message upon the computer. Performing and detecting operations including removing non-linguistic characters from the text message and detecting at least one of alphabets and scripts present in the sanitized text message (i Performing an alphabet-based language detection test to determine a first set of scores each representing a likelihood that the sanitized text message includes an alphabet for one of a plurality of different languages; ii) each sanitized text message At least one of performing a script-based language detection test to determine a second set of scores representing a likelihood that the script includes a script for one of a plurality of different languages. . The operation further determines the language in the sanitized text message based on at least one of a first set of scores, a second set of scores, and a combination of the first and second sets of scores. Including identifying.

  The example elements described with respect to a given aspect of the subject matter may be used in various examples of other aspects of the subject matter. For example, it is contemplated that a dependent claim feature that is dependent on one independent claim may be used in any of the devices, systems, and / or methods of the other independent claim.

FIG. 2 is a diagram of an example system for performing language detection. 2 is a flowchart of an exemplary method for detecting a language in a text message. 3 is a flowchart of an exemplary n-gram method for detecting a language in a text message. 3 is a flowchart of an exemplary dictionary-based method for detecting a language in a text message. 3 is a flowchart of an exemplary alphabet-based method for detecting a language in a text message. 3 is a flowchart of an exemplary script-based method for detecting a language in a text message. 3 is a flowchart of an exemplary user language profile method for detecting a language in a text message. FIG. 3 is a schematic diagram of an exemplary language detection module. FIG. 2 is a schematic diagram of an exemplary classifier module. 10 is a flowchart of an exemplary method for detecting a language in a text message using the language detection module of FIG. 7 and the classifier module of FIG. 2 is a flowchart of an exemplary method for detecting a language in a text message. 2 is a flowchart of an exemplary method for detecting a language in a text message. 2 is a flowchart of an exemplary method for detecting a language in a text message. 1 is a schematic diagram of an exemplary system for detecting a language in a text message. FIG. 2 is a flowchart of an exemplary method for detecting a language in a text message. 2 is a flowchart of an exemplary method for detecting a language in a text message.

DETAILED DESCRIPTION Generally, the language detection system and language detection method described herein is used when language information about a message (eg, keyboard information from a client device) does not exist, is in an abnormal format, or is not reliable Can be used to identify the language in the text message. The system and method improves the accuracy of language translation methods used to translate text messages from one language to another. Language translation generally requires that the source language be accurately identified. Otherwise, the resulting translation may be inaccurate.

  FIG. 1A shows an exemplary system 10 for detecting a language in a message, such as a text message or an audio message. Server system 12 provides message analysis and language detection functions. Server system 12 includes, for example, software components and databases that can be deployed at one or more data centers 14 at one or more geographic locations. The software components of the server system 12 may include a detection module 16, a classifier module 18, and a manager module 20. Software components may include subcomponents that may be executed on the same or different individual data processing devices. The database of server system 12 may include training data 22, dictionary 24, alphabet 26, script 28, and user profile information 30. A database may reside in one or more physical storage systems. Software components and data are further described below.

  An application, such as a web-based application, can be provided as an end user application to allow a user to provide messages to the server system 12. End user applications may be accessed over the network 32 by users of client devices such as personal computers 34, smartphones 36, tablet computers 38 and laptop computers 40. Other client devices are possible. The user message may be accompanied by information about the device used to create the message, such as information about the keyboard, client device and / or operating system used to create the message.

  FIG. 1A shows classifier module 18 and manager module 20 connected to a database (ie, training data 22, dictionary 24, alphabet 26, script 28 and user profile information 30), but classifier module 18 and / or manager Module 20 is not necessarily connected to some or all of the databases. In general, classifier module 18 may receive input from detection module 16 and manager module 20 may receive input from classifier module 18. Other inputs need not be received by the classifier module 18 and / or the manager module 20.

  FIG. 1B shows an exemplary method 100 for using the system 10 to detect a language in a message. The method 100 begins by receiving or obtaining a text message generated by a user (step 102). The text message is analyzed using one or more language detection methods (eg, by detection module 16) that each provide an indication of the language or languages that are present in the message (step 104). The output from the language detection method is then combined using one or more classifiers (eg, by the classifier module 18) that provide further indications of the languages present in the message (step 106). The one or more classifiers may include, for example, a supervised learning model, a partially supervised learning model, an unsupervised learning model, and / or interpolation. Other classifiers are possible. The output from the one or more classifiers is then used to determine (step 108) the language in the message (eg, using manager module 20).

  In some implementations, language instructions from the one or more classifiers may be selected by the manager module 20 according to the calculated confidence score and / or language domain. For example, the classifier may calculate a confidence score that indicates the confidence associated with the language prediction. Additionally or alternatively, certain classifier outputs may be selected according to the language domain associated with the user or message. For example, if the message originates from a computer gaming environment, a particular classifier output can be selected as providing the most accurate language prediction. Similarly, if the message originates from the context of a sport (eg, related to a sporting event), a different classifier output can be selected as appropriate by the sports language domain. Other possible language domains include, for example, news, parliamentary minutes, politics, health, travel, web pages, newspaper articles, and microblog messages. In general, a language detection method or combination of language detection methods (eg, from a classifier) is more accurate for a language domain when compared to other language domains. In some implementations, the domain can be determined based on the presence of words from the domain vocabulary in the message. For example, a domain vocabulary for computer gaming may include common slang used by gamers.

  Language detection methods used by detection module 16 may include, for example, n-gram methods (eg, byte n-gram methods), dictionary-based methods, alphabet-based methods, script-based methods, and user language profile methods. Other language detection methods are possible. Each of these language detection methods can be used to detect the language present in the message. The output from each method can be, for example, a set or vector of probabilities associated with each of the possible languages in the message. In some cases, two or more of the language detection methods can be performed simultaneously using parallel computing, which can significantly reduce computation time.

  In one implementation, the byte n-gram method uses byte n-grams instead of word or character n-grams to detect language. The byte n-gram method is preferably trained for a mixture of byte n-grams (eg, 1 ≦ n ≦ 4) using a naive Bayes classifier with a multinomial event model. The model preferably generalizes to data from different language domains so that the default configuration of the model is accurate across a diverse set of domains including newspaper articles, online games, web pages and microblog messages. Information about language identification tasks can be integrated from various domains.

  The task of achieving high accuracy can be relatively easy in the case of language identification in traditional text category classification settings where in-domain training data is available. This task can be more difficult when attempting to use learned model parameters for one language domain to classify or categorize data from separate language domains. This problem can be addressed by noting important features associated with language identification tasks. This can be based, for example, on a concept called information gain. Information gain was originally introduced for decision trees as a splitting criterion and was subsequently found useful for selecting features in text category classification. In one implementation, a detection score representing the difference in information gain for the domain and language may be calculated. A feature with a high detection score can provide information about the language without providing information about the domain. For simplicity, candidate feature sets can be pruned by feature selection based on term-frequency before the information gain is calculated.

  With reference to FIG. 2, an exemplary byte n-gram method 200 begins by training the method using training data 22. For example, the method can be trained on a mixture of byte n-grams using a naive Bayes classifier with a multinomial event model. Training data 22 is preferably collected for many different languages (step 202) and adjusted so that the amount of data available for each language is uniform. A small portion of the training data 22 is set as a separate test set (step 204). Once the training data 22 has been selected, the byte n-gram model is trained on the training data 22 with appropriate smoothing and backoff techniques (step 206). The input feature to the model is the byte stream from each input sentence, and the model knows the source language labels for these sentences so that it can learn the typical byte sequence for a given language. Adjust its parameters. The initially isolated test set is then used to predict language labels based on the trained model (step 208). The accuracy of the prediction gives the performance of this byte n-gram language identification system. In some cases, for each language domain, it is difficult to train such a byte n-gram system by collecting data across multiple languages for the domain. This difficulty arises from the lack of sufficient data per domain. Thus, these byte n-gram systems are typically trained to serve the general domain rather than any particular domain. The trained model can be compiled into a program with intermediate machine parameters (step 210). The program can function as a general-purpose language identification system.

  In general, dictionary-based language detection methods count the number of tokens or words belonging to each language by looking up words in the dictionary or other word lists associated with the language. The language with the most words in the message is selected as the best language. If there are multiple best languages, one of these best languages that is used more frequently or more commonly can be selected. The language dictionary can be stored in the dictionary database 24.

  FIG. 3 is a flowchart of an exemplary dictionary-based language detection method 300. A text message is provided (step 302) and a set of possible languages for the text message is identified using, for example, the detection module 16 (step 304). A first possible language is then selected from the set (step 306). The words in the text message that are present in the dictionary for the possible languages are counted (step 308). If yet another possible language from the set has not yet been considered (step 310), a new potential language is selected (step 312) and step 308 is repeated. Once all possible languages from the set are considered, the language with the most words in the text message can be identified as the language in the message (step 314). Alternatively or additionally, the method can be used to calculate the likelihood that a language is present in the message for each language in the set. For example, the output from the dictionary-based method can be a vector of probabilities for each language in the set.

  To ensure the accuracy of dictionary-based language detection methods, especially in the case of short sentences, a dictionary containing informal words or informal words or chat words (eg abbreviations, acronyms, slang and profane words) It is preferable to use it. Informal words are commonly used in short text communication and chat rooms. As new informal words appear and are used, the dictionary is preferably augmented to include informal words on a progressive basis.

  Alphabet-based methods are generally based on character counts for each language alphabet and rely on the observation that many languages have different sets of unique alphabets or characters. For example, each of Russian, English, Korean, and Japanese uses a different alphabet. While alphabet-based methods may not be able to accurately identify some languages (for example, languages that use a similar alphabet, such as Latin), alphabet-based methods generally quickly Can be detected. In some cases, as discussed herein, it is preferred to use alphabet-based methods in combination with one or more other language detection methods (eg, using a classifier). The language alphabet may be stored in the alphabet database 26.

  FIG. 4 is a flowchart of an exemplary alphabet-based language detection method 400. A text message is provided (step 402) and a set of possible languages for the text message is identified using, for example, the detection module 16 (step 404). A first possible language is then selected from the set (step 406). The characters in the text message that are present in the alphabet for the possible language are counted (step 408). If additional potential languages from the set have not yet been considered (step 410), a new potential language is selected (step 412) and step 408 is repeated. Once all possible languages from the set are considered, the language with the most characters in the text message can be identified as the language in the message (step 414). Alternatively or additionally, an alphabet-based method can be used for each language in the set to calculate the likelihood that the language is present in the message. For example, the output from the alphabet-based method can be a vector of probabilities for each language in the set.

  In general, script-based language detection methods determine the character count for each possible script (eg, Latin script, CJK script, etc.) present in a message. Script-based methods rely on the observation that different languages can use different scripts, such as Chinese and English. The method preferably uses a mapping that maps the script to a list of languages that use the script. For example, the mapping may consider UNICODE values for characters or symbols present in the message, and these UNICODE values may be mapped to the corresponding language or set of possible languages for the message. Language scripts and UNICODE values or ranges may be stored in the script database 28.

  Referring to FIG. 5, in an example script-based method 500, a text message is provided (step 502), and scripts present in the message are identified using, for example, the detection module 16 (step 504). . Thereafter, the number of characters corresponding to each script is counted (step 506). The script with the largest number of characters is considered the best script (step 508) and the language corresponding to the best script is identified (step 510). If the best script corresponds to only one language, that language may be considered the best language. In another aspect, if the best script corresponds to more than one language, additional language detection methods can be used to perform further detection. In some implementations, the output from the script-based method is a set of probabilities (eg, in vector form) for each possible language in the message.

  The user language profile-based method uses a user profile information database 30 that stores historical messages sent by various users. The language of these stored messages uses, for example, one or more other language detection methods (eg, byte n-gram methods) described herein to identify the language used by each user. Is detected. For example, if all of a user's previous messages are in Spanish, the language profile for that user may indicate that the user's preferred language is Spanish. Similarly, if a user's previous message is a mixture of different languages, the language profile for that user may indicate the probability associated with the different languages (eg, 80% for English, 15% for French, Spanish for 5%). In general, user language profile-based methods address language detection problems associated with very short messages that often do not have sufficient information to make accurate language decisions. In such a case, the user's language preference can be used to predict the language in the user's message by assuming that the user will continue to use the previously used language.

  Referring to FIG. 6, an exemplary user language profile detection method 600 starts by storing a user's previous message (step 602) and detecting the language present in the stored message (step 604). To do. The frequency at which different languages appear in the user's message is determined (step 606), and the usage probabilities for such languages are output (step 608).

  With reference to FIG. 7, various language detection methods may be incorporated into the detection module 16. A text message may be input to the detection module 16 and one or more of the language detection methods may identify the language in the message. For example, each language detection method may provide a vector of probabilities. Each probability in the vector is associated with a possible language in the message and represents the likelihood that the message is a given language. Due to the different methods used and the information available in the messages, the probability from each language detection method may not be consistent. The detection module 16 includes, for example, an n-gram module 702 for performing an n-gram detection method (eg, byte n-gram detection method 200), a dictionary base module 704 for executing a dictionary-based method 300, and an alphabet-based May include an alphabet-based module 706 for performing the method 400, a script-based module 708 for performing the script-based method 500, and a user language profile module 710 for performing the user language profile method 600; Or it can be used. Additional or alternative language detection methods can be incorporated into the detection module 16 as desired. Some known methods include, for example, using word level n-grams, Markov models and predictive modeling techniques.

  Outputs from various language detection methods in the detection module 16 can be combined using the classifier module 18. With reference to FIG. 8, the classifier module 18 may include an interpolation module 802, a support vector machines (SVM) module 804 and a linear SVM module 806.

  Interpolation module 802 is used to perform linear interpolation of results from two or more language detection methods. For illustrative purposes, the language of the text message can be determined by interpolating between the results from the byte n-gram method and the dictionary-based method. For the chat message “lol gtg”, the byte n-gram method has an English likelihood of 0.3, a French likelihood of 0.4, and a Polish likelihood of 0.3. (E.g., the output from the byte n-gram method can be {en: 0.3, fr: 0.4, pl: 0.3})). The dictionary-based method may determine that the likelihood of English is 0.1, the likelihood of French is 0.2, and the likelihood of Polish is 0.7 (eg, the output is {en : 0.1, fr: 0.2, pl: 0.7}). To interpolate between the results of these two methods, the output from the byte n-gram method is multiplied by a first weight and the output from the dictionary-based method is multiplied by a second weight. The first and second weights are added to 1. The weighted outputs from the two methods are then added together. For example, if a byte n-gram result is given a weight of 0.6, the dictionary-based result is given a weight of 0.4, and the interpolation between these two methods is {en: 0.3 , Fr: 0.4, pl: 0.3} × 0.6 + {en: 0.1, fr: 0.2, pl: 0.7} × 0.4 = {en: 0.22, fr: 0.32, pl: 0.46}. Other weightings are possible.

  In general, the optimal weight for interpolating between two or more values can be determined numerically by trial and error. Different weights can be tried to identify the best set of weights for a given set of messages. In some cases, the weight can be a function of the number of words or characters in the message. Alternatively or additionally, the weight may depend on the language domain of the message. For example, the optimal weight for a gaming environment may differ from the optimal weight for a sports environment. For the combination of the byte n-gram method and the dictionary-based method, good results may be obtained with a weight of 0.1 for the byte n-gram method and a weight of 0.9 for the dictionary-based method.

  The SVM module 804 can be a supervised learning model that analyzes language data and recognizes language patterns, or can include the supervised learning model. For example, the SVM module 804 can be a multi-class SVM classifier. In the case of an English SVM classifier, the feature vector can be a concatenation of the above two distributions (ie {en: 0.3, fr: 0.4, pl: 0.3, en: 0.1 , Fr: 0.2, pl: 0.7}). The SVM classifier is preferably trained with labeled training data. The trained model functions as a predictor for input. The feature selected in the case of language detection can be, for example, a sequence of bytes, words or phrases. The input training vector can be mapped to a multidimensional space. The SVM algorithm may then use the kernel to identify the optimal separation hyperplane between these dimensions, which gives the algorithm the discriminatory ability to predict the language (in this case). . The kernel can be, for example, a linear kernel, a polynomial kernel, or a radial basis function (RBF) kernel, although other suitable kernels are possible. The preferred kernel for the SVM classifier is the RBF kernel. After training the SVM classifier using the training data, the classifier can be used to output the best language of all possible languages.

  The training data can be, for example, output vectors from different language detection methods, or can include such output vectors, eg, for multiple messages with different message lengths, language domains and / or languages, the correct language Or may include such instructions. The training data may include a number of messages where the language in each message is known.

  The linear SVM module 806 may be a large scale linear classifier or may include the linear classifier. An SVM classifier with a linear kernel may perform better than other linear classifiers such as linear regression. The linear SVM module 806 differs from the SVM module 804 at the kernel level. The polynomial model may perform better than the linear model, and vice versa. The optimal kernel may depend on the language domain of the message data and / or the nature of the data.

  Other possible classifiers used by the systems and methods described herein include, for example, decision tree learning, association rule learning, artificial neural networks, inductive logic programming, random forest, clustering, Bayesian networks , Reinforcement learning, display learning, similarity learning and metric learning, and sparse dictionary learning. One or more of these classifiers or other classifiers may be integrated into the classifier module 18 and / or may form part of the classifier module 18.

  Referring to FIG. 9, the example method 900 uses the detection module 16, the classifier module 18, and the manager module 20 to detect the language in the message. The message is provided or delivered to the detection module 16 (step 902). The message may be accompanied by information about the message and / or information about the user who generated the message. The information may relate to, for example, an identification number for the user, information about the keyboard used by the user to generate the message, and / or an operating system that controls software used by the user to generate the message. Information can be included. For example, the message may be accompanied by data indicating that the user uses a French keyboard to generate the message and the user's operating system is English.

  In the detection module 16, one or more language detection methods are used to detect the language of the message (step 904). Each method used by the detection module 16 may output a prediction regarding the language present in the message. The prediction can be in the form of a vector that contains the probabilities for each possible language that may be present in the message.

  Thereafter, the output from the detection module 16 is delivered to the classifier module 18 and results from two or more language detection methods may be combined (step 906). Various combinations of results from language detection methods can be obtained. In one example, the results from the byte n-gram method and the dictionary-based method are combined in the classifier module 18 by interpolation. In another example, SVM combinations or classifications are performed on results from byte n-gram methods, dictionary-based methods, alphabetic methods, and user profile methods. Alternatively or additionally, the combination may include or take into account results from script-based methods. Yet another example includes large linear combinations of byte n-gram methods, language profile methods and dictionary methods. In general, however, results from any two or more of the above language detection methods may be combined in the classifier module 18.

  The method 900 uses the manager module 20 to select an output from a particular classifier (step 908). The output may be selected based on, for example, a confidence score calculated by the classifier, expected language detection accuracy, and / or language domain for the message. The best language is then selected from the selected classifier output (step 910).

  In some cases, the systems and methods described herein select a language detection method according to the length of the message. For example, referring to FIG. 10, method 1000 includes receiving or providing a message that may include information regarding the language of the keyboard used to generate the message (step 1002). If the message is larger than the threshold length (any suitable threshold length is possible, eg 25 bytes or 25 characters) (step 1004), then the byte n-gram method (or other method or combination of methods) ) May be used to detect the language (step 1006). The language of the message can then be selected based on the results from the byte n-gram method (step 1008). In another aspect, if the message is less than or equal to the threshold length, the system may determine whether a keyboard language is available (step 1010). If a keyboard language is available, the message language may be selected to be the same as the keyboard language (step 1012). Alternatively, if the keyboard language is not available, the method 1000 may again consider the message length. For example, if the length of the message is less than the second threshold (any suitable second threshold is possible, eg 4 bytes or 4 characters) (step 1014), the dictionary based method Can be used to detect (step 1016) and select a language. In another aspect, if the message length is greater than the second threshold, a byte n-gram method (or other method or combination of methods) may be used to detect the language in the message (steps). 1018). The results from the byte n-gram method and the dictionary-based method can be combined (eg, using an interpolator or other classifier), and the language of the message can be determined based on the combination (step 1020).

  FIG. 11 is an exemplary method 1100 for identifying a language in a text message. A text message generated on the user's client device is received or provided (step 1102). Alphabet-based methods and / or script-based methods are used to determine the alphabet and / or script associated with the text message (step 1104). Candidate languages associated with the alphabet and / or script are identified. If the candidate language is a language having a unique alphabet and / or script (eg, Russian, Arabic, Hebrew, Greek, Chinese, Taiwanese, Japanese or Korean, etc.) (step 1106) The candidate language is determined to be the language of the text message (step 1108).

  In another aspect, if the candidate language is not a language with a unique alphabet and / or script, the length of the text message is evaluated. The text message is used by the client device (step 1110) when the message length is less than the threshold length (any suitable threshold length is possible, eg 4 bytes or 4 characters) If the keyboard language is included or accompanied by the keyboard language, the message language is selected to be the keyboard language (step 1112).

  Alternatively, if the message length is greater than the threshold length or the keyboard language is not available, the message is processed in an n-gram method (eg, a byte n-gram method) to allow for the possibility of the text message. A first set of languages is identified (step 1114). In addition, the message is processed in a dictionary-based manner to identify a second set of possible languages for the text message (step 1116). If a user language profile exists for the user (step 1118), a user language profile is obtained (step 1120) and a first set of possible languages (eg, using an SVM classifier or a large linear classifier). And combined with a second set of possible languages, thereby obtaining a first combination of possible languages (step 1122). Thereafter, the language of the text message is selected based on the first combination of possible languages (step 1124). In another aspect, if a user language profile is not available, a first set of possible languages and a second set of possible languages are used (eg, using a linear interpolator or other classifier). ) To obtain a second combination of possible languages (step 1126). Finally, the language of the text message is selected based on the second possible combination of languages (step 1128).

  In some cases, language detection is performed by combining outputs from multiple language detection methods in two or more steps. For example, the first step may be, for example, Chinese (cn), Japanese (ja), Korean (ko), Russian (ru), Hebrew (he), Greek (el) and Arabic (ar) The alphabet-script-based method may be used to detect special languages that use a unique alphabet or script. Alphabet-script-based methods refer, for example, to using one or both of alphabet-based methods and script-based methods. If necessary, the second step may use a plurality of detection methods (eg, byte n-gram method, user language profile based method, and dictionary) to detect other languages (eg, Latin) in the message. A combination of base methods) may be used.

  In one example, a message provided or received for language detection is a number, letter or image that is not specific to any particular language and / or recognizable to any user, regardless of language preference (e.g., Emoticon or emoji). The systems and methods described herein can ignore such characters or images when performing language detection, and can ignore messages that contain only such characters or images. Alternatively or additionally, the system and method may remove such characters or images from the message before performing language detection. The process of removing unwanted characters or images from a message may be referred to herein as “sanitize the message”. With the sanitizing process, the detection time can be faster and / or the language detection accuracy can be improved.

  FIG. 12 is a flowchart of an exemplary method 1200 for detecting the language of a message. The method uses detection module 16, classifier module 18 and manager module 20 to identify the most likely or best language 1202 for a given input message 1204. Input message 1204 may be accompanied by information about the user or information about the system used to generate the message. For example, the input message 1204 may be used to generate a user identification number (or other user identifier), information about the keyboard (eg, keyboard language) used to generate the message, and / or the message. May be accompanied by information about the operating system (eg the language of the operating system).

  In the exemplary method 1200 shown, the detection module 16 includes ten different language detection methods. Three of the language detection methods in detection module 16 are byte n-gram A 1206, byte n-gram B 1208, and byte n-gram C 1210, which are all byte n-gram methods and detect different sets or numbers of languages. Can be configured as follows. For example, byte n-gram A 1206 can be configured to detect 97 languages, byte n-gram B 1208 can be configured to detect 27 languages, and byte n-gram C 1210 can be configured to detect 20 languages. Two of the language detection methods in detection module 16 are dictionary A1212 and dictionary B1214, both of which are dictionary-based methods and can be configured to detect different sets or numbers of languages. For example, dictionary A1212 may be configured to detect 9 languages and dictionary B1214 may be configured to detect 10 languages. Two of the language detection methods in detection module 16 are language profile A 1216 and language profile B 1218, which are user language profile methods and can be configured to detect different sets or numbers of languages. For example, language profile A 1216 may be configured to detect 20 languages and language profile B 1218 may be configured to detect 27 languages. Two of the language detection methods in detection module 16 are alphabet A1220 and alphabet B1222, which are alphabet-based methods and can be configured to detect different sets or numbers of languages. For example, alphabet A1220 may be configured to detect 20 languages and alphabet B1222 may be configured to detect 27 languages. The detection module 16 further includes a script-based language detection method 1224.

  Outputs from different language detection methods in the detection module 16 are combined and processed by the classifier module 18. For example, interpolation classifier 1226 combines the output from byte n-gram B1208 and dictionary B1214. The weight for interpolation is, for example, 0.1 for byte n-gram B1208 and 0.9 for dictionary B1214. The classifier module 18 may further use an SVM classifier 1228 that combines the output from byte n-gram C1210, dictionary B1214, language profile B1218, and alphabet B1222. The classifier module 18 may further use the first combination 1230 of the script-based method 1224 and the SVM classifier combination of byte n-gram C1210, dictionary A1212, language profile A1216, and alphabet A1220. Further, classifier module 18 may use a second combination 1232 of script-based method 1224 and a linear SVM classifier combination of byte n-gram C1210, dictionary A1212, and language profile A1216. Although FIG. 12 shows a combination of the specific language detection test, classifier and detection test output used in the classifier module 18, other language detection tests, classifiers and / or combinations may be used.

  For both the first combination 1230 and the second combination 1232, script-based methods 1224 and classifiers may be used in a tiered approach. For example, the script-based method 1224 can be used to quickly identify languages with unique scripts. If such a language is identified in message 1204, it may not be necessary to use an SVM classifier in the first combination 1230 or a linear SVM classifier in the second combination.

  In general, manager module 20 may select a particular language detection method, classifier, and / or combination of detection method outputs to identify the language in message 1204. The manager module 20 may make the selection according to the language domain or according to the language expected for the message. Manager module 20 may select a particular classifier according to a confidence score determined by the classifier. For example, manager module 20 may select the output from the most reliable classifier in the prediction.

  In one implementation, the systems and methods described herein are suitable for making language detection available as a service to multiple users. Such services are enabled or enhanced by the speed with which systems and methods identify languages and the ability of systems and methods to handle multiple identification technologies at runtime based on service requests from various clients. .

  Referring to FIG. 13, in one example, the language detection system 1300 obtains or receives a text message 1302 and identifies a language present in the text message 1302, a sanitizer module 1304, a grouper module 1306, and One or more language detector modules are used. In general, the sanitizer module 1304 prepares the text message 1302 for language detection by deleting certain characters or other unnecessary elements from the text message 1302. For example, the sanitizer module 1304 may include one or more numbers (eg, “1”, “3”, “15”, “249”, etc.), one or more pictograms (eg, an image and / or “:-D” or And / or one or more other non-language characters or elements that may not be useful for language detection. Examples of the one or more other non-linguistic characters or elements include, for example, certain punctuation marks (eg, periods or commas), extra spaces, and / or carriage returns. In an alternative example, unnecessary elements are not removed from the text message 1302, but are simply ignored or avoided during subsequent language detection. In such cases, the sanitizer module 1304 may be used to flag or identify unwanted elements. References to “text message 1302” in the following paragraphs are intended to cover, for example, text messages with unnecessary elements removed or ignored. Herein, in various examples, unnecessary elements are referred to as “non-language characters”.

  In general, the grouper module 1306 is used to perform an initial classification of languages in the text message 1302 and one or more subsequent languages based on the initial classification to make a final determination of the languages in the text message 1302. Select the detection method. In the preferred example, the grouper module 1306 performs the initial classification by detecting alphabets and / or scripts that are present in the text message 1302. The alphabets and / or scripts can be detected using, for example, alphabet-based methods and / or script-based methods described herein. In some cases, the alphabet-based method may determine a first set of scores for the text message 1302. Each score represents the probability or likelihood that the alphabet is for one of a plurality of different languages. Grouper module 1306 may detect the alphabet in text message 1302 based on the highest score from the first set of scores. Similarly, the script-based method may determine a second set of scores for text message 1302. Each score represents the probability or likelihood that the script is for one of a plurality of different languages. Grouper module 1306 may detect a script in text message 1302 based on the highest score from the second set of scores. Alternatively or additionally, the grouper module 1306 may detect alphabets and / or scripts in the text message 1302 (eg, using an interpolator or other classifier) and alphabet based methods and script bases. Results or scores from the methods can be combined. Once the alphabet and / or script is detected, the grouper module 1306 may select a language detector module to use to make a final determination of the language in the text message 1302, as described herein below. select. Grouper module 1306 may pass results from alphabet-based methods and / or script-based methods or other information (eg, one or more scores) to the selected language detector module.

  In the example shown, language detection system 1300 may include or utilize language detector modules such as alphabet-distinguishable language detector 1308, Cyrillic language detector 1310, Latin language detector 1312 and backoff language detector 1314. Can do. However, other additional or alternative language detector modules can be included or utilized. Each of these language detector modules 1308, 1310, 1312 and 1314 may include a detection method module and a classifier module. For example, the alphabet distinguishable language detector 1308 can include a detection method module 1316 and a classifier module 1318, the Cyrillic language detector 1310 can include a detection method module 1320 and a classifier module 1322, and the Latin language detector 1312 , Detection method module 1324 and classifier module 1326, and backoff language detector 1314 may include detection method module 1328 and classifier module 1330.

  In general, the detection method modules 1316, 1320, 1324, and 1328 include or utilize one or more language detection methods. The one or more language detection methods may be, for example, an n-gram method (eg, byte n-gram method), a dictionary-based method, an alphabet-based method, a script-based method, and / or a user language profile method, or May be included. Other language detection methods are also conceivable. Detection method modules 1316, 1320, 1324 and 1328 may use a language detection method that produces an output that provides an indication of the language present in text message 1302. The output may be, for example, one or more scores representing the likelihood that the text message 1302 is in one or more languages, or may include the one or more scores. In some cases, the language in text message 1302 is determined directly from the output of one of detection method modules 1316, 1320, 1324, or 1328. Alternatively or additionally, the language in text message 1302 may be determined from the output of one of classifier modules 1318, 1322, 1326, or 1330. In general, each classifier module 1318, 1322, 1326, or 1330 processes the output from the corresponding detection method module 1316, 1320, 1324, or 1328 to provide further indications of the language present in the text message. provide. The classifier modules 1318, 1322, 1326, and 1330 may preferably use or include one or more classifiers such as supervised learning models, partially supervised learning models, unsupervised learning models, and / or interpolation.

  For example, if alphabets and / or scripts detected by the grouper module 1306 are associated with one or more alphabetic distinguishable languages, the grouper module 1306 selects the alphabetic distinguishable language detector 1308. In general, an alphabet distinguishable language is a language that has a unique alphabet and / or a unique script, and once the alphabet and / or script for that language is detected, the language in text message 1302 can be determined. It is. Examples of alphabet distinguishable languages are, for example, simplified Chinese (cn), traditional Chinese (tw), Japanese (ja), Arabic (ar), Hebrew (he), Greek (el), Korean (Ko) and Thai (th). In various cases, grouper module 1306 may alphabetize results (eg, one or more scores and probabilities, detected alphabets, and / or detected scripts) from alphabet-based methods and / or script-based methods. Pass to the distinguishable language detector 1308. Alternatively or additionally, if the grouper module 1306 does not pass such results to the alphabet distinguishable language detector 1308, the detection method module 1316 may detect the alphabet and / or script in the text message 1302. Alphabet-based methods and / or script-based methods may be performed. Alphabet distinguishable language detector 1308 may determine the language in text message 1302 once the alphabet and / or script is detected. In some cases, such a determination can be made using the classifier module 1318 to process any output from the detection method module 1316.

  In some examples, the grouper module 1306 selects the Cyrillic language detector 1310 if the alphabets and / or scripts detected by the grouper module 1306 are associated with one or more Cyrillic languages. Examples of Cyrillic languages include, for example, Bulgarian (bg), Ukrainian (uk) and Russian (ru). To determine a particular Cyrillic language in the text message 1302, the detection method module 1320 can include one or more language detection methods described herein, such as a byte n-gram method and / or a dictionary-based method. Or can be utilized. In a preferred example, the detection method module 1320 utilizes a dictionary-based method that can use one or more dictionaries specific to the Cyrillic language. The dictionary-based method may count the number of tokens or words belonging to one or more Cyrillic languages in the text message 1302 by looking up words in the one or more dictionaries. In some examples, the Cyrillic language that has the most tokens or words in the text message 1302 is determined to be the language in the text message 1302. Alternatively or additionally, detection method module 1320 may provide output from one or more language detection methods (eg, dictionary-based methods) to classifier module 1322, which may be a text message. The output may be processed to determine the language in 1302. For example, the classifier module 1322 may receive a set of scores from the detection method module 1320 and may determine the Cyrillic language in the text message 1302 by identifying the language with the highest score.

  In some cases, the grouper module 1306 selects the Latin language detector 1312 if the alphabet and / or script detected by the grouper module 1306 is associated with one or more Latin languages. Examples of Latin languages are English (en), French (fr), Spanish (es), German (de), Portuguese (pt), Dutch (nl), Polish (pl), Italian ( it), Turkish (tr), Catalan (ca), Czech (cs), Danish (da), Finnish (fi), Hungarian (hu), Indonesian (id), Norwegian (no) , Romanian (ro), Slovak (sk), Swedish (sv), Malay (ms), Vietnamese (vi). To determine a particular Latin language in the text message 1302, the detection method module 1324 may include or utilize one or more language detection methods described herein. In a preferred example, the detection method module 1324 includes or utilizes a byte n-gram method and / or a dictionary-based method. Output from one or both of these preferred methods may be processed or combined using classifier module 1326 to determine a particular Latin language in text message 1302. For example, n-gram methods and dictionary-based methods may each output a set of scores. Each score represents the likelihood that the text message 1302 is one of a plurality of different Latin languages. The classifier module 1326 may process the set of scores using, for example, one or more classifiers and / or interpolation techniques described herein to determine the Latin language in the text message 1302.

  In some examples, the grouper module 1306 selects the backoff language detector 1314 to detect the language in the text message 1302. The backoff language detector 1314 may be selected, for example, if the grouper module 1306 does not select the alphabet distinguishable language detector 1308, the Cyrillic language detector 1310, or the Latin language detector 1312. For example, such a situation may occur when the grouper module 1306 fails to detect an alphabet and / or script associated with an alphabetic language, a Cyrillic language, or a Latin language. When backoff language detector 1314 is selected, detection method module 1328 and / or classifier module 1330 may be used to identify the language in text message 1302. Language detection methods used by detection method module 1328 include, for example, n-gram methods (eg, byte n-gram methods), dictionary-based methods, alphabet-based methods, script-based methods, user language profile methods, and any of its Or a combination thereof. The particular classifier used by classifier module 1330 can be or include, for example, a supervised learning model, a partially supervised learning model, an unsupervised learning model, interpolation and / or any combination thereof. Other language detection methods and / or classifiers can be used. In general, the backoff language detector 1314 may use any of the language detection methods and classifiers described herein. The backoff language detector 1314 is preferably flexible and may be configured to include or use new detection methods and / or new combinations of detection methods. This is because such new methods and / or combinations will be developed or available. In some cases, by exercising the back-off language detector 1314, the language detection system 1300 can provide a valid output rather than a NULL output.

  For illustrative purposes, FIG. 14 is a flowchart of an exemplary method 1400 for identifying a language in a text message. Method 1400 includes receiving a text message (step 1402) and detecting at least one of a Latin alphabet and a Latin script in the text message (step 1404). A dictionary-based language detection test is performed to determine a first set of scores (step 1406). Each score in the first set of scores represents the likelihood that the text message is one of a plurality of different Latin languages. An n-gram language detection test is performed to determine a second set of scores (step 1408). Each score in the second set of scores represents the likelihood that the text message is one of a plurality of different Latin languages. The first and second sets of scores are combined (step 1410) using, for example, one or more classifiers and / or interpolation techniques. The language in the text message is identified based on the combination (step 1412).

  FIG. 15 is a flowchart of an exemplary method 1500 for identifying a language in a text message. A text message is obtained (step 1502), non-language characters are removed from the text message (step 1504), and a sanitized text message is generated. The alphabet and / or script may include (i) an alphabet-based language detection test that determines a first set of scores (step 1506) and (ii) a script-based language detection test that determines a second set of scores ( Detected in the sanitized text message by performing at least one of step 1508). Each score in the first set of scores represents the likelihood that the sanitized text message contains an alphabet for one of a plurality of different languages. Each score in the second set of scores represents the likelihood that the sanitized text message contains a script for one of a plurality of different languages. The language in the sanitized text message is identified based on the first set of scores, the second set of scores, and / or the combination of the first and second sets of scores (step 1510).

  Embodiments of the subject matter and operations described herein can be implemented in digital electronic circuitry, or include computer software, firmware, or hardware that includes the structures disclosed herein and equivalents of those structures. Can be implemented in hardware or in one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs. That is, it may be implemented as one or more modules of computer program instructions encoded on a computer storage medium for execution by the data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions encode information for transmission to a receiving device suitable for execution by a data processing device, eg, a machine-generated electrical signal, optical signal or electromagnetic signal. Can be encoded on an artificially generated propagation signal generated as described above. The computer storage medium may be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Further, although a computer storage medium is not a propagated signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may further be or be included in one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

  The operations described herein may be implemented as operations performed by a data processing device on data stored on one or more computer readable storage devices or received from other sources.

  The term “data processing device” encompasses all types of devices, devices and machines for processing data, including by way of example a programmable processor, a computer, a system on chip, or a plurality or combination of the foregoing. To do. The device may include special purpose logic such as, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). In addition to hardware, the apparatus may further include code that creates an execution environment for the appropriate computer program. Such code includes, for example, code that forms a processor firmware, protocol stack, database management system, operating system, cross-platform runtime environment, virtual machine, or one or more combinations thereof. The device and execution environment may implement a variety of different computing model infrastructures such as web services, distributed computing and grid computing infrastructure.

  A computer program (also known as a program, software, software application, script or code) can be written in any form of programming language, including compiled or interpreted language, declarative language or procedural language, stand-alone program Or in any form including as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system, but need not correspond to a file. A program may be another program or data (e.g., in a single file dedicated to the program, or in a plurality of coordinated files (e.g., files that store one or more modules, subprograms or portions of code)). One or more scripts stored in a markup language document). A computer program can be deployed to run on one computer, or run on multiple computers located at one location or distributed across multiple locations and interconnected by a communications network Can be deployed as

  The processing and logic flows described herein execute one or more computer programs such that one or more programmable processors perform actions by operating on input data and generating output. Can be done by. The processing and logic flow may be performed by special purpose logic circuits such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits), and the device may also be realized by the special purpose logic circuits.

  Processors suitable for the execution of computer programs include, by way of example, both general and special purpose microprocessors and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Generally, a computer further includes or includes data from one or more mass storage devices, such as a magnetic disk, magneto-optical disk, optical disk, or solid state drive, for storing data. And / or transfer data to the one or more mass storage devices, or both. However, the computer need not have such a device. In addition, the computer may be, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (eg, a universal serial bus (USB) flash drive) Embedded in another device. Suitable devices for storing computer program instructions and data include, by way of example, semiconductor memory devices such as EPROM, EEPROM, flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD- Includes all forms of non-volatile memory, media and memory devices including ROM and DVD-ROM discs. The processor and memory can be supplemented by, or integrated with, special purpose logic circuitry.

  In order to provide user interaction, embodiments of the subject matter described herein include a display device for displaying information to a user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for example. It can be implemented on a computer having a keyboard that can be used by a user to provide input to the computer and a pointing device such as a mouse, trackball, touchpad or stylus. Other types of devices can be used as well to provide user interaction. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be any form including acoustic input, speech input, or tactile input. Can be received at. Furthermore, the computer may interact with the user by sending the document to a device used by the user and by receiving the document from the device. For example, the user may be interacted with by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

  Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, for example, as a data server, or can be implemented in a computing system that includes a middleware component, for example, as an application server, or Can be implemented in a computing system that includes a front-end component such as, for example, a client computer having a graphical user interface or web browser that allows a user to interact with an implementation of the subject matter described herein, or such back A computing system that includes any combination of end, middleware, or front-end components May be Oite realized. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), an internetwork (eg, the Internet), and a peer-to-peer network (eg, an ad hoc peer-to-peer network).

  The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is caused by computer programs that are executed on the respective computers and have a client-server relationship with each other. In some embodiments, the server sends data (eg, an HTML page) to the client device (eg, for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Send. Data generated at the client device (eg, a result of user interaction) may be received from the client device at the server.

  This specification includes details of many specific implementations, but these should not be construed as limitations on the scope of any invention or what may be claimed, but rather specific embodiments of a particular invention. Should be construed as an explanation of the features specific to Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Further, in the above, features may be described as acting in a combination and initially claimed as such, but one or more features from the claimed combination may in some cases be Combinations can be deleted from the combinations, and the claimed combinations can be directed to sub-combinations or variations of sub-combinations.

  Similarly, operations are shown in a particular order in the drawings, but this requires that such operations be performed in the particular order shown or sequential order to achieve the desired result. Should not be construed as being, or should be understood as requiring that all indicated actions be performed. In certain situations, multitasking and parallel processing may be advantageous. For example, parallel processing can be used to perform multiple language detection methods simultaneously. Further, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems are generally It should be understood that it can be integrated together into a single software product or packaged into multiple software products.

  Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Also, the processes shown in the accompanying drawings do not necessarily require the particular order shown, or sequential order, to achieve the desired result. In certain implementations, multitasking and parallel processing may be advantageous.

  Claims

Claims (20)

A computer-implemented method for identifying the language in a message,
Getting a text message,
Removing non-language characters from the text message so as to generate a sanitized text message;
Detecting at least one of an alphabet and a script present in the sanitized text message;
The detecting is
(I) performing an alphabet-based language detection test to determine a first set of scores each representing a likelihood that the sanitized text message includes an alphabet for one of a plurality of different languages; When,
(Ii) A script-based language detection test is performed to determine a second set of scores each representing the likelihood that the sanitized text message includes a script for one of the plurality of different languages. And at least one of
The method further includes:
Based on at least one of the first set of scores, the second set of scores, and the combination of the first and second sets of scores, the language in the sanitized text message is A method comprising identifying.   The method of claim 1, wherein the non-language characters include at least one of a pictograph and a number.   The method of claim 1, wherein the combination includes an interpolation between the first and second sets of scores.   Identifying the language in the sanitized text message to generate a third set of scores each representing a likelihood that the sanitized text message includes one of a plurality of different languages. The method of claim 1, comprising performing a language detection test on the sanitized text message.   The language detection test is based on the at least one of the first set of scores, the second set of scores, and the combination of the first and second sets of scores. The method of claim 4, selected from language detection tests.   The method of claim 4, wherein the language detection test includes a language detection method and one or more classifiers.   The language detection method includes at least one of a dictionary-based language detection test, an n-gram language detection test, an alphabet-based language detection test, a script-based language detection test, and a user language profile language detection test. The method of claim 6.   The method of claim 6, wherein the one or more classifiers include at least one of a supervised learning model, a partially supervised learning model, an unsupervised learning model, and interpolation.   5. The method of claim 4, comprising processing the third set of scores using one or more classifiers to identify the language in the sanitized text message.   The method of claim 9, comprising outputting an indication from the one or more classifiers that the sanitized text message is the identified language, wherein the indication includes a confidence score. A computer-implemented system for identifying the language in a message,
A sanitizer module, wherein the sanitizer module obtains a text message and removes non-language characters from the text message to generate a sanitized text message;
The system further includes:
A grouper module that detects at least one of alphabets and scripts present in the sanitized text message, the grouper module comprising:
Performing an alphabet-based language detection test to determine a first set of scores each representing the likelihood that the sanitized text message includes an alphabet for one of a plurality of different languages;
Performing a script-based language detection test to determine a second set of scores each representing a likelihood that the sanitized text message includes a script for one of the plurality of different languages. Is operable to perform an action including at least one of
The system further includes:
A language detector module, wherein the language detector module is at least one of the first set of scores, the second set of scores, and the combination of the first and second sets of scores. Identifying the language in the sanitized text message based on   The system of claim 11, wherein the non-language characters include at least one of pictographs and numbers.   The system of claim 11, wherein the combination includes an interpolation between the first and second sets of scores. The grouper module is
Based on the at least one of the first set of scores, the second set of scores, and the combination of the first and second sets of scores, from a plurality of language detector modules The system of claim 11, wherein the system is operable to perform an operation that includes selecting a language detector module.   The language detector module is configured for the sanitized text message to generate a third set of scores each representing a likelihood that the sanitized text message includes one of a plurality of different languages. The system of claim 11, comprising a language detection method module operable to perform operations including performing a language detection test.   The language detection test includes at least one of a dictionary-based language detection test, an n-gram language detection test, an alphabet-based language detection test, a script-based language detection test, and a user language profile language detection test. The system according to claim 15.   The language detector module performs an operation including processing the third set of scores using one or more classifiers to identify the language in the sanitized text message. The system of claim 15, comprising a classifier module operable on the system.   The system of claim 17, wherein the one or more classifiers include at least one of a supervised learning model, a partially supervised learning model, an unsupervised learning model, and interpolation.   The classifier module is operable to perform an operation that includes outputting an indication that the sanitized text message is the identified language, wherein the indication includes a confidence score. The system described in. Goods,
Including a non-transitory computer readable medium storing instructions, wherein the instructions, when executed by one or more computers, include:
Getting a text message,
Removing non-language characters from the text message to generate a sanitized text message;
Detecting at least one of an alphabet and a script present in the sanitized text message,
The detecting is
(I) performing an alphabet-based language detection test to determine a first set of scores each representing a likelihood that the sanitized text message includes an alphabet for one of a plurality of different languages; When,
(Ii) A script-based language detection test is performed to determine a second set of scores each representing the likelihood that the sanitized text message includes a script for one of the plurality of different languages. And at least one of
The instructions are further executed by the one or more computers to the computer,
Based on at least one of the first set of scores, the second set of scores, and the combination of the first and second sets of scores, the language in the sanitized text message is An article that causes an action including identifying to be performed.

Images