JPH11250057A - Morpheme analyzer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To serve even an inputting part including an unknown word part which is long to some degree for an actual use in a short time by allowing a morpheme analyzer main body to execute morpheme analysis to a decided each partial character string. SOLUTION: An unknown word processor 2 confirms whether an inputted sentence includes an unknown word. Then, when the inputted sentence includes an unknown word, the processor 2 divides the inputted sentence into plural parts (analyzing object part) which do not include an unknown word part based on the range of the unknown word to allow the morpheme analyzer main body 3 to execute morpheme analysis concerning each analyzing object part to compose plural morpheme analyzing result and the range of the unknown word obtained concerning each analyzing object part to give to an input/output device 1 as output information. When the inputted sentence does not include the unknown word, the processor 2 allows the main body 3 to execute morpheme analysis concerning the inputted sentence itself and gives the obtained morpheme analyzing result to the device 1 as output information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a morphological analyzer, and more particularly, to a morphological analyzer that utilizes a stochastic technique to analyze digitized natural language text without using a dictionary. .

[0002]

2. Description of the Related Art A large number of computerized natural language texts have been easily available due to an increase in opportunities for text creation by word processors and the spread of Internet-compatible devices. For various application systems that apply natural language processing that handles a large amount of natural language text, such as character recognition systems, machine translation systems, information retrieval systems, information extraction systems, etc., morphological analysis processing is specialized processing (search , Composition, comparison, classification, summaries, etc.)
This is a very important process for determining a semantic unit in a sentence such as a word or a phrase, that is, a morpheme.

[0003] If this morphological analysis is erroneous, it is difficult to repair the error in the syntax analysis processing, semantic analysis processing, and the like in the subsequent stage. Even if it could be repaired, the process would be complicated and, therefore, would not be able to process large amounts of text in the expected time.

In a morphological analysis process, a language such as Japanese without word delimiter is not only part-of-speech estimated but also has word segmentation compared to a language text such as English which has an orthography that uses a space as a word delimiter. A major issue is whether it can be performed with high accuracy and at high speed. Further, in the morphological analysis device, it is a big problem that the morphological analysis can be performed with high accuracy even if the morphological analysis target sentence (natural language text) includes an unknown word.

[0005] Reference 1 "Japanese Patent Laid-Open No. Hei 8-315078"
In the method disclosed in Document 1, unknown words are dealt with by introducing a word model using a chain probability of a character string. However, this method uses only the chain probability within a word, and only indirectly expresses the possibility of the word depending on the context before and after by the chain probability of the part of speech. That is, unless the chain probability of the entire context or the chain probability of a character string having a length exceeding the range of the unknown character string is used, the unknown character string cannot be correctly determined or separated. In addition, since this conventional method is a word-based method, if the morphological analysis device assumes that there is an unknown word character string, word positions of an arbitrary length are assumed at all positions in the sentence. Must be taken into account, thus causing an increase in computational complexity.

Reference 2 "Mikio Yamamoto, Masakazu Masuyama,"
Japanese morphological analysis using chain probability of extended characters including delimiter information ”, Proceedings of the 3rd Annual Meeting of the Linguistic Processing Society,
The method disclosed in "March 1997" Literature 2 is based on a character string having a predetermined number of characters N (N is, for example, 3) (hereinafter, sometimes referred to as N-gram data) instead of a word. The Japanese morphological analysis is performed using a probabilistic model, and can solve the problem of Reference 1.

[0007]

Problems to be Solved by the Invention However, Document 2
The morphological analysis method described in (1) also has the following problems.

If an attempt is made to perform a morphological analysis on an input sentence containing an unknown word of N characters or more as much as possible, (N-
1) Probability models with a small number of characters such as -gram data and (N-2) -gram data are prepared, and N-gr
(N-1) -gram data and (N-2) -g
Apply ram data to determine morpheme breaks. In this case, not only the N-gram data but also (N-1) -g
Probability models with a small number of characters, such as ram data and (N-2) -gram data, are also created and stored in advance.
The required storage capacity becomes enormous.

In order to avoid this, only a probabilistic model of N-gram data is prepared, and the N-gram data can be applied to an unknown word part from which a morphological analysis result cannot be obtained by applying N-gram data. It is conceivable that -1) -gram data or (N-2) -gram data is created at that time from a probability model of N-gram data and used for morphological analysis. However, generating (N-1) -gram data or (N-2) -gram data from a probability model of N-gram data requires a lot of arithmetic processing and takes a lot of time. .

Also, a calculation for calculating a value for evaluating a connection relationship (a chain probability of the entire sentence) for ordering a plurality of morphological analysis result candidates even for a part which is clearly known as an unknown word. Is executed, it can be said that the calculation time is wasted.

Therefore, there is a need for a morphological analyzer capable of obtaining a morphological analysis result that can be put to practical use in a short time even for an input sentence including an unknown word part to some extent.

[0012]

In order to solve this problem, the present invention comprises an extended character table storing extended character information, and the morphological analyzer main unit uses the extended character table to input data. A morphological analysis device that performs a morphological analysis on the analyzed sentence to be analyzed; (1) analysis target partial character string determination means for determining one or a plurality of partial character strings to be analyzed in the input analysis target sentence; 2) analysis control means for causing the morphological analysis apparatus main body to perform morphological analysis on each of the determined partial character strings; and (3) the morphological analysis apparatus main body.
A result output unit that outputs result information including at least an analysis result of each partial character string to be analyzed.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS (A) First Embodiment Hereinafter, a first embodiment of a morphological analyzer according to the present invention will be described in detail with reference to the drawings.

FIG. 1 is a functional block diagram showing the configuration of the morphological analyzer according to the first embodiment. That is, the morphological analysis device of the first embodiment is actually realized on an information processing device such as a workstation or a personal computer having an input / output device, a processing device, a storage device (or a communication device), and the like. However, functionally, it has the configuration shown in FIG.

In FIG. 1, a morphological analyzer according to the first embodiment includes an input / output device 1, an unknown word processor 2, a morphological analyzer main body 3, an extended character table 4, an input intermediate file 5, and an output intermediate file 6. Is provided.

Here, the unknown word processing device 2 includes an unknown word determining unit 21, an unknown word range estimating unit 22, an input sentence dividing unit 23, and an output result combining unit 24. Further, the morphological analyzer main body 3 includes an extended character string generation unit 31, a chain probability calculation unit 32, and an optimum route search unit 33.

Although not shown, the input / output device 1 naturally includes an input unit and an output unit. The input unit may be configured by any means such as a keyboard, a mouse, an OCR (optical character recognition device), and a speech recognition device for inputting a natural language text (input sentence) to the morphological analyzer. It may be configured as means for receiving a communication signal from the outside via a communication medium such as a network. The output unit outputs the morphological analysis result obtained by the morphological analyzer to various external display units and communication units, as in the case of a normal information processing device.

The unknown word processing device 2 checks whether or not an input sentence includes an unknown word. When the unknown word processing device 2 includes an unknown word,
Based on the unknown word range, the input sentence is divided into a plurality of portions (hereinafter, referred to as analysis target portions) that do not include the unknown word portion, and the morphological analysis device main body 3 executes morphological analysis for each analysis target portion. A plurality of morphological analysis results obtained for the analysis target portion and the unknown word range are combined and given to the input / output device 1 as output information. On the other hand, if the input sentence does not include an unknown word, the unknown word processing device 2 causes the morphological analysis device main body 3 to perform a morphological analysis on the input sentence itself, and outputs the obtained morphological analysis result to the input / output device 1. It is given as information.

In the unknown word processing device 2, the unknown word determination unit 21 determines whether or not the input sentence includes an unknown word while referring to the storage contents of the extended character table 4. When the input sentence does not include an unknown word, the unknown word determination unit 21 causes the morphological analysis device main body 3 to perform morphological analysis on the input sentence itself, and outputs the obtained morphological analysis result to the input / output device 1. It is given as information.

The unknown word range estimating section 22 estimates the unknown word range while referring to the contents stored in the extended character table 4 when the input sentence includes an unknown word.

The input sentence dividing unit 23 divides the input sentence into a plurality of analysis target parts that do not include an unknown word part (estimated unknown word range), stores the divided analysis target parts in the input intermediate file 5, and The morphological analyzer main body 3 executes morphological analysis on the divided analysis target portion.

The output result combining unit 24 combines a plurality of morphological analysis results obtained for each part to be analyzed and the unknown word range through an operation of storing the unknown word range in the output intermediate file 6, and outputs the information obtained by the combining. This is given to the input / output device 1 as final morphological analysis result information (output information).

In FIG. 1, the internal configuration of the unknown word processing device 2 is clearly divided into functions.
In practice, it is often formed so that it cannot be clearly distinguished (see FIG. 2 described later).

The input intermediate file 5 is, as described above,
It stores an analysis target portion (an input sentence input stage) obtained by dividing the input sentence.

As described above, the output intermediate file 6 stores the morphological analysis result and the unknown word range (in the middle of the output result) for each analysis target portion.

The extended character table 4 stores an extended character string (N-gram data) having the number of characters N and its chain probability (partial chain probability). The extended character table 4 is learned and prepared in advance by a training text (corpus). (Storage device). Here, the extended character ei is a character ci,
This is a pair of extended information di consisting of delimiter information (and part of speech information of the morpheme to which the character ci belongs) indicating whether or not the character ci is a morpheme delimiter. Note that the following description is made assuming that only the delimiter information is included as the extended information di.

The morphological analyzer main body 3 is similar to the conventional one using a probability model based on characters. It should be noted that the morphological analysis apparatus main body 3 is not only provided with an input sentence as a morphological analysis target, but also in some cases provided with an analysis target portion obtained by dividing the input sentence.
Although it is different from the conventional device, the detailed configuration is not different from the conventional device.

The extended character string generation unit 31 in the morphological analyzer main body 3 generates extended characters corresponding to each character of the input sentence or the part to be analyzed, and generates all combinations of the extended characters (each combination is referred to as an extended character string). (Referred to as a route) in a score table incorporated in the chain probability calculation unit 32.

The chain probability calculating section 32 calculates the chain probability for the path of the expanded character string stored in the built-in score table based on the partial chain probability stored in the extended character table 4. The chain probability for the path of the extended character string is given, for example, as the product of the partial chain probabilities of all the N-gram data constituting the path.

The optimal route search unit 33 includes the chain probability calculation unit 3
From the chain probabilities calculated in step 2, a path of an extended character string that satisfies an optimal condition (for example, a maximum value of the chain probability) is selected as an optimal extended character string. The expression is converted and returned as a morphological analysis result to the unknown word processing device 2 which has obtained a morphological analysis. For example, the delimiter information indicating that it is not a morpheme delimiter is deleted, and the delimiter information indicating that it is a morpheme delimiter is replaced with “/” to convert the expression of the extended information part.

As described above, the morphological analyzer main body 3
Since it is the same as the conventional one, the description of its operation is omitted,
In the following, the operation of the unknown word processing device 2 constituting the feature of the first embodiment will be described with reference to the flowchart of FIG. In the following description, N is assumed to be 3. That is, it is performed assuming that 3-gram data is applied.

First, the first three characters (N characters) of the input character string that have not yet been checked against the contents stored in the extended character table 4 are extracted from the input / output device 1 (step 100). Then, it is checked whether or not the arrangement of the extracted three characters exists in the extended character table 4 (step 101). The extended character table 4 stores an extended character string with extended information.
Performed for character strings excluding extended information.

If there is an arrangement of the extracted three characters in the extended character table 4, the first character of the three characters is stored in the input intermediate file 5, and then the position of the three characters is advanced by one character. And returns to step 100 (step 107).

On the other hand, when there is no arrangement of the extracted three characters in the extended character table 4, the third character is recognized as the head of the unknown word string, and the first and second characters that are not unknown words are recognized. Is stored in the input intermediate file 5 (step 10).
2). Then, the character string stored in the input intermediate file 5 is given to the morphological analysis device main body 3 to execute morphological analysis on the character string (step 103). At this time, the morphological analysis device main body 3 outputs, for example, a character string (morphological analysis result) in which a delimiter (/) is added to the morphological delimiter position in the character string, and outputs the morphological analysis result to the output intermediate file 6. Are stored following the contents stored so far, and the input intermediate file 5 is emptied (step 104).

Thereafter, for the character string starting from the character recognized as the head of the unknown word in step 102, the unknown word is determined for each three characters by the same processing as the processing loop of steps 100, 101 and 107. While the unknown word continues, the process is repeated, and it is assumed that the characters of the unknown word have continued for the number of times until three non-unknown words appear (step 105). That is, the unknown word range is estimated.

Then, a delimiter (/) is added after the character string estimated to be an unknown word so that the input intermediate file 5 is emptied while the output intermediate file 6 is stored following the contents stored so far. The process returns to step 100 (step 106).

When the process proceeds to step 100 after the input character string has been read to the end, a result is obtained that the character string cannot be read anymore. At this time, the character string stored in the input intermediate file 5 at that time is given to the morphological analyzer main body 3 to execute morphological analysis on the character string (step 108). Then, the character string (morphological analysis result) added with a delimiter (/) at the morphological delimiter position in the character string returned from the morphological analysis device main body 3 is stored in the output intermediate file 6 so far. Then, the input intermediate file 5 is emptied (step 109).

Finally, the contents stored in the output intermediate file 6 are output to the input / output device 1, and a series of processes for the current input sentence is completed (step 110).

Hereinafter, the processing of the morphological analyzer according to the first embodiment will be described using a specific example with reference to FIGS. 3 and 4.

FIG. 3 is an explanatory diagram showing the processing and the flow of data for the input sentence "This is Kankiri." FIG. 4 is an explanatory diagram of a storage example of the extended character table 4. FIG. 4 shows only a character string of three characters (N characters), and illustration of the extended information and the chain probability is omitted. In FIG. 4, “O” indicates that the character string exists in the extended character table 4, and “×” indicates that the character string does not exist in the extended character table 4. "And" x "are not stored in the extended character table 4 as information.

In response to the input sentence "This is Kankiri.", A character string of three characters such as "this is", "reha ka", "ha kang", ... The collation with the contents stored in Table 4 is performed.

When the character strings "this", "rehaka" or "hakan" are collated, it is confirmed that they exist in the extended character table 4. “Ko”, “re” or “ha” is additionally stored sequentially.

When the character string "kanki" is to be collated with the extended character table 4, since the character string "kanki" does not exist in the extended character table 4, the preceding two characters "kan" are stored in the input intermediate file. 5 and also stored in
The character "" is recognized as the first character of the unknown word. Then, the stored content of the input intermediate file 5 “this is can” is given to the morphological analyzer main body 3 and subjected to morphological analysis, and the obtained analysis result “this / ha / can /” is stored in the output intermediate file 6. In both cases, the contents of the input intermediate file 5 are emptied.

In order to estimate a character string that forms an unknown word starting with the character “K”, character strings of three characters after the character “K” such as “Kiride”, “Ri”,. ,
The comparison is performed with the contents stored in the extended character table 4. That is, the first three characters until a character string existing in the extended character table 4 is found are regarded as characters constituting an unknown word. The input intermediate file 5 is also used at the time of such collation.

In this example, since the character string "is." Exists in the extended character table 4, the character string "Kiri" which connects the first characters of the collation character strings "Kiride" and "Kirida" up to that point is unknown. Guessed as a word string. Then, the character string “Kiri /” obtained by adding a delimiter (/) to the unknown word character string “Kiri” is used as a morphological analysis result for the unknown word part,
It is additionally stored in the output intermediate file 6, and then the contents of the input intermediate file 5 are emptied.

When the reading of the last character "." Of the input character string is completed, the character string "is." Is stored in the input intermediate file 5. The stored content “is.” Is given to the morphological analyzer main body 3 and subjected to morphological analysis, and the obtained analysis result “is /./” is additionally stored in the output intermediate file 6 and the contents of the input intermediate file 5 Is emptied.

Finally, according to the morphological analyzer of the first embodiment, the stored contents of the output intermediate file 6 “this / ha / can / kiri / is /./” are output as the morphological analysis results. An unknown word portion is guessed and determined based on whether or not the character string of N characters constituting the input sentence exists in the extended character table, and each portion of the input sentence other than the unknown word portion and divided by the unknown word portion is determined. Morphological analysis is performed for each character string, and the obtained morphological analysis result and the unknown word part are combined to obtain the final morphological analysis result, so information prepared and stored in advance for morphological analysis It is possible to obtain a morphological analysis result that can be put to practical use in a short time even for an input sentence including an unknown word part with a certain length while keeping the amount substantially the same as in the related art.

That is, basic N-gram data such as (N-1) -gram data and (N-2) -gram data.
There is no need to prepare a probability model having fewer characters than m data, and an increase in storage capacity can be suppressed. Also, even if an unknown word is found, (N-1) -gram data or (N-2)-
There is no need to create gram data, and even if an unknown word is included, a morphological analysis result can be obtained quickly. Further, there is no need to estimate the partial chain probability for the unknown word portion, or to perform comprehensive chain probability calculation for the unknown word portion, and the morphological analysis result can be quickly obtained from this point.

(B) Second Embodiment Next, a second embodiment of the morphological analyzer according to the present invention will be described in detail with reference to the drawings.

FIG. 5 is a functional block diagram showing the configuration of the morphological analyzer according to the second embodiment, in which the same or corresponding parts as those in FIG. 1 are denoted by the same reference numerals.

In FIG. 5, the morphological analyzer according to the second embodiment includes an input / output device 1, a partial character string processing device 7,
It includes a morphological analysis device main body 3, an extended character table 4, a partial character string storage file 8, and a partial character string analysis result storage file 9.

Here, the partial character string processing device 7 has a partial character string designation unit 71, a partial character string generation unit 72, and a partial character string analysis unit 73.

The input / output device 1, the morphological analyzer main body 3, and the extended character table 4 are the same as those in the first embodiment, so that the description of their functions will be omitted.

The partial character string processing device 7 captures whether to execute morphological analysis on the entire input sentence or to execute morphological analysis on the partial character string of the input sentence. Then, in the former case, the partial character string processing device 7 supplies the entire input sentence to the morphological analysis device main body 3 to perform morphological analysis, and outputs the returned morphological analysis result to the input / output device 1. is there. On the other hand, in the latter case, the partial character string processing device 7 takes in the specification of the partial character string,
A specified partial character string is extracted from the input sentence and stored in the partial character string storage file 8 together with the position information in the input sentence.
, The morphological analysis is returned, and the returned morphological analysis result is stored in the partial character string analysis result storage file 9. Further, the partial character string processing device 7 combines the analysis result stored in the partial character string analysis result storage file 9 with another partial character string of the input sentence that is not subjected to analysis and outputs the result to the input / output device 1. Is what you do. The function allocation in the partial character string processing device 7 is as follows.

The partial character string designating section 71 reads from the input / output device 1 whether to perform morphological analysis on the entire input sentence or morphological analysis on the partial character string of the input sentence. In the case of, the entire input sentence is given to the morphological analysis device main body 3 to perform morphological analysis, and the returned morphological analysis result is output to the input / output device 1. In the latter case, the designation of a partial character string is performed. It is something to take in.

The partial character string generation unit 72 extracts the specified partial character string from the input sentence and
Is stored together with the position information in the input sentence. In addition, the partial character string generation unit 72 clarifies the fact and stores the character string of the part for which analysis is not required in the partial character string storage file 8.

The partial character string analysis unit 73 gives the partial character string extracted from the input sentence to the morphological analysis device main body 3 to perform morphological analysis, and returns the returned morphological analysis result to the partial character string analysis result storage file 9. In addition to storing the result, the morphological analysis result is combined with another partial character string of the input sentence whose analysis is not required and output to the input / output device 1.

As described above, the partial character string storage file 8 stores information on partial character strings for which morphological analysis has been obtained and for other partial character strings for which analysis is required and which is not obtained.

The partial character string analysis result storage file 9 stores the morphological analysis result for the partial character string as described above.

In the second embodiment as well, the morphological analyzer main body 3 is the same as the conventional one, so that the description of its operation is omitted, and the features of the second embodiment will be described below. The operation of the partial character string processing device 7 will be described with reference to the flowchart of FIG.

An operation mode in which an input sentence is input and then a morphological analysis is performed on a partial character string of the input sentence instead of performing a morphological analysis on the entire input sentence is performed via the input / output device 1. When designated by the user, the processing shown in FIG. 6 is started.

Then, first, the user designates a partial character string to be subjected to morphological analysis from the input sentence displayed on the screen via the input / output device 1, and fetches the designation information (step 200). As this designation method, an existing method such as a method of designating a character string portion to which a special modification such as underlining or shading in a word processor or the like is designated can be applied.
Alternatively, two or more partial character strings may be designated as an analysis target for one input sentence. Further, the length (the number of characters) of the specified partial character string is compared with a preset minimum length, and a specified partial character string shorter than the minimum length is rejected. For example, N characters are set as the minimum length. This is because the extended character string stored in the extended character table 4 is N-gram data.

Next, a specified partial character string to be analyzed is extracted from the input sentence (step 201). In other words, the partial character string to be analyzed is separated from the other partial character strings to be analyzed. Thereafter, the extracted partial character string to be analyzed is distinguished from the other partial character strings to be analyzed, and the positional relationship is clarified and stored in the partial character string storage file 8 (step 202).

Then, the partial character string to be analyzed stored in the partial character string storage file 8 is given to the morphological analyzer main body 3 for morphological analysis, and the returned morphological analysis result is stored in the partial character string analysis result storage file. 9 (step 203). If a plurality of partial character strings to be analyzed are specified, the morphological analyzer main unit 3
Sending the analysis target partial character string, receiving the returned morphological analysis result, and storing it in the partial character string analysis result storage file 9 are repeated by the specified number.

Finally, the analysis result of the partial character string to be analyzed stored in the partial character string analysis result storage file 9 and the non-analysis partial character string stored in the partial character string storage file 8 Are combined and output to the input / output device 1 to be displayed on the screen (step 204).

Next, the processing of the morphological analyzer according to the second embodiment will be described with a specific example. Here, it is assumed that the contents as shown in FIG. In FIG. 7, illustration of the chain probability is omitted. Further, in FIG. 7, in the extended information, only “Yes” is indicated by a delimiter (/) in the presence / absence information of a delimiter. That is, FIG. 7 is simply shown for convenience of description, and is different from a practical storage method.

Now, "I like apples and tangerines."
Is specified via the input / output device 1 to execute a morphological analysis on a partial character string of the input sentence instead of performing a morphological analysis on the entire input sentence. Suppose it was done.

At this time, the process proceeds to the specification of the partial character string to be analyzed. FIG. 8 shows a screen on which "Nango Mika" which is a part of the input sentence "I like apples and oranges" is specified. This example shows that the underlined portion is a partial character string to be analyzed, and is an example of a specification method in which the specification is confirmed by operating the execution key. When specified in this way, the partial character string "Nango Mika" to be analyzed is distinguished from the other non-analyzed partial character strings "I hate" and "I like n". And stored in the partial character string storage file 8.

Further, the partial character string "Nango Mika" to be analyzed is given to the morphological analyzer main body 3 and subjected to morphological analysis. In this case, the contents stored in the extended character table 4 are as shown in FIG.
The result of the analysis of the partial character string “Nango / Mika” is “nggo / to / mika”, and the analysis result “nggo / to / mika” is the partial character string analysis result. It is stored in the storage file 9.

Finally, the analysis result "ngo / to / mika" stored in the partial character string analysis result storage file 9 and the non-analysis target partial character string "stored in the partial character string storage file 8""Ilike" and "I like n" are combined, output to the input / output device 1, and displayed on the screen. FIG.
Shows the display screen at this time. In FIG.
The underline indicates the part subjected to the analysis processing.

As described above, according to the morphological analyzer of the second embodiment, morphological analysis can be executed not only for the entire input sentence but also for a partial character string of the input sentence. Since it is possible, the usability of the user can be improved.

For example, the user can specify a part unknown as to whether or not the device regards an unknown word, perform morphological analysis, and confirm whether the device regards the unknown word. Also,
For example, the user can perform a morphological analysis by designating a partial character string excluding a portion that is considered to be an unknown word by the device, and obtain an analysis result other than that assumed to be an unknown word portion. As described above, as described above, the processing speed for input sentences including unknown words is slow, and the storage capacity may be increased. By using it, the amount of information prepared and stored in advance for morphological analysis remains almost the same as before,
It can be expected that a morphological analysis result that can be put to practical use in a short time can be obtained even for an input sentence including an unknown word part to some extent.

Incidentally, none of the conventional morphological analyzers using a word dictionary has an analysis function for designating a part of an input sentence. This is because morphological analysis cannot be performed unless the partial designation is a designation at a position where words are clearly separated. For example, a partial character string starting in the middle of a word or a partial character string ending in the middle of a word cannot be correctly analyzed. It is not possible to analyze the partial character string “Nango Mika” in the above specific example.

On the other hand, in the morphological analyzer according to the second embodiment using the character-based probability model, the start position or the end position is based on the character and is not aware of the word. Even partial character strings in the middle can be correctly analyzed.

(C) Other Embodiments In the first embodiment, in the output of the analysis result when processed as an unknown word, the part processed as an unknown word is not clarified, but is processed as an unknown word. A part may be distinguished (for example, color-coded or underlined) from other parts and output (presented).

In the second embodiment, an input sentence is input first, and then a partial character string to be analyzed is specified by an interactive exchange with the user. You may make it apply. For example, a special symbol that defines the start and end positions of a partial character string to be analyzed is included in an input sentence and input to the device, and the device extracts a character string between those symbols as a target to be analyzed and executes the analysis. You may do it.

Furthermore, in the second embodiment, the partial character string to be analyzed is directly specified (the partial character string to be analyzed is indirectly specified thereby). The partial character string to be analyzed may be directly specified (the partial character string to be analyzed is thereby indirectly specified). This designation method is suitable as a designation method in a case where analysis is performed excluding a portion considered to be an unknown word.

Further, in each of the above embodiments,
Although the morphological analysis device in which the input sentence or the partial character string to be analyzed is in Japanese has been described, the present invention is naturally applicable to morphological analysis devices for other languages.

[0079]

As described above, according to the present invention, the analysis target partial character string determining means for determining one or a plurality of partial character strings to be analyzed in the input analysis target sentence is determined. Analysis control means for causing the morphological analysis device to perform morphological analysis on each of the partial character strings, and output of result information including at least an analysis result of each of the partial character strings to be analyzed, obtained by the morphological analysis device main body Morphological analysis results that can be put to practical use in a short time, even for input sentences containing unknown words, without increasing the information stored in advance. Obtainable.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a first embodiment.

FIG. 2 is a flowchart illustrating a process performed by the unknown word processing device according to the first embodiment;

FIG. 3 is an explanatory diagram showing a process and a data flow for a specific input sentence in the first embodiment.

FIG. 4 is an explanatory diagram of a storage example of an extended character table according to the first embodiment;

FIG. 5 is a block diagram illustrating a configuration of a second embodiment.

FIG. 6 is a flowchart illustrating processing of the partial character string processing device according to the second embodiment.

FIG. 7 is an explanatory diagram of a storage example of an extended character table according to the second embodiment.

FIG. 8 is an explanatory diagram showing an example of a partial character string designation screen according to the second embodiment.

FIG. 9 is an explanatory diagram illustrating an example of a partial character string analysis result display screen according to the second embodiment.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... I / O device, 2 ... Unknown word processing device, 3 ... Morphological analyzer main body, 4 ... Extended character table, 5 ... Input intermediate file, 6 ... Output intermediate file, 7 ... Partial character string processing device,
8 ... partial character string storage file, 9 ... partial character string analysis result storage file, 21 ... unknown word determination unit, 22 ... unknown word range estimation unit, 23 ... input sentence division unit, 24 ... output result connection unit,
31: extended character string generator, 32: chain probability calculator, 33
... Optimal route search unit, 71... Partial character string designation unit, 72... Partial character string generation unit, 73.

Claims (5)

[Claims] 1. A morphological analysis device comprising an extended character table storing extended character information, wherein a morphological analysis device body performs morphological analysis on an input analysis target sentence using information stored in the extended character table. An analysis target partial character string determining means for determining one or more partial character strings to be analyzed in the input analysis target sentence; and a morphological analysis unit for the determined partial character strings by the morphological analysis device main body. Morphological analysis, comprising: analysis control means for executing a morphological analysis unit; and result output means for outputting result information including at least an analysis result of each partial character string to be analyzed, obtained by the morphological analysis device main body. apparatus. 2. The analysis target partial character string determination means compares the partial character string of a predetermined number of characters in the input analysis target sentence with the stored contents of the extended character table to determine an unknown word part in the analysis target sentence. 2. The morphological analyzer according to claim 1, wherein a partial character string other than the unknown word part is estimated and determined as a partial character string to be analyzed. 3. The result output means outputs the analysis result of each partial character string to be analyzed, obtained by the morphological analysis device main body, and the unknown word part estimated by the analysis target partial character string determination means. Must be combined according to the order of those parts in the sentence to be analyzed, and be combined by adding delimiters to the start position and end position of the unknown word part, and output the combined contents as result information The morphological analyzer according to claim 2, wherein: 4. The analysis method according to claim 1, wherein the analysis target partial character string determination unit determines a partial character string in the analysis target sentence designated by the user as a partial character string to be subjected to morphological analysis. Morphological analyzer. 5. The sentence to be analyzed includes a special symbol for specifying a partial character string to be subjected to morphological analysis.
2. The morphological analyzer according to claim 1, wherein the analysis target partial character string determining means determines a partial character string to be subjected to morphological analysis based on the search for the special symbol.