KR20250126241A - Multilateral Real-Time Voice Recognition Method - Google Patents

Abstract

The present invention relates to a multi-party real-time speech recognition method, which enables real-time communication between multiple people speaking different languages within a given space by translating and interpreting each other's voices in real time while they are conversing using their own terminals, thereby enabling real-time communication between them within the same space.
The multi-party real-time speech recognition method proposed in the present invention enables real-time conversation between N people (N is a natural number greater than or equal to 2) who speak different native languages within a given space through terminals (N terminals) each having a given translation app installed.
A step in which the translation app installed on each of the N terminals is activated and linked to the server through a gateway installed in the designated space;
A step of allocating N IDL channels to each of the N terminals in the multi-IDL agent module of the above server;
A step of interconnecting the N IDL channels assigned to each of the N terminals to interconnect the communication networks between the N terminals;
When at least one person's voice information is sequentially input through each of the N terminals, the voice information is sequentially transmitted to the multi-IDL agent module of the server through the gateway, sequentially transmitted to the N IDL channels corresponding to the terminals into which the voice information was input, and sequentially transmitted to the N IDL channels of each of all remaining terminals linked to the N IDL channels into which the voice information was transmitted, and then the voice information is sequentially transmitted to all remaining terminals connected to the terminal into which the voice information was input through the gateway via a communication network, thereby enabling real-time multi-party voice information transmission.
When the present invention is implemented, there is an advantage in that a number of people speaking different languages within a given space can communicate with each other in real time by translating each other's voices in real time using their own terminals.

Description

Multilateral Real-Time Voice Recognition Method

The present invention relates to a multi-party real-time speech recognition method, which enables real-time communication between multiple people speaking different languages within a given space by translating and interpreting each other's voices in real time while they are conversing using their own terminals, thereby enabling real-time communication between them within the same space.

One of the biggest challenges in smoothly conducting academic conferences and business meetings is language. Even with extensive foreign language study, presenting in a foreign language can be challenging. Presenting in your native language and having an interpreter simultaneously interpret your presentation can significantly reduce the burden, facilitate easier question-answering, and even lead to positive business outcomes. However, simultaneous interpreters are difficult to find in a timely manner, and must be present at the appropriate venue, often at significant cost if overseas. Therefore, real-time simultaneous interpretation technology is needed, enabling users to access simultaneous interpretation services anytime, anywhere.

These problems can also arise when talking to locals you meet while traveling, or when talking to foreigners over the phone.

For this reason, various programs that translate or interpret multilingual conversations in real time have been developed and are available as apps or on the web.

However, most translation programs determine when the speaker has finished speaking and then translate or interpret what the speaker has said up to that point into a designated foreign language.

For example, the widely known Google or Papago translation systems or translation apps installed on mobile devices have limitations in that they are structured as systems focused on 1:1 translation.

In addition, even when multiple people speaking different native languages are conversing in the same space, the 1:1 interpretation and translation method is mostly used, and even when multiple people are conversing simultaneously, only sequential interpretation and translation is possible, and real-time interpretation and translation is not possible during multiple people's conversations.

Normally, when multiple people are talking in the same space, there is at least one speaker, and if they speak the same native language, there is no problem. However, when different native languages are used and interpretation is required, there is virtually no way to provide interpretation and translation in real time to multiple parties.

Accordingly, the present invention proposes a method for providing real-time interpretation and translation of multi-party conversations when multiple people speaking different native languages are having a conversation within a given space.

1. Patent Application No. 10-2014-001431, Title of the Invention: Speaker Segmentation-Based Multi-Party Automatic Interpretation and Translation Operation System and Method and Device Supporting the Same 2. Patent Application No. 10-2018-0021969, Title of the Invention: Real-time multi-party interpretation wireless transmission/reception system and method thereof

The present invention proposes a multi-party real-time speech recognition method that enables real-time communication between multiple people speaking different languages within a given space by translating and interpreting each other's voices in real time while they are conversing using their own terminals, thereby enabling real-time communication between them within the same space.

The multi-party real-time speech recognition method proposed in the present invention enables real-time conversation between N people (N is a natural number greater than or equal to 2) who speak different native languages within a given space through terminals (N terminals) each having a given translation app installed.

A step in which the translation app installed on each of the N terminals is activated and linked to the server through a gateway installed in the designated space;

A step of allocating N IDL channels to each of the N terminals in the multi-IDL agent module of the above server;

A step of interconnecting the N IDL channels assigned to each of the N terminals to interconnect the communication networks between the N terminals;

When at least one person's voice information is sequentially input through each of the N terminals, the voice information is sequentially transmitted to the multi-IDL agent module of the server through the gateway, sequentially transmitted to the N IDL channels corresponding to the terminals into which the voice information was input, and sequentially transmitted to the N IDL channels of each of all remaining terminals linked to the N IDL channels into which the voice information was transmitted, and then the voice information is sequentially transmitted to all remaining terminals connected to the terminal into which the voice information was input through the gateway via a communication network, thereby enabling real-time multi-party voice information transmission.

When the present invention is implemented, there is an advantage in that a number of people speaking different languages within a given space can communicate with each other in real time by translating each other's voices in real time using their own terminals.

FIG. 1 is a conceptual diagram illustrating a V-TOT (Voice-Time In Target) method that enables a multi-party real-time speech recognition method according to the present invention.
FIG. 2 is a drawing illustrating the function of a multi-IDL agent module that enables a multi-party real-time speech recognition method according to the present invention.
FIG. 3 is an example of a conceptual diagram for explaining a real-time translation and conversation method using voice recognition according to the present invention.
Figure 4 is an embodiment of a real-time translation and conversation method using voice recognition according to the present invention.

Hereinafter, the multi-party real-time speech recognition method, which is the technical idea of the present invention, will be described.

For reference, the multi-party real-time speech recognition method proposed in the present invention can be applied to conversations through terminals having a predetermined translation app installed. Here, the terminals of the present invention include all smartphones, computers, tablets, etc. that are linked through communication with a server.

First, in order to implement the present invention, it is applied when a terminal holder having a predetermined translation app installed converses within a predetermined space.

Here, it is assumed that among the multiple terminal holders conversing within a given space, at least one is a foreign language speaker, and it is also possible that the group is comprised of foreigners who all speak different native languages.

FIG. 1 is a conceptual diagram illustrating a V-TOT (Voice-Time In Target) method that enables a multi-party real-time speech recognition method according to the present invention.

The artificial intelligence-based voice recognition technology according to the present invention requires three processes: ① voice streaming, ② voice conversion into text, and ③ translation of the converted text into a corresponding foreign language when voice input is performed.

Multi-person voice recognition of one or more people is made possible through parallelization of the Bidirectional RNN (bidirectional recurrent neural network) technology used in artificial intelligence voice recognition according to the present invention.

The core of the parallelization of the Bidirectional RNN (bidirectional recurrent neural network) technology according to the present invention is V-TOT (Voice-Time On Target).

FIG. 2 is a drawing illustrating the function of a multi-IDL agent module that enables a multi-party real-time speech recognition method according to the present invention.

As illustrated in Figure 2, the present invention will explain a case where each person converses in his or her native language through a plurality of terminals in which a predetermined translation app is installed within a predetermined space.

In Figure 2, we assume that three people are talking through their own terminals, but the same applies to multi-party conversations with more than one person.

In the system of Figure 2, each of the three speakers carries its own terminal.

Each of the above terminals is connected to a voice recognition gateway installed within a designated space.

The voice recognition gateway is connected to a server, and as illustrated, the server of the present invention is provided with multiple IDL agents.

The multi-IDL agent of the present invention provides a predetermined number of IDL channels to each terminal.

Here, the number of IDL channels provided to each terminal is equal to the number of terminals participating in the conversation within the given space. For reference, in the case of Figure 2, the number of IDL channels provided to each terminal is three. If, for example, one additional person joins the conversation, bringing the number of terminals to four, the number of IDL channels provided to each terminal changes to four and is then allocated.

The server enables communication by interconnecting the three IDL channels assigned to each terminal.

Afterwards, if at least one of the terminal holders attempts to have a conversation, each person's voice information will be transmitted to the gateway through the terminal they are holding and then forwarded to the server.

On the server, three IDL channels are formed, each assigned to a terminal, and these IDL channels are connected to the IDL channels of each other terminal. Therefore, voice information sequentially input through each terminal is sequentially processed by multiple IDL agents and then transmitted through the terminals that are interconnected. At this time, interpretation and translation are processed in the manner described in Figures 3 and 4, and this method enables real-time conversations between multiple parties using different native languages within a specific space.

The technical idea of the present invention can be summarized as follows.

First, the present invention described in FIG. 2 relates to a multi-party real-time speech recognition method that enables real-time conversation between N people (N is a natural number greater than or equal to 2) who use different native languages within a given space through terminals (N terminals) each having a predetermined translation app installed.

The implementation of the present invention proceeds in the following order.

Step 1: The translation app installed on each of the above N terminals is activated and linked to the server through the gateway installed in the above designated space.

Step 2; A step of allocating N IDL channels to each of the N terminals in the multi-IDL agent module of the server.

Step 3: A step in which the N IDL channels assigned to each of the N terminals are interconnected to interconnect the communication networks between the N terminals.

Step 4: When at least one person's voice information is sequentially input through each of the N terminals, the voice information is sequentially transmitted to the multi-IDL agent module of the server through the gateway, sequentially transmitted to the N IDL channels corresponding to the terminals into which the voice information was input, and sequentially transmitted to the N IDL channels of each of all remaining terminals linked to the N IDL channels into which the voice information was transmitted, and then the voice information is sequentially transmitted to all remaining terminals connected to the terminal into which the voice information was input through a communication network through the gateway, thereby enabling real-time multi-party voice information transmission.

By implementing the above method, it can be seen that real-time conversation and interpretation/translation becomes possible between multiple parties who speak different native languages.

Next, a server having a function that enables real-time translation and conversation method through voice recognition of the present invention to provide real-time multi-party interpretation and translation of voice information transmitted through the aforementioned multiple LDL modules will be described.

The server of the present invention is equipped with an automatic speech recognition system (ASR) learned by artificial intelligence, and the automatic speech recognition system of the present invention includes at least an audio extractor, an audio processor, a vocabulary processor, a language processor, etc. The ASR implemented in the present invention passes the voice input through the terminal through the VAD module. Here, the VAD (Voice Activity Detection) module is trained to be able to distinguish background noise, silence information (silent time), and voice information among the information input through the terminal. In the present invention, by utilizing the VAD module, background noise can be removed and silence information can be acquired, and this silence information performs an important function in the real-time interpretation and translation of the present invention, and also determines the end point (end of a spoken sentence) of the input voice input.

By means of the VAD module of the present invention, only voice information is extracted from the information transmitted to the server through the terminal and transmitted to the ASR described below.

FIG. 3 illustrates an example of a conceptual diagram for explaining a real-time translation and conversation method using voice recognition according to the present invention, and FIG. 4 illustrates an embodiment of a real-time translation and conversation method using voice recognition according to the present invention.

In the present invention, a sound extractor (e.g., an MFCC-based sound extractor), which is a component of ASR, receives voice information input through a single unit linked to a server and generates data information (spectrogram information) that extracts unique features of the voice information.

Next, the sound processor of the present invention can be configured as a sound processor based on a Conformer model, receives spectrogram information from an acoustic extractor, analyzes voice features of the spectrogram information, and then extracts a pronunciation text having the highest similarity to the voice features.

Next, the vocabulary processor of the present invention is equipped with a vocabulary dictionary module learned by artificial intelligence, and compares the pronunciation text received through the sound processor with the learned vocabulary dictionary module to convert it into an actual vocabulary text with the highest similarity.

Here, actual vocabulary text refers to the process of converting pronunciation text information into standard language text that is easy to translate or converting unconfirmed pronunciation text information into actual vocabulary text with accurate meaning by estimating and supplementing it.

For example, if the pronunciation text information of the sound processor is "This is Eden-gyo," the sound processor performs the function of converting it into an actual vocabulary text that can be matched to the standard language, such as "Where is this?"

Here, the sound processor of the present invention performs a function of converting text information such as "Hello, may I ask you something. Where is a good restaurant around here?" into text in the form of "Hello, may I ask you something. Where is a good restaurant around here?", and the sound processor of the present invention does not simply infer words and convert them into a predetermined text, but can convert the input text information into a standardized text through word, sentence, paragraph unit inference and operations.

Next, the language processor of the present invention (e.g., a language processor based on Hidden Markov Models (HMMs)) performs a function of combining sequentially input actual vocabulary text information to match an actual language structure and then producing entire text information.

Meanwhile, the server of the present invention additionally has a real-time conversation translator to perform real-time conversation translation, i.e., real-time interpretation.

The real-time conversation translator of the present invention performs a function of receiving and translating actual vocabulary text information output from a vocabulary processor.

The real-time translation of the present invention performs the following functions to enable sequential translation of voice information input through a terminal before its final conclusion.

First, the real-time conversation translator of the present invention has a function of analyzing punctuation (meaning the ending of a colloquial sentence, a colloquial sentence), parts of speech, and sentences of actual vocabulary text information received from a vocabulary processor.

The real-time conversation translator of the present invention is provided with a function for determining whether actual vocabulary text information received from a vocabulary processor is a conditional sentence, a function for determining whether it is an adverb or an adverbial phrase, a function for determining whether actual vocabulary text information sequentially input is a predicate, and a function for determining whether a punctuation mark indicating the end of a sentence is included.

For example, if the actual vocabulary text information judged by the real-time translation processor of the present invention corresponds to a conditional sentence, it can be translated into a foreign language such as English and output in real time.

In addition, if the actual vocabulary text information judged by the real-time translation processor of the present invention corresponds to a pre-learned adverb or adverbial phrase, it can be translated into a foreign language such as English and output in real time.

In addition, if the actual vocabulary text information determined by the real-time translation processor of the present invention includes a predicate, the actual vocabulary text information including the predicate is translated into a foreign language such as English in real time and output.

In addition, if the real vocabulary text information judged by the real-time translation processor of the present invention is determined to be a sentence with punctuation marks, the real vocabulary text information including punctuation marks is translated into a foreign language such as English in real time and output.

The translation described above is performed independently and sequentially for each sentence containing a conditional sentence, an adverb or adverbial phrase, a predicate, or a sentence containing punctuation.

For example, for the information "If it rains tomorrow, you should refrain from going on outings," the conditional sentence "If it rains tomorrow" is translated first, and then the next sentence, which includes punctuation, "You should refrain from going on outings," is translated sequentially.

Here, if "tomorrow" is judged as an adverb or "adverbial phrase", "tomorrow" can be translated and then "if it rains" can be translated sequentially.

To explain this in detail, it is as follows.

For example, let's assume that the actual vocabulary text information sequentially input into the real-time translation processor is as follows.

If the conversation entered through the terminal is "And, if you want to have an interesting conversation, you have to think from the other person's perspective," the actual vocabulary text information can be sequentially as follows.

1. And

2. Have fun

3. Conversation

4. Share

5. If you want

6. The other party's

7. From the position

8. Thinking

9. I have to do it

If the above conversation were translated into English, it would be something like this:

"And if you want to have a fun conversation, you have to think from the other person's point of view"

However, in the present invention, in order to translate in real time, the content of the conversation is translated in real time by breaking it up in the following manner.

First, in the present invention, “and” is an adverb, so it is translated into the corresponding foreign language (e.g., English) word “And” and output in real time.

Next, since "fun" is also an adverb, you can output the corresponding foreign word, funny, in real time.

Next, the conditional statement "If you want to have a conversation" can be translated into "if you want to have a conversation" and output in real time. Note that this may also apply if the translation includes the preceding text information, including the predicate.

Next, "from the other person's perspective" can be translated as "from the other person's point of view" and displayed in real time. Note that this case is considered an adverbial phrase.

Next, "I have to think about it" (which includes a predicate) can be translated into "you have to think about it" and output in real time. Note that this is an example of a translation that includes the preceding text information, including the predicate ("I have to think about it").

As a result, the speaker's counterpart can hear, in real time, the sentence, "and, funny, if you want to have a conversation, from the other person's point of view, you have to think about it."

Of course, the above sentence may differ somewhat from the exact translation.

However, when implementing the method of the present invention, the speaker can sufficiently convey his/her intentions to the other party in real time within the scope of his/her intended message. Whether sufficient communication is possible may vary depending on the paragraph analysis capabilities of the real-time translation processor. However, implementing the method proposed in the present invention offers the advantage of allowing both the speaker and the other party to understand the meaning of the other party's speech in real time, even when using different languages, and thus respond in real time.

For example, since the speaker's conversation can be translated in real time and provided to the other party, the other party has the advantage of being able to convey their thoughts in real time during the speaker's conversation.

That is, if the voice information input through the terminal proposed in the present invention is standardized through ASR, and then converted into text through a translator, and the text is analyzed to determine Korean characters, adverbs, adverbial phrases, predicates, punctuation, etc., and the text is sequentially translated in real time and output to another terminal, there is an advantage in that the intention of both parties can be understood in real time without any interruption in the conversation.

Claims (1)

A multi-party real-time speech recognition method that enables real-time conversation between N people (N is a natural number greater than or equal to 2) who speak different native languages within a given space through terminals (N terminals) each with a given translation app installed.
A step in which the translation app installed on each of the N terminals is activated and linked to the server through a gateway installed in the designated space;
A step of allocating N IDL channels to each of the N terminals in the multi-IDL agent module of the above server;
A step of interconnecting the N IDL channels assigned to each of the N terminals to interconnect the communication networks between the N terminals;
A multi-party real-time speech recognition method characterized in that when at least one person's voice information is sequentially input through each of the N terminals, the voice information is sequentially transmitted to the multi-IDL agent module of the server through the gateway, sequentially transmitted to the N IDL channels corresponding to the terminals into which the voice information was input, and sequentially transmitted to the N IDL channels of each of all remaining terminals linked to the N IDL channels into which the voice information was transmitted, and then the voice information is sequentially transmitted to all remaining terminals connected to the terminal into which the voice information was input through a communication network through the gateway, thereby enabling real-time multi-party speech information transmission.