CN115132184A - Voice interaction method, server and storage medium - Google Patents

Abstract

The application provides a voice interaction method, a server and a storage medium. The voice interaction method comprises the following steps: acquiring phonemes of different languages, and combining the phonemes of the different languages into a first syllable according to a pronunciation rule; recognizing the combined syllables of pronunciation by using training materials of different languages, and screening out second syllables from the combined syllables according to pronunciation adhesion; merging the first syllable and the second syllable to obtain a modeling syllable; generating an acoustic model from the modeled syllables; receiving a voice request sent by a user in a vehicle cabin and forwarded by a vehicle; and recognizing the voice request according to the acoustic model, generating a recognition result and sending the recognition result to the vehicle to finish voice interaction. The scheme provided by the application can realize multi-language unified modeling, is more convenient for voice recognition and voice interaction of different languages, reduces deployment cost and avoids machine resource waste.

Description

Voice interaction method, server and storage medium

technical field

The present application relates to the field of voice technology, and in particular, to a voice interaction method, server and storage medium.

Background technique

With the continuous development of the automotive industry and human-computer interaction technology, smart cars also provide users with voice interaction functions.

Voice interaction functions rely on voice recognition technology. At present, the speech data to be recognized may not only be a single-language speech, but may also be a bilingual mixed speech or a multilingual mixed speech, etc. Therefore, the speech recognition methods are also different. Taking Europe as an example, there are many languages and there are multiple language families, and the number of languages exceeds ten. In related technologies, the context-dependent syllable modeling method is generally used to construct an acoustic model, but it is highly dependent on the scene and cannot achieve unified modeling in multiple languages, which is not conducive to speech recognition and speech interaction in different languages. If a speech recognition system is deployed for each language, it will greatly increase the cost and waste machine resources.

SUMMARY OF THE INVENTION

In order to solve or partially solve the problems existing in the related art, the present application provides a voice interaction method, a server and a storage medium, which can realize multilingual unified modeling, facilitate voice recognition and voice interaction in different languages, reduce deployment costs, and avoid Waste of machine resources.

A first aspect of the present application provides a voice interaction method, comprising: acquiring phonemes of different languages, combining the phonemes of different languages into a first syllable according to phonetic rules; identifying the combined syllables of pronunciation by using training materials of different languages, Screen out the second syllable from the combined syllable according to the degree of pronunciation adhesion; combine the first syllable and the second syllable to obtain a modeled syllable; generate an acoustic model according to the modeled syllable; The voice request sent by the user in the vehicle cabin; the voice request is recognized according to the acoustic model, and the recognition result is generated and sent to the vehicle to complete the voice interaction. The present application merges the phonemes of different languages into the first syllable according to the phonetic rules, selects the second syllable from the combined syllables of the training materials of different languages according to the pronunciation adhesion, and then merges to obtain the modeling syllables, these syllables have nothing to do with the context, In this way, various languages can use the same modeling system, so that multi-language unified modeling can be achieved, which is more convenient for speech recognition and speech interaction in different languages, reduces deployment costs, and avoids wasting machine resources.

Merging the phonemes of the different languages into the first syllable according to the phonetic rules, including: pre-merging the phonemes of the different languages according to the rules of the IWC; merging the phonemes after the pre-merging according to the phonetic rules for the first syllable. The present application performs phoneme merging based on ipa, which can reduce the number of phonemes, so that dozens of languages can be mixed using one output layer, which can reduce the amount of calculation and delay.

The described phoneme after pre-merging is merged into the first syllable according to the phonetic rule, including: from the phoneme after the pre-merging, the first syllable is obtained by merging the phonemes of the initial consonant and the final consonant, and the remaining single initial consonant is merged into the first syllable. The phoneme of , and the phoneme of a single final stand alone as the first syllable. In this application, the phonemes are merged into syllables, which can be merged with reference to the phonetic rules, so that the merged syllables are more in line with pronunciation habits.

The recognizing the combined syllables of pronunciation by using training materials of different languages includes: identifying the combined syllables of pronunciation by using the training materials of audio and/or video of different languages. This application can make full use of audio and/or video as training material in different languages.

Screening out the second syllable from the combined syllable according to the degree of pronunciation adhesion, comprising: performing forced frame alignment on the combined syllable, obtaining the average pronunciation duration of the aligned combined syllable and the average pronunciation of all combined syllables Duration: The ratio of the average pronunciation duration of the combined syllables and the average pronunciation duration of all combined syllables is used as the articulation stickiness, and the combined syllables with the described pronunciation stickiness less than the set threshold are used as the second syllable. The present application selects syllables with reference to pronunciation cohesion, which can make the selected syllables more accurate.

Before performing the forced frame alignment on the combined syllables, the method further includes: filtering out combined syllables that conform to the syllable merging rule from the combined syllables; and performing the forced frame alignment on the combined syllables includes: combining the matched syllables. Regularly combining syllables to enforce frame alignment. Before the mandatory frame alignment is performed in the present application, the syllable merging rule is used for filtering, which can reduce the workload of subsequent screening and improve the processing efficiency.

The syllable merging rules include at least one of the following rules: initial + initial + final; initial + initial + final + special initial; initial + final + special initial. The syllable merging rules of the present application can be applied to various situations.

The average pronunciation duration of the combined syllable is determined according to the ratio of the total pronunciation duration after forced frame alignment to the number of times the combined syllable appears in the training audio. The average pronunciation duration is obtained by averaging processing, which can make the parameters more accurate.

The performing forced frame alignment on the combined syllables includes: performing forced frame alignment on the combined syllables with a final vowel as the core. Forced frame alignment with finals as the core is more in line with the actual situation of pronunciation.

A second aspect of the present application provides a server, comprising: a phoneme processing module for acquiring phonemes of different languages, and combining the phonemes of different languages into a first syllable according to phonetic rules; a training processing module for using different languages The training material identifies the combined syllable of pronunciation, and the second syllable is screened from the combined syllable according to the degree of pronunciation adhesion; The syllable merge module is used to obtain the first syllable obtained by the phoneme processing module and the training processing module. The second syllables are merged to obtain the modeled syllables; the model generation module is used to generate an acoustic model according to the modeled syllables; the request receiving module is used to receive the voice request sent by the user in the vehicle cockpit forwarded by the vehicle; the speech recognition module is used for The voice request received by the request receiving module is recognized according to the acoustic model generated by the model generating module, and the recognition result is generated and sent to the vehicle to complete the voice interaction. The present application merges the phonemes of different languages into the first syllable according to the phonetic rules, selects the second syllable from the combined syllables of the training materials of different languages according to the pronunciation adhesion, and then merges to obtain the modeling syllables, these syllables have nothing to do with the context, In this way, various languages can use the same modeling system, so that multi-language unified modeling can be achieved, which is more convenient for speech recognition and speech interaction in different languages, reduces deployment costs, and avoids wasting machine resources.

The phoneme processing module includes: a pre-merging sub-module for pre-merging the phonemes of the different languages according to the rules of the IWC; a phone-merging sub-module for merging the pre-merged phonemes according to the phonetic rules for the first syllable. The present application performs phoneme merging based on ipa, which can reduce the number of phonemes, so that dozens of languages can be mixed using one output layer, which can reduce the amount of calculation and delay.

Described training processing module comprises: Alignment and statistics module, is used to carry out forced frame alignment of described combined syllable, obtains the average pronunciation duration of described combined syllable after alignment and the average pronunciation duration of all combined syllables; Syllable screening module , for taking the ratio of the average pronunciation duration of the combined syllable and the average pronunciation duration of all combined syllables as the articulation stickiness, and using the combined syllable whose pronunciation stickiness is less than the set threshold as the second syllable. The present application selects syllables with reference to pronunciation cohesion, which can make the selected syllables more accurate.

A third aspect of the present application provides a server, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor is caused to execute the above-mentioned method.

A fourth aspect of the present application provides a computer-readable storage medium on which executable codes are stored, and when the executable codes are executed by a processor of an electronic device, the processor is caused to execute the above method.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

The above and other objects, features and advantages of the present application will become more apparent from the more detailed description of the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the exemplary embodiments of the present application. same parts.

1 is a schematic flowchart of a voice interaction method shown in the present application;

2 is a schematic flowchart of another voice interaction method shown in the present application;

3 is a schematic flowchart of voice modeling in the voice interaction method shown in the present application;

4 is a schematic diagram of an application framework for speech recognition using speech modeling shown in the present application;

5 is a schematic diagram of the comparison between context-dependent modeling and context-independent modeling shown in the present application;

6 is a schematic diagram of a modeling unit shown in the present application;

7 is a schematic structural diagram of a server shown in the present application;

8 is a schematic structural diagram of another server shown in the present application;

FIG. 9 is another schematic structural diagram of the server shown in this application.

Detailed ways

Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of this application to those skilled in the art.

The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, such information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information without departing from the scope of the present application. Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature. In the description of the present application, "plurality" means two or more, unless otherwise expressly and specifically defined.

In the related art, a context-dependent syllable modeling method is generally used to construct an acoustic model, which cannot achieve unified modeling in multiple languages, which is not conducive to speech recognition and speech interaction of different languages. The present application provides a voice interaction method, which can realize multilingual unified modeling, facilitates voice recognition and voice interaction in different languages, reduces deployment costs, and avoids waste of machine resources.

The technical solutions of the present application are described in detail below with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart of a voice interaction method shown in the present application. This method can be applied to the server.

Referring to Figure 1, the method includes:

S101. Acquire phonemes of different languages, and combine the phonemes of different languages into a first syllable according to a phonetic rule.

Among them, the phonemes of different languages can be pre-merged according to the rules of the Universal Phonetic Alphabet. For example, after merging English phonemes and French phonemes using ipa, there are only 56 phonemes, thus reducing the number of phonemes through the merging process. Then, the pre-merged phonemes can be merged into the first syllable according to phonetic rules. For example, from the pre-merged phonemes, the first syllable can be obtained by merging the phonemes of the initial and the final, and the remaining phonemes of the single initial and the phoneme of the single final can be used alone as the first syllable.

S102: Identify the combined syllables of pronunciation by using training materials of different languages, and select the second syllable from the combined syllables according to the degree of pronunciation adhesion.

Wherein, the combined syllables of the pronunciation can be identified by using the audio and/or video training materials in different languages. This application can make full use of audio and/or video as training material in different languages.

Among them, the combined syllables can be forced frame alignment; the average pronunciation duration of the aligned combined syllables and the average pronunciation duration of all combined syllables are determined; the average pronunciation duration of the combined syllables and all combined syllables The average pronunciation duration of The ratio is used as the articulation cohesion, and the combined syllable whose articulation cohesion is less than the set threshold is regarded as the second syllable. The smaller the pronunciation cohesion, the more the phonemes in the combined syllable should be grouped together.

The syllable merging rules include at least one of the following rules: initial + initial + final; initial + initial + final + special initial; initial + final + special initial.

The average pronunciation duration of the combined syllable is determined according to the ratio of the total pronunciation duration after forced frame alignment to the number of times the combined syllable appears in the training audio.

The forced frame alignment of combined syllables includes: forced frame alignment of combined syllables with finals as the core.

It should be noted that there is no sequence relationship between S101 and S102.

S103. Combine the first syllable and the second syllable to obtain a modeled syllable.

The first syllable and the second syllable obtained respectively in the above different steps are combined to generate the final syllable, that is, the modeled syllable.

S104. Generate an acoustic model according to the modeled syllable.

After the modeled syllables are obtained, an acoustic model can be generated according to the modeled syllables using existing related technologies.

S105: Receive a voice request sent by a user in the vehicle cabin forwarded by the vehicle.

The server may receive the voice request sent by the user in the vehicle cabin forwarded by the vehicle. The voice request may be a French voice request or a German voice request, etc.

S106: Identify the voice request according to the acoustic model, generate a recognition result and send it to the vehicle to complete the voice interaction.

After receiving the voice request from the user, the server recognizes the voice request according to the acoustic model, generates a recognition result and sends it to the vehicle to complete the voice interaction. The method for recognizing the voice request according to the acoustic model may adopt the existing recognition method in the related art, which is not limited in this application.

In the present application scheme, the phonemes of different languages are merged into the first syllable according to the phonetic rules, the second syllable is selected from the combined syllables of the training materials of different languages according to the pronunciation adhesion degree, and then the modeling syllables are obtained by merging them. In this way, different languages can use the same modeling system, so that multi-language unified modeling can be achieved, which is more convenient for speech recognition and speech interaction in different languages, reduces deployment costs, and avoids wasting machine resources.

FIG. 2 is a schematic flowchart of another voice interaction method shown in the present application. This method can be applied to the server. In the method of FIG. 2 , the training materials in different languages are used as audio training materials (referred to as training audio) as an example, but are not limited thereto, and may also be video training materials (referred to as training videos).

Referring to Figure 2, the method includes:

S201. Acquire phonemes of different languages, and pre-merge the phonemes of different languages according to the rules of the IWC; and merge the pre-merged phonemes into the first syllable according to the phonetic rules.

The first syllable can be obtained by combining the phonemes of the initial and the final from the pre-merged phonemes, and the remaining phonemes of the single initial and the single final are used as the first syllable.

S202 , acquiring training audios in different languages, identifying the combined syllables of the pronunciation from the training audios, and selecting from the combined syllables the combined syllables that conform to the syllable merging rule.

The syllable merging rules include at least one of the following rules: initial + initial + final; initial + initial + final + special initial; initial + final + special initial.

It should be noted that there is no sequence relationship between S201 and S202.

S203, performing forced frame alignment on the combined syllables that meet the syllable merging rules, and determining the average pronunciation duration of the aligned combined syllables and the average pronunciation duration of all combined syllables.

Among them, the combined syllables can be forced to frame alignment with the finals as the core.

The average pronunciation duration of the combined syllable is determined according to the ratio of the total pronunciation duration after forced frame alignment to the number of times the combined syllable appears in the training audio.

S204, taking the ratio of the average pronunciation duration of the combined syllable to the average pronunciation duration of all combined syllables as the articulation stickiness, and taking the combined syllable whose pronunciation stickiness is less than the set threshold as the second syllable.

S205. Combine the first syllable and the second syllable to obtain a modeled syllable.

Wherein, for S205, reference may be made to the description of S103, which will not be repeated here.

S206. Generate an acoustic model according to the modeled syllable.

For S206, reference may be made to the description of S104, which will not be repeated here.

S207: Receive a voice request sent by a user in the vehicle cabin forwarded by the vehicle.

The server may receive the voice request sent by the user in the vehicle cabin forwarded by the vehicle. The voice request may be a French voice request or a German voice request, etc.

S208: Recognize the voice request according to the acoustic model, generate a recognition result and send it to the vehicle to complete the voice interaction.

After receiving the voice request from the user, the server recognizes the voice request according to the acoustic model, generates a recognition result and sends it to the vehicle to complete the voice interaction. The method for recognizing the voice request according to the acoustic model may adopt the existing recognition method in the related art, which is not limited in this application.

In the solution of the present application, phoneme merging based on ipa can make a softmax (logistic regression) output layer available for mixing dozens of languages, which can reduce the amount of calculation and delay. This application generates syllables by combining phonetic rules and training audio data statistics, which can increase the modeling units of a single language from 40 to 500, thereby greatly reducing the difficulty of learning, improving the speech recognition rate, and making it more convenient and different. Language speech recognition and speech interaction.

FIG. 3 is a schematic flowchart of voice modeling in the voice interaction method shown in the first application of the present application. This method can be applied to the server. In the method of FIG. 3 , training materials in different languages are used as audio training materials (referred to as training audio for short) as an example. The training materials in different languages in this application may be audio training materials or video training materials, or audio training materials and video training materials used together.

Referring to Figure 3, the method includes:

S301. Acquire phonemes of different languages.

Taking Europe as an example, there are many European languages and there are several language families, with more than ten languages.

In this step, phonemes of different languages can be obtained, for example, French phonemes, English phonemes, and German phonemes can be obtained. For example, the acquired French phonemes include: brem...; the acquired German phonemes include: kam....

A phone is the smallest unit of speech divided according to the natural properties of speech. It is analyzed according to the pronunciation action in the syllable, and an action constitutes a phoneme. Phonemes can be divided into two categories: vowels and consonants. For example, the Chinese syllable "ah" (ā) has only one phoneme, "love" (ài) has two phonemes, and "dai" (dài) has three phonemes. A phoneme is also the smallest unit or the smallest segment of speech that makes up a syllable. The phonetic symbols of the International Phonetic Alphabet (also known as the "Universal Phonetic Alphabet") correspond one-to-one with the phonemes of all human languages.

A syllable is the smallest phonetic unit in a language in which a single vowel phoneme and a consonant phoneme are combined and pronounced. A syllable refers to the basic unit of phonetic structure composed of one or several phonemes in phonetics; and a phoneme is the smallest phonetic unit. For example, the syllable of the Chinese character "好" is h_ao3, where the 3 represents the tone. It should be noted that syllables can also be without tones.

S302. Pre-merge phonemes of different languages according to ipa.

Among the different languages in Europe, many languages actually belong to the same language family and have a lot in common in pronunciation. This application fully considers the correlation between different languages in Europe, and uses the international unified standard ipa (International Phonetic Alphabet, also known as the International Phonetic Alphabet) to merge the phonemes of different languages. ipa is a system for notation, based on the Latin alphabet, devised by the International Phonetic Society as a standardized method of notation for spoken sounds. The characteristic of ipa is that the more similar languages, the more phonemes that overlap. For example, in ipa, English has 39 phonemes and French has 36 phonemes, but 19 of them are coincident, so after combining English phonemes and French phonemes with ipa, there are only 56 phonemes, thus reducing the number of phonemes through the merging process .

Since the phonemes of multiple different languages are merged, multiple languages can be directly mixed and modeled in one model. There is only one output layer, but all languages are included, so there is no need to perform language discrimination processing. Through ipa-based phoneme merging, multiple languages can share modeling units, and then share some data, so that different languages can improve each other's effects.

S303. Combine the pre-merged phonemes into a first syllable according to phonetic rules.

In S303, the first syllable can be obtained by combining the phonemes of the initial consonant and the final consonant from the phonemes after the phoneme pre-merging by ipa, and the remaining single initial consonant phoneme and the phoneme of the single final consonant are used alone as the first syllable.

The phonetic rule of the present application may be to combine the initial consonant and the final consonant as one syllable, and after the combination, a single initial consonant and a single final consonant are left, which are also used as a single syllable. The syllables obtained after merging based on phonetic rules may generally contain 1-2 phonemes. The initial consonant, that is, the consonant before the final, forms a complete syllable together with the final. Consonants are sounds produced by various obstructions to the airflow in the mouth. A final composed of a vowel is called a single final, also known as a single final. Some syllables have no initials at the beginning, and only one final can become a syllable independently.

For example: Salad (5 phonemes included: s ae l ax d) merge -> (3 syllables: s_ae l_axd), so 5 phonemes are merged into 3 syllables.

For example, the acquired French phonemes brem... and the acquired German phonemes kam... etc. are combined with phonemes according to phonetic rules to obtain syllables b_e, r_e, k_a, etc. as the first syllable. The first syllable generally contains 1-2 phonemes.

This application performs phoneme merging based on ipa, which is context-independent, so that context-dependent triphones can be converted into context-independent syllables, and the number of pronunciation units can be sufficiently large, which is more distinguishable from each other and easier to learn. The number of context-independent syllables is taken into account, and larger pronunciation units have stronger anti-noise ability.

Context-dependent means that the modeling unit is related to the context in which it is located. The same phonetic symbol in different contexts is also a different modeling unit. Context-free means that as long as the diacritics are the same, regardless of the context, they are the same modeling unit.

The context-dependent modeling method is used, which is highly discriminative. The same modeling unit has the same context, and the modeling unit is easy to learn. However, one of the disadvantages is the large number of modeling units. For example, suppose there are 10 languages. If there are 30 phonemes in a language, then the modeling units for diphones are 300*300=90,000, and the modeling units for triphones are 300*300*300. Generally, it is difficult for the system to accept a large number of modeling units. Therefore, context-sensitive modeling units are generally not available for multilingualism. Another disadvantage of context-sensitive modeling is that it has poor transferability. If it is highly bound to the training corpus, for example, a model trained with a music corpus is difficult to transfer to navigation. Therefore, related technologies use context-dependent modeling methods, which are highly dependent on scenes and cannot achieve unified modeling in multiple languages.

FIG. 5 is a schematic diagram showing the comparison between context-dependent modeling and context-independent modeling shown in the present application. Referring to FIG. 5 , taking the ae pronunciation symbol as an example, for the English word happy, its phoneme includes haepiy, and for the English word salad, its phoneme includes saelaxd. If the context-dependent modeling method is used, two modeling units h_ae and p_iy are included. If the context-independent modeling method of the present application is used, only one modeling unit ae is included.

Referring further to FIG. 6, FIG. 6 is a schematic diagram of the modeling unit shown in the present application. As shown in Figure 6, the left box is the context-dependent phoneme, x represents the context; the middle box is the context-independent phoneme, and the two ae represent the same modeling unit; the right box is the context-independent phoneme The phonemes are merged into context-free syllables.

S304 , acquiring training audios in different languages, and identifying the combined syllables of the pronunciation from the training audios.

In this step, training audios in different languages can be obtained, for example, French training audios, English training audios, German training audios, and the like can be obtained.

Using the existing speech recognition technology, the combined syllables of pronunciation, such as b_r_e, b_e_m, r_e_m, r_e_b, etc., can be identified from the training audio.

It should be noted that there is no sequence relationship between S304 and S301.

S305. Screen out the combined syllables that conform to the syllable combination rule from the combined syllables.

After identifying the combined syllables of pronunciation from the training audio, it is judged whether these combined syllables conform to the syllable merging rule, and the combined syllables that conform to the syllable merging rule are screened out according to the judgment result.

Syllable merging rules, including at least one of the following rules: initial + initial + final; initial + initial + final + special initial; initial + final + special initial.

1) Initial + initial + final

2) Initial + initial + final + special initial [n/m]

For example, b_r_i_n, b_r_e_m, where the special initial n/m is a monophone with short cohesiveness that is commonly pronounced in monophone statistics.

3) Initial + final + special initial [n/m]

After identifying the combined syllables of pronunciation, such as b_r_e, b_e_m, r_e_m, and r_e_b from the training audio, filter according to the syllable merging rules, and filter out the combined syllables b_r_e, b_e_m, r_e_m that meet the syllable merging rules, where r_e_b is the end of the initial consonant, Not finals or special initials endings, and therefore do not meet the syllable merger rules are excluded.

S306, performing forced frame alignment on the combined syllables that conform to the syllable merging rule.

From the training audio, the combined syllables that conform to the syllable merging rules are subjected to forced frame alignment to obtain the result of forced frame alignment.

Forced frame alignment refers to the process of obtaining the corresponding annotation of each frame from the known audio and corresponding text annotations. Among them, the combined syllables in the training audio can be forced to frame alignment with the finals as the core.

For example: for 200 frames of audio, the text is marked as n i3 h ao3 (where 3 represents the tone), and the forced frame alignment is performed. The alignment result is: n(1-30)i3(31-100)h(101-120)ao3 (121-200). That is, frames 1-30 are n, frames 31-100 are i3, frames 101-120 are h, and frames 121-200 are ao3.

It should be noted that, in the speech modeling, it may be modeling with or without tones.

S307. Determine the average pronunciation duration of the aligned combined syllables and the average pronunciation duration of all combined syllables.

After obtaining the result of forced frame alignment, for the aligned combined syllables, count the pronunciation durations of all combined syllables of length 3 and 4 (including phoneme combinations) in different contexts, and take the average as the combined syllables (belonging to The average pronunciation duration of the consonants). The average pronunciation duration of the combined syllable can be determined according to the ratio of the total pronunciation duration after forced frame alignment to the number of times the combined syllable appears in the training audio. In addition, the average pronunciation duration of all combined syllables is also counted.

For example, the three phonemes b, r, and e of the combined syllable b_r_e appear 1000 times in the training audio. After alignment (b+r+e), the total pronunciation duration is 500s. The average pronunciation duration of the phoneme combinations is N(b_r_e)=500/1000=0.5s.

For another example, the a_b phoneme combination appears 1000 times in 10000 sentences, and the duration of the 1000 times is added up and divided by 1000, and the average pronunciation duration of the a_b phoneme combination can be obtained.

S308: Determine the pronunciation cohesion, and use the combined syllable whose pronunciation cohesion is less than the set threshold as the second syllable.

Wherein, the ratio of the average pronunciation duration of the combined syllable to the average pronunciation duration of all combined syllables can be used as the pronunciation stickiness, and the combined syllable with the pronunciation stickiness less than the set threshold can be used as the second syllable. The smaller the pronunciation cohesion, the more the phonemes in the combined syllable should be grouped together.

Assuming that the set threshold is 0.5, when the pronunciation cohesion is less than 0.5, the coherent combined syllable is output as the second syllable.

The following is an example of the determination process of pronunciation adhesion in different situations in the syllable merging rule, where xym in the relevant formula represents initials, z represents finals, s represents any initials/finals, N represents the average duration of pronunciation, and p represents Pronunciation stickiness, T represents the duration of pronunciation.

1) Initial + initial + final: pronunciation adhesion of x, y, z

2) Initial + initial + final + special initial: pronunciation stickiness of x, y, z, m

3) Initial + final + special initial: pronunciation adhesion of x, z, m

Among them, taking the pronunciation adhesion of x, y, and z as an example, the denominator in the formula represents the average value (average pronunciation duration) of all the phoneme combinations x, s, and z (all combined syllables); the numerator represents the phoneme combination x ,y,z (aligned combined syllables) average (average pronunciation duration). The smaller the pronunciation stickiness p, the shorter the duration of the three phonemes x, y, and z when they appear, the more sticky they are, and the more they should be grouped together.

for example:

The pronunciation stickiness of the combined syllable b_r_e is:

p(b_r_e)=N(b_r_e)/N(b_*_e)

Among them, * represents all initials, N represents the average pronunciation duration, N(b_r_e) represents the average pronunciation duration of the combined syllable b_r_e, N(b_*_e) represents the combined syllables of all initials combined with b and e. Average pronunciation duration.

If b_r_e has high stickiness, the average pronunciation duration of b_r_e is generally shorter than that of b_*_e.

The pronunciation stickiness of the combined syllable r_e_m is:

p(r_e_m)=N(r_e_m)/N(r_e_*)

Among them, * represents all initials, N represents the average pronunciation duration, N(r_e_m) represents the average pronunciation duration of the combined syllable r_e_m, and N(r_e_*) represents the average of all combined syllables after all initials are combined with r and e. Pronunciation duration.

For example, the above-mentioned combined syllables b_r_e, b_e_m, and r_e_m that meet the syllable merging rules are judged by the degree of pronunciation adhesion, and the qualified combined syllables b_r_e and b_e_m are obtained as the second syllables. The second syllable after the judgment of pronunciation adhesion generally contains 3-4 phonemes.

S309. Combine the first syllable and the second syllable to obtain a modeled syllable.

The first syllable and the second syllable obtained respectively in the above different steps are combined to generate the final syllable, that is, the modeled syllable.

For example, the first syllables b_e, r_e, k_a and the second syllables b_r_e, b_e_m obtained above are combined to obtain the final modeled syllables b_e, r_e, k_a, b_r_e, b_e_m . . .

The modeling method of the present application can be context-independent, so that the number of modeling units can be reduced, and different contexts can share the same modeling unit, so that dozens of languages can use the same modeling system, so that multiple languages can be realized. Unified modeling is more convenient for speech recognition and speech interaction in different languages. The application can realize data sharing, and the same modeling unit may exist in different languages, which can greatly reduce the dependence on the data volume of a single language. This application combines phonemes into syllables based on ipa, and generates modeling syllables by combining phonetics rules and training audio data statistics, which can increase the modeling units of a single language from 40 to 500, thereby greatly reducing learning. Difficulty, improve the recognition rate. This application uses ipa-based syllable modeling technology, so that dozens of languages can be mixed using a softmax output layer, which not only reduces the amount of calculation and delay, but also enables the comparison between different languages. It has become a target that can be learned without artificially giving different language weights to compare. The context-independent syllable modeling technology of the present application based on ipa enables the modeling unit to be converted from context-dependent triphones to context-independent syllables, so that the number of pronunciation units is sufficiently large, and the distinction between them is greater and easier. learning, while taking into account the number of context-independent syllables, and larger pronunciation units have stronger anti-noise ability.

FIG. 4 is a schematic diagram of an application framework for speech recognition using speech modeling shown in the present application.

Referring to FIG. 4 , the on-board system of the vehicle sends the voice request (query) to the server after receiving the user in the cockpit, and the server receives the voice request sent by the user in the cockpit of the vehicle forwarded by the vehicle, and uses this application according to modeling syllables. The established acoustic model and related decoder perform speech recognition on the voice request, obtain the speech recognition result, and send the speech recognition result to the vehicle to complete the speech interaction. Among them, the construction process of the acoustic model can be seen on the right side of Figure 4. For the voice and audio sent by the user, a neural network layer such as the LSTM (Long short-term memory, long short-term memory network) hidden layer can be used to process the output phoneme features. Then, based on the ipa, the phonemes of different languages, such as German phonemes, English phonemes, French phonemes, etc., are combined phoneme. Dozens of different languages can be mixed using a softmax output layer, and subsequent language discrimination processing is not required. A more detailed construction process of the acoustic model of the present application can be described with reference to the flow chart of FIG. 3 . The present application uses ipa-based phoneme merging, so that multiple languages can share modeling units, and then share some data, so that different languages can improve the effect of each other.

Corresponding to the foregoing application function implementation method, the present application further provides a server.

FIG. 7 is a schematic structural diagram of a server shown in this application.

7 , the server 70 provided by this application includes: a phoneme processing module 71 , a training processing module 72 , a syllable merging module 73 , a model generating module 74 , a request receiving module 75 , and a speech recognition module 76 .

The phoneme processing module 71 is configured to acquire phonemes of different languages, and combine the phonemes of different languages into the first syllable according to the phonetic rules. The phoneme processing module 71 can pre-merge phonemes of different languages according to the rules of the Universal Phonetic Symbols; and merge the pre-merged phonemes into the first syllable according to the phonetic rules.

The training processing module 72 is used for acquiring training audios of different languages, identifying the combined syllables of pronunciation by using the training materials of different languages, and selecting the second syllable from the combined syllables according to the degree of pronunciation adhesion. The training processing module 72 can perform forced frame alignment of the combined syllables; determine the average pronunciation duration of the aligned combined syllables and the average pronunciation duration of all combined syllables; The ratio of the duration is used as the pronunciation cohesion, and the combined syllable whose pronunciation cohesion is less than the set threshold is regarded as the second syllable. The present application can use the audio and/or video training materials in different languages to recognize the combined syllables of pronunciation.

The syllable combining module 73 is configured to combine the first syllable obtained by the phoneme processing module 71 and the second syllable obtained by the training processing module 72 to obtain a modeled syllable.

The model generation module 74 is used for generating an acoustic model according to the modeled syllables.

The request receiving module 75 is configured to receive the voice request sent by the user in the vehicle cabin forwarded by the vehicle.

The speech recognition module 76 is used for recognizing the speech request received by the request receiving module 75 according to the acoustic model generated by the model generation module 74, and generates a recognition result and sends it to the vehicle to complete the speech interaction.

FIG. 8 is a schematic structural diagram of another server shown in the present application.

8 , the server 70 provided by this application includes: a phoneme processing module 71 , a training processing module 72 , a syllable merging module 73 , a model generating module 74 , a request receiving module 75 , and a speech recognition module 76 .

Wherein, the phoneme processing module 71 includes: a pre-merging sub-module 711 and a phoneme-merging sub-module 712 .

The pre-merging sub-module 711 is used to pre-merge phonemes of different languages according to the rules of the Universal Phonetic Alphabet.

The phoneme merging sub-module 712 is used for merging the pre-merged phonemes into the first syllable according to the phonetic rule. For example, the phoneme merging submodule 712 may combine the phonemes of the initial consonant and the final vowel from the pre-merged phonemes to obtain the first syllable, and use the remaining phoneme of the single initial consonant and the phoneme of the single final as the first syllable alone.

The training processing module 72 includes: an alignment and statistics module 721 and a syllable screening module 722 .

The alignment and statistics module 721 is configured to perform forced frame alignment of the combined syllables to obtain the average pronunciation duration of the aligned combined syllables and the average pronunciation duration of all combined syllables.

The syllable screening module 722 is configured to use the ratio of the average pronunciation duration of the combined syllable to the average pronunciation duration of all combined syllables as the articulation stickiness, and use the combined syllable whose pronunciation stickiness is less than the set threshold as the second syllable. The average pronunciation duration of the combined syllable can be determined according to the ratio of the total pronunciation duration after forced frame alignment to the number of times the combined syllable appears in the training audio.

The training processing module 72 may further include: a rule screening module 723 .

The rule screening module 723 selects the combined syllables conforming to the syllable merging rules from the combined syllables; the alignment and statistics module 721 enforces frame alignment of the combined syllables that meet the syllable combining rules.

The syllable merging rules include at least one of the following rules: initial + initial + final; initial + initial + final + special initial; initial + final + special initial.

The server of the present application combines the phonemes of different languages into the first syllable according to the phonetic rules, selects the second syllable from the combined syllables of the training audio according to the pronunciation stickiness, and then merges to obtain the modeling syllables, these syllables have nothing to do with the context, In this way, various languages can use the same modeling system, so that multi-language unified modeling can be achieved, which is more convenient for speech recognition and speech interaction in different languages, reduces deployment costs, and avoids wasting machine resources.

Regarding the server in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

FIG. 9 is another schematic structural diagram of the server shown in this application.

Referring to FIG. 9 , the server 1000 includes a memory 1010 and a processor 1020 .

The processor 1020 may be a central processing unit (Central Processing Unit, CPU), and may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-available processors Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

Memory 1010 may include various types of storage units, such as system memory, read only memory (ROM), and persistent storage. The ROM may store static data or instructions required by the processor 1020 or other modules of the computer. Persistent storage devices may be readable and writable storage devices. Permanent storage may be a non-volatile storage device that does not lose stored instructions and data even if the computer is powered off. In some embodiments, persistent storage devices employ mass storage devices (eg, magnetic or optical disks, flash memory) as persistent storage devices. In other embodiments, persistent storage may be a removable storage device (eg, a floppy disk, an optical drive). System memory can be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. System memory can store some or all of the instructions and data that the processor needs at runtime. Additionally, memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (eg, DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic and/or optical disks may also be employed. In some embodiments, memory 1010 may include a removable storage device that is readable and/or writable, such as a compact disc (CD), a read-only digital versatile disc (eg, DVD-ROM, dual-layer DVD-ROM), Read-only Blu-ray Disc, Ultra-Density Disc, Flash Card (eg SD Card, Min SD Card, Micro-SD Card, etc.), Magnetic Floppy Disk, etc. Computer readable storage media do not contain carrier waves and transient electronic signals transmitted over wireless or wire.

Executable codes are stored on the memory 1010, and when the executable codes are processed by the processor 1020, the processor 1020 can be caused to execute some or all of the above-mentioned methods.

Furthermore, the method according to the present application can also be implemented as a computer program or computer program product comprising computer program code instructions for performing some or all of the steps in the above method of the present application.

Alternatively, the present application can also be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) on which executable codes (or computer programs or computer instruction codes) are stored, When the executable code (or computer program or computer instruction code) is executed by the processor of the electronic device (or server, etc.), the processor is caused to perform some or all of the steps of the above method according to the present application.

Various embodiments of the present application have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (12)

1. A method for voice interaction, comprising:

acquiring phonemes of different languages, and combining the phonemes of the different languages into a first syllable according to a pronunciation rule;

recognizing the combination syllables of pronunciation by using training materials of different languages, and screening out second syllables from the combination syllables according to pronunciation adhesion;

merging the first syllable and the second syllable to obtain a modeling syllable;

generating an acoustic model according to the modeling syllables;

receiving a voice request sent by a user in a vehicle cabin and forwarded by a vehicle;

and recognizing the voice request according to the acoustic model, generating a recognition result and sending the recognition result to the vehicle to finish voice interaction.

2. The method of claim 1, wherein said combining phones of said different languages into a first syllable according to pronunciation rules comprises:

pre-combining the phonemes of different languages according to the universal phonetic symbol rule;

and merging the pre-merged phonemes into a first syllable according to a pronunciation rule.

3. The method of claim 2 wherein said combining said pre-combined phones into a first syllable according to pronunciation rules comprises:

and combining the phones of the initial consonants and the final consonants to obtain a first syllable from the pre-combined phones, and taking the remaining phones of the single initial consonants and the remaining phones of the single final consonants as the first syllable independently.

4. The method of claim 1, wherein the identifying of the pronounced combined syllable using the training materials of different languages comprises:

the combined syllables of the pronunciation are identified using audio and/or video training material of different languages.

5. The method of claim 1, wherein the selecting the second syllable from the combined syllables based on pronunciation adhesion comprises:

performing forced frame alignment on the combined syllable;

determining the average pronunciation duration of the aligned combined syllables and the average pronunciation duration of all the combined syllables;

and taking the ratio of the average pronunciation duration of the combined syllables to the average pronunciation duration of all the combined syllables as pronunciation adhesion, and taking the combined syllables with the pronunciation adhesion smaller than a set threshold value as second syllables.

6. The method of claim 5,

before the forced frame alignment of the combined syllable, the method further comprises the following steps: screening out combined syllables which accord with a syllable combination rule from the combined syllables;

the forced frame alignment of the combined syllable comprises: and performing forced frame alignment on the combined syllables according with the syllable merging rule.

7. The method of claim 5, wherein the rules for syllable combination include at least one of the following rules:

initial + final;

initial + final + special initial;

initial consonant + vowel + special initial consonant.

8. The method of claim 5, wherein:

and the average pronunciation duration of the combined syllable is determined according to the ratio of the total pronunciation duration after forced frame alignment to the occurrence frequency of the combined syllable in the training audio.

9. The method of claim 5, wherein said forcing frame alignment of said combined syllable comprises:

and performing forced frame alignment on the combined syllable by taking a final as a core.

10. A server, comprising:

the phoneme processing module is used for acquiring phonemes of different languages and combining the phonemes of the different languages into a first syllable according to a pronunciation rule;

the training processing module is used for recognizing the combined syllables of the pronunciation by using training materials of different languages and screening out second syllables from the combined syllables according to the pronunciation adhesion degree;

the syllable merging module is used for merging the first syllable obtained by the phoneme processing module and the second syllable obtained by the training processing module to obtain a modeling syllable;

the model generation module is used for generating an acoustic model according to the modeling syllables;

the request receiving module is used for receiving a voice request sent by a user in a vehicle cabin and forwarded by a vehicle;

and the voice recognition module is used for recognizing the voice request received by the request receiving module according to the acoustic model generated by the model generating module, generating a recognition result and transmitting the recognition result to the vehicle to finish voice interaction.

11. A server, comprising:

a processor; and

a memory having executable code stored thereon, which when executed by the processor, causes the processor to perform the method of any one of claims 1-9.

12. A computer-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-9.

Images