CN112489622B - A multi-language continuous speech stream speech content recognition method and system - Google Patents

Abstract

The invention provides a method and a system for recognizing multi-language continuous voice stream voice content, wherein the method comprises the following steps: inputting the multi-language continuous voice stream to be identified into a frame-level language classification model, and outputting segment-level language feature vectors; inputting the segment level language feature vector into a segment level language classification model, and outputting posterior probability distribution of the segment level language state; calculating an optimal language state path of the multi-language continuous voice stream based on a Viterbi search algorithm according to posterior probability distribution of the segment-level language states; dividing the multi-language continuous voice stream to be recognized according to the optimal language state path to obtain a language state interval; and sending the segmented language state interval into a multilingual acoustic model and a corresponding multilingual decoder for decoding to obtain a content recognition result of the multilingual continuous voice stream. The invention solves the problem of dynamic detection and recognition of language types with concurrent multilingual content in continuous voice streams by fusing the language classification model with the Viterbi retrieval algorithm.

Description

A multi-language continuous speech stream speech content recognition method and system

Technical field

The present invention relates to the field of speech recognition, specifically, to a multi-language continuous speech stream speech content recognition method and system.

Background technique

With the application of hidden Markov technology and deep neural network technology in the field of automatic speech recognition, automatic speech recognition technology has achieved unprecedented development. For languages such as Chinese and English that are widely spoken, the performance of the corresponding single-language speech recognition system can even reach human recognition levels. With the economic and trade exchanges between countries around the world, the economic and cultural integration of countries around the world is accelerating. Building a hybrid multi-language speech recognition system has become a necessary condition for detecting multi-language speech stream content.

The traditional multi-language speech recognition system is based on the language recognition front-end connected in series with multiple parallel single-language speech recognition system back-ends. Generally speaking, the language recognition front-end performs sentence-level classification and judgment on the language type of the speech based on the phonetic characteristics of the entire speech. In the multi-language recognition task of multi-language continuous speech streams, this language classification method based on the sentence level cannot cope with the language classification task of multiple languages coexisting in the speech stream.

Contents of the invention

The purpose of the present invention is to solve the problem that the language classification method based on the sentence level cannot cope with the language classification task of multiple languages coexisting in the speech stream.

In order to achieve the above objectives, the present invention proposes a multi-language continuous speech stream speech content recognition method, which method includes:

Input the multi-language continuous speech stream to be recognized into the frame-level language classification model and output the segment-level language feature vector;

Input the segment-level language feature vector into the segment-level language classification model, and output the posterior probability distribution of the segment-level language status;

According to the posterior probability distribution of the segment-level language status and based on the Viterbi retrieval algorithm, the optimal language status path of the multi-language continuous speech stream is calculated;

Segment the multi-language continuous speech stream to be recognized according to the optimal language status path to obtain the language status interval;

The language status interval is sent to the multi-language acoustic model and the corresponding multi-language decoder for decoding, and the content recognition result of the multi-language continuous speech stream is obtained.

As an improvement of the method, the method also includes a training step of a multi-language acoustic model, and the specific steps are:

Step 1-1) Construct a multi-language acoustic model based on a multi-task learning framework, the model including several shared hidden layers and language-specific output layers;

Step 1-2) Extract the spectral features of the multi-language continuous speech stream of the training set based on the acoustic state labels of the multi-language speech data, input the spectral features into the shared hidden layer for non-linear transformation; output several single-language data to several language-specific output layer;

Step 1-3) Calculate the error loss function value of the single language data in the language-specific output layer corresponding to the input spectral feature, then the error loss function is:

where F _loss,i is the error loss value of the i-th language-specific output layer, p _model,i (x _L ) is the output of the L-th language-specific output layer corresponding to the spectral feature x _L of the L-th language, q _label,L is the acoustic state label corresponding to the spectral feature x _L ; the error loss function value of other output layers is zero;

Step 1-4) Return the error loss value F _loss,i in reverse; each language-specific output layer parameter is updated according to the data of the corresponding single language, and the language-specific output layer parameter gradient ΔΦ _i is calculated:

where Φ _i is the parameter of the i-th language-specific output layer;

The parameters of the shared hidden layer are calculated from the error loss values F _loss,i returned by several language-specific output layers: Calculate the gradient ΔΦ of the shared hidden layer parameters:

Among them, Φ is the parameter of the shared hidden layer, and L is the number of language categories corresponding to the specific language output layer of the multi-language acoustic model;

Step 1-5) When F _loss,i > given threshold, then go to step 1-2),

When F _loss,i < given threshold, the trained multi-language acoustic model is obtained.

As an improvement of the method, the method also includes a training step of a frame-level language classification model. The specific steps are:

Step 2-1) Construct a frame-level language classification model, and the frame-level language classification model is a deep neural network;

Step 2-2) Extract the frame-level spectral features of the multi-language continuous speech stream of the training set, input the frame-level spectral features into the frame-level language classification model, perform long-term statistics on the output vector of the current hidden layer, and calculate the current hidden layer. Contains the mean vector, variance vector and segment-level language feature vector of the layer output vector;

The mean vector is:

The variance vector is:

The segment-level language feature vector:

h _segment =Append(μ,σ) (6)

where h _i is the output vector of the current hidden layer at time i, T is the long-term statistical period, μ is the mean vector of long-term statistics, σ is the variance vector of long-term statistics, h _segment is the segment-level language feature vector, so The paragraph-level language feature vector is the mean vector and variance vector spliced together, and its dimension is twice the h _i dimension; Append(μ,σ) means that μ and σ are spliced together to form a high-dimensional vector;

Step 2-3) Use the mean vector and variance vector as the input of the next hidden layer, and train them through error calculation and reverse gradient backpropagation process according to the frame-level language label, so that each hidden layer outputs a segment-level language feature vector, Obtain the trained frame-level language classification model.

As an improvement of the method, the method also includes a training step of a segment-level language classification model. The specific steps are:

Step S2-1) Construct a segment-level language classification model;

Step S2-2) Extract the frame-level spectral features of the multi-language continuous speech stream of the training set, input the frame-level spectral features into the hidden layer of the trained frame-level language classification model, and extract the frame-level language classification model from the trained frame-level language classification model. Extract segment-level language feature vectors from the hidden layer;

Step S2-3) Set a segment-level language label for each segment-level language feature vector, input the segment-level language feature vector into the segment-level language classification model, and train and output the posterior probability distribution of the language status corresponding to the segment-level language label, Obtain the trained segment-level language classification model.

As an improvement of the method, the multi-language continuous speech stream to be recognized is input into a frame-level language classification model and a segment-level language feature vector is output; the segment-level language feature vector is input into the segment-level language classification model to output the language status. Posterior probability distribution; specifically includes:

Extract the frame-level spectral features to be recognized from the multi-language continuous speech stream to be recognized;

Input the frame-level spectral features to be recognized into the trained frame-level language classification model according to specific step sizes and window lengths, and output the segment-level language feature vector h _segment ;

The segment-level language feature vector h _segment is input into the trained segment-level language classification model, and the posterior probability distribution of the language status corresponding to the segment-level language feature vector is output.

As an improvement of the method, according to the posterior probability distribution of the language status and based on the Viterbi retrieval algorithm, the optimal language status path of the multi-language continuous speech stream is calculated, specifically including:

Step 3-1) According to the posterior probability distribution of the language state, set the rotation probability p _loop and jump probability p _skip of the language state retrieved by Viterbi, and obtain the transition matrix A of the language state as:

Among them, p _loop represents the rotation probability of the language status, and p _skip represents the jump probability of the language status. The rotation probability and jump probability values of each language are the same. The language status labels are set according to the language category. The language status labels are different. The language category label uses Arabic numerals 1, 2,...,N as the language status label; the corresponding relationship between each element of the transfer matrix A and the language status label is:

Step 3-2) Perform Viterbi retrieval on the predicted language status sequence, and calculate the objective function based on Viterbi retrieval:

where p _trans (s _T+1 |s _T ) represents the transition probability from the language state s _{T of the multilingual continuous speech stream at time T} to the language state s T+1 at time _T+1 :

Among them, the language classification labels corresponding to the language status s _T and the language status s _T+1 are within the marked range of language classification labels, and T is the statistical period corresponding to the segment-level language feature h _segment ;

p _emit (s _T+1 |h _segment ) represents the posterior probability predicted for the segment-level language feature h _segment on the language state s _T+1 :

p _emit (s _T+1 |h _segment )=DNN-LID _{segment level} (h _segment ) (11)

Among them, DNN-LID is a segment-level language classifier based on the deep neural network DNN;

Step 3-3) Using the objective function The language status sequence with the largest value is the best language status sequence. Based on the best language status sequence, language status backtracking is performed to obtain the best language status path.

The present invention also proposes a multi-language continuous speech stream speech content recognition system, which system includes:

The segment-level language feature extraction module is used to input the multi-language continuous speech stream to be recognized into the frame-level language classification model and output the segment-level language feature vector;

The posterior probability calculation module of language status inputs the segment-level language feature vector into the segment-level language classification model and outputs the posterior probability distribution of segment-level language status;

The language status path acquisition module is used to calculate the optimal language status path of multi-language speech streams based on the posterior probability distribution of segment-level language status and based on the Viterbi retrieval algorithm;

A language status interval segmentation module, configured to segment the multi-language continuous speech stream to be recognized according to the optimal language status path to obtain the language status interval; and

The content recognition module of the multilingual speech stream is used to send the segmented language status intervals into the multilingual acoustic model and the corresponding multilingual decoder for decoding, and obtain the content recognition result of the multilingual speech stream.

The present invention also proposes a computer device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above is implemented. method described.

The present invention also proposes a computer-readable storage medium that stores a computer program. When executed by a processor, the computer program causes the processor to perform any of the above methods.

Compared with the existing technology, the advantages of the present invention are:

1. The multi-language continuous speech stream speech content recognition method and system of the present invention can solve the problem of dynamic detection of language types in the continuous speech stream when multi-language content coexists by integrating the language classification model and the Viterbi retrieval algorithm.

2. The multi-language continuous speech stream speech content recognition method of the present invention can perform dynamic language switching point discrimination and corresponding multi-language content recognition on the multi-language content in the continuous speech stream.

Description of the drawings

Figure 1 is a schematic diagram of the multi-language continuous speech stream speech content recognition method of the present invention.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

The present invention proposes the multi-language continuous speech stream speech content recognition method and system of the present invention. The method includes:

Step 1) Construct a multi-language acoustic model based on multi-task learning; this acoustic model unifies the multi-language acoustic modeling tasks under a neural network classification framework based on multi-task learning, and simultaneously uses the acoustic features of multiple languages to Joint optimization of multi-language acoustic models; specifically including:

Step 1-1) Construct a multi-language acoustic model based on a multi-task learning neural network classification framework. The model consists of multiple shared hidden layers and language-specific output layers; the model parameters of the shared hidden layer are shared by multi-language data. Optimization; the language-specific output layer is optimized by the data of each single language;

Step 1-2) During the forward calculation process of the model, the shared hidden layer and the language-specific output layer of the multi-language acoustic model perform non-linear transformation on the input multi-language spectrum feature vector, and all language-specific output layers have information output ;

Step 1-3) In the calculation process of the error loss function of the model update, according to the acoustic state label corresponding to the spectral feature, the error loss function value is only calculated in the language-specific output layer corresponding to the spectral feature, and other values are not related to the language of the spectral feature. The error loss function value calculated by the corresponding language-specific output layer is zero; the corresponding loss function calculation formula is as follows:

where F _loss,i is the error loss function value of the i-th language-specific output layer, p _model,i (x _L ) is the acoustic model in the L-th language-specific output layer corresponding to the spectral feature x _L of the L-th language Output, q _label,L is the acoustic state label corresponding to the spectral feature x _L ;

Step 1-4) In the process of reverse transmission of the model classification error, the error loss value F _loss,i is transmitted back in reverse, and each language-specific output layer parameter is trained on the model parameters based on the data of the corresponding single language; shared The parameters of the hidden layer are calculated by the error loss values F _loss,i returned from several language-specific output layers;

The language-specific output layer parameter gradient calculation formula is:

where Φ _i is the parameter of the i-th language-specific output layer.

The gradient calculation formula of shared hidden layer parameters is:

Among them, Φ is the parameter of the shared hidden layer, and L is the number of language categories corresponding to the specific language output layer of the multi-language acoustic model.

Step 1-5) Repeat steps 1-2)-step 1-4) until the model parameters converge.

Step 2) Construct a frame-level language classification model that integrates long-term statistical features based on a deep neural network model; extract a language feature vector representing language type characteristics based on the frame-level language classification model; the frame-level language classification model integrates long-term statistical components, In the forward calculation process of the frame-level language classification model, the long-term statistics component performs segment-level statistics on the output vector of the previous hidden layer, calculates the mean and variance statistics of the output vector of the previous hidden layer, and compares the mean and variance The vector of statistics is used as the input of the next hidden layer, and finally the error calculation of the language classification model is performed based on the frame-level language label and the model is updated through the reverse gradient backpropagation process;

The specific steps for training the frame-level language classification model include:

Step 2-1) Construct a frame-level language classification model, and the frame-level language classification model is a deep neural network;

Step 2-2) Extract the frame-level spectral features of the multi-language continuous speech stream of the training set, use the frame-level spectral features as input features to input the frame-level language classification model, and perform long-term statistics on the output vector of the current hidden layer, Calculate the mean vector, variance vector and segment-level language feature vector of the current hidden layer output vector;

The mean vector is:

The variance vector is:

The segment-level language feature vector is:

h _segment =Append(μ,σ) (6)

where h _i is the output vector of the current hidden layer at time i, T is the long-term statistical period, μ is the mean vector of long-term statistics, σ is the variance vector of long-term statistics, h _segment is the segment-level language feature vector, so The paragraph-level language feature vector is the mean vector and variance vector spliced together, and its dimension is twice the h _i dimension; Append(μ,σ) means that μ and σ are spliced together to form a high-dimensional vector;

Step 2-3) Use the mean vector and variance vector as the input of the next hidden layer, and train them through error calculation and reverse gradient backpropagation process according to the frame-level language label, so that each hidden layer outputs a segment-level language feature vector, Obtain the trained frame-level language classification model.

Based on the trained frame-level language classification model, the segment-level language feature vector is extracted from the hidden layer of the frame-level language classification model, and a segment-level language label is constructed for each segment-level language feature vector. According to the segment-level language feature vector and segment The level language label trains the segment level language classification model. Specifically include:

Step S2-1) Construct a segment-level language classification model;

Step S2-2) Extract the frame-level spectral features of the multi-language continuous speech stream of the training set, use the frame-level spectral features as input features to input the hidden layer of the trained frame-level language classification model, and start from the trained frame-level Extract segment-level language feature vectors from the hidden layer of the language classification model;

Step S2-3) Set a segment-level language label for each segment-level language feature vector, input the segment-level language feature vector into the segment-level language classification model, and train and output the posterior probability distribution of the language status corresponding to the segment-level language label, Obtain the trained segment-level language classification model.

Step 3) Use the trained frame-level language classification model to extract segment-level language feature vectors from the multi-lingual continuous speech stream to be recognized, and classify the segment-level language feature vectors according to the segment-level language classification model, combined with the Viterbi retrieval algorithm. , conduct real-time detection of language switching points for multi-language continuous speech streams; finally, according to the language detection results, the continuous speech stream is segmented and the content of the multi-language speech stream is recognized through the multi-language acoustic model and the corresponding decoder. Specific steps include:

Step 3-1) Extract the segment-level language feature vector from the frame-level language classification model according to the specific step size and window length of the speech spectrum characteristics of the multi-language continuous speech stream to be recognized;

Classify the segment-level language feature vectors through the segment-level language classification model, and obtain the posterior probability distribution of the language status corresponding to the segment-level language feature vectors;

Set the rotation probability and jump probability of the language status of Viterbi retrieval, and reduce the language classification errors caused by the inaccurate classification of the segment-level language classification model by increasing the rotation probability of language filling; including:

Based on the posterior probability distribution of the language status, set the rotation probability and jump probability of the language status for Viterbi retrieval, and obtain the transition matrix A of the language status as:

Among them, p _loop represents the rotation probability of the language status, and p _skip represents the jump probability of the language status. The rotation probability and jump probability values of each language are the same. The language status labels are set according to the language category. The language status labels are different. The language category label uses Arabic numerals 1, 2 and N as the language status label; the corresponding relationship between each element of the transfer matrix A and the language status label is:

Step 3-2) Calculate the posterior probability p _emit (s _T+1 |h _segment ) of the predicted segment-level language status, and calculate the predicted language status based on the preset rotation probability p _loop and jump probability p _skip of the language status. Status for Viterbi retrieval, including:

The optimal language state sequence of the continuous speech stream is calculated based on the objective function of Viterbi retrieval. The objective function is:

where p _trans (s _T+1 |s _T ) represents the transition probability from the language state s _{T of the multilingual continuous speech stream at time T} to the language state s T+1 at time _T+1 :

Among them, the language classification labels corresponding to the language status s _T and the language status s _T+1 are within the marked range of language classification labels, and T is the statistical period corresponding to the segment-level language feature h _segment ;

p _emit (s _T+1 |h _segment ) represents the posterior probability of predicting the segment-level language feature h _segment on the language state s _T+1 ;

p _emit (s _T+1 |h _segment )=DNN-LID _{segment level} (h _segment ) (11)

Among them, DNN-LID is a segment-level language classifier based on the deep neural network DNN;

Step 3-3) Through the above recursive formula, the best language status can be predicted for retrieval through the posterior probability of the segment-level language status predicted by the segment-level language classification model and the preset rotation probability and jump probability of the language status. The sequence with the largest final objective function value is the best language status sequence corresponding to the multilingual continuous speech stream. The best language status path can be obtained by performing language status backtracking based on the best language status sequence.

Step 4) According to the optimal language status path, the multi-language speech stream can be segmented according to the language status interval, and the segmented language status interval speech stream is sent to the multi-language acoustic model and the corresponding multi-language decoder for decoding. , the corresponding content recognition result of the multi-language continuous speech stream can be obtained.

The present invention also proposes a multi-language continuous speech stream speech content recognition system, which system includes:

The segment-level language feature extraction module is used to input the multi-language continuous speech stream to be recognized into the frame-level language classification model and output the segment-level language feature vector;

The posterior probability calculation module of language status inputs the segment-level language feature vector into the segment-level language classification model and outputs the posterior probability distribution of segment-level language status;

The language status path acquisition module is used to calculate the optimal language status path of multi-language speech streams based on the posterior probability distribution of segment-level language status and based on the Viterbi retrieval algorithm;

A language status interval segmentation module, configured to segment the multi-language continuous speech stream to be recognized according to the optimal language status path to obtain the language status interval; and

The content recognition module of the multilingual speech stream is used to send the segmented language status intervals into the multilingual acoustic model and the corresponding multilingual decoder for decoding, and obtain the content recognition result of the multilingual speech stream.

The present invention also proposes a computer device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above is implemented. method described.

The present invention also proposes a computer-readable storage medium that stores a computer program. When executed by a processor, the computer program causes the processor to perform any of the above methods.

The rationality and effectiveness of the speech recognition system based on the present invention have been verified on the actual system. The results are shown in Table 1:

Table 1

The method of the present invention conducts joint training of multi-language acoustic models using Cantonese, Turkish and Vietnamese data, and simultaneously constructs a frame-level language classification model and a segment-level language classification model based on three languages, and utilizes Viterbi algorithm-based The multilingual continuous speech stream speech content recognition method performs language classification and speech content recognition of continuous multilingual speech. It can be seen from Table 1 that the accuracy of language recognition is improved from 82.1% to 92.4% through the method of the present invention. It is verified that the multi-language continuous speech stream speech content recognition method based on the Viterbi algorithm of the present invention can effectively improve the accuracy of continuous multi-language speech stream. The result of language detection.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the embodiments, those of ordinary skill in the art will understand that modifications or equivalent substitutions may be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and they shall all be covered by the scope of the present invention. within the scope of the claims.

Claims (8)

1. A method of multi-lingual continuous voice stream voice content recognition, the method comprising:

inputting the multi-language continuous voice stream to be identified into a frame-level language classification model, and outputting segment-level language feature vectors;

inputting the segment level language feature vector into a segment level language classification model, and outputting posterior probability distribution of the segment level language state;

calculating an optimal language state path of the multi-language continuous voice stream based on a Viterbi search algorithm according to posterior probability distribution of the segment-level language states;

dividing the multi-language continuous voice stream to be recognized according to the optimal language state path to obtain a language state interval;

inputting the language state interval into a multi-language acoustic model and a corresponding multi-language decoder for decoding to obtain a content identification result of the multi-language continuous voice stream;

according to the posterior probability distribution of segment-level language states, based on a Viterbi search algorithm, calculating an optimal language state path of the multi-language continuous voice stream, wherein the method specifically comprises the following steps:

step 3-1) setting the autorotation probability p of the language state of the Viterbi search according to the posterior probability distribution of the language state _loop And probability of jump p _skip The obtained transition matrix A of the language state is as follows:

the method comprises the steps of setting a language state label according to language categories, wherein the rotation probability and the jump probability of each language are the same, the language state label is a label of different language categories, arabic numerals 1 and 2 are adopted, and N is the language state label; the corresponding relation between each element of the transfer matrix A and the language state label is as follows:

step 3-2) carrying out Viterbi search on the predicted language state, and calculating an objective function based on Viterbi search:

wherein p is _trans (s _T+1 |s _T ) Language state s representing a multilingual continuous speech stream from time T _T Language state s up to time T+1 _T+1 Is a transition probability of (2):

wherein the language state s _T And language state s _T+1 The corresponding language classification label is within the range of the labeled language classification label, and T is the segment levelLanguage feature h _segment Corresponding statistical period;

p _emit (s _T+1 |h _segment ) Representing para-level language features h _segment In language state s _T+1 Posterior probability of upper prediction:

p _emit (s _T+1 |h _segment )＝DNNLID _{segment level} (h _segment ) (11)

The DNNLID is a segment-level language classifier based on a deep neural network DNN;

step 3-3) as an objective functionAnd carrying out language state backtracking according to the optimal language state sequence to obtain an optimal language state path.

2. The method for recognizing speech content in a multilingual continuous speech stream according to claim 1, further comprising a training step of the multilingual acoustic model, comprising the specific steps of:

step 1-1) constructing a multi-language acoustic model based on a multi-task learning neural network, wherein the model comprises a plurality of shared hidden layers and a language specific output layer;

step 1-2) extracting spectral features of multi-language continuous voice streams of a training set based on acoustic state labels of multi-language continuous voice data, and inputting the spectral features into a shared hidden layer for nonlinear transformation; outputting the data of a plurality of single languages to a plurality of language specific output layers;

step 1-3) calculating an error loss function value at a language specific output layer corresponding to the input spectral features from the single-language data:

the error loss function F _loss,i The method comprises the following steps:

wherein F is _loss,i Error loss value for the ith language specific output layer, p _model,i (x _L ) Spectral feature x for the L-th language _L Corresponding output at the L-th language specific output layer, q _label,L For spectral feature x _L A corresponding acoustic state tag; the error loss function value of other output layers is zero;

step 1-4) comparing the error loss value F _loss,i Back-pass, each language specific output layer parameter is updated according to the data of the corresponding single language, calculating the language specific output layer parameter gradient delta phi _i ：

Wherein phi is _i Parameters for the i-th language specific output layer;

the parameters sharing the hidden layer are represented by the returned error loss values F of a plurality of language-specific output layers _loss,i Updating: calculating the gradient delta phi of the shared hidden layer parameters:

wherein phi is a parameter of a shared hidden layer, and L is the number of language types corresponding to a specific language output layer of the multi-language acoustic model;

step 1-5) when F _loss,i >If the threshold value is given, the step 1-2) is carried out;

when F _loss,i <And (5) given a threshold value, obtaining a trained multilingual acoustic model.

3. The method of claim 1, further comprising the step of training a frame-level language classification model, comprising the steps of:

step 2-1), constructing a frame-level language classification model, wherein the frame-level language classification model is a deep neural network;

step 2-2) extracting frame-level spectrum features of the multi-language continuous voice stream of the training set, inputting the frame-level spectrum features into a frame-level language classification model, carrying out long-time statistics on the output vector of the current hidden layer, and calculating a mean value vector, a variance vector and a segment-level language feature vector of the output vector of the current hidden layer;

the mean vector μ is:

the variance vector sigma is:

the segment-level language feature vector h _segment The method comprises the following steps:

h _segment ＝Append(μ,σ) (6)

wherein h is _i For the output vector of the current hidden layer at the moment i, T is the long-term statistical period, mu is the average value vector of the long-term statistics, sigma is the variance vector of the long-term statistics, h _segment The segment-level language feature vector is formed by splicing a mean vector and a variance vector, and the dimension is h _i 2 times the dimension, where application (μ, σ) represents stitching μ and σ to form a high-dimensional vector;

step 2-3) taking the mean vector and the variance vector as the input of the next hidden layer, training according to the frame-level language labels through error calculation and reverse gradient feedback process, and enabling each hidden layer to output segment-level language feature vectors to obtain a trained frame-level language classification model.

4. The method of claim 1, further comprising the step of training a segment-level language classification model, comprising the steps of:

s2-1), constructing a segment level language classification model;

s2-2) extracting frame-level spectrum features of the multi-language continuous voice stream of the training set, inputting the frame-level spectrum features into an implicit layer of a trained frame-level language classification model, and extracting segment-level language feature vectors from the implicit layer of the trained frame-level language classification model;

step S2-3), setting segment level language feature vectors for each segment level language feature vector, inputting the segment level language feature vectors into a segment level language classification model, training and outputting posterior probability distribution of language states corresponding to the segment level language feature vectors, and obtaining a trained segment level language classification model.

5. The method for recognizing speech content according to claim 1, wherein the multi-language continuous speech stream to be recognized is input into a frame-level language classification model to output segment-level language feature vectors; inputting the segment level language feature vector into the segment level language classification model to output posterior probability distribution of the language state; the method specifically comprises the following steps:

extracting to-be-identified frame-level spectrum features from a multi-language continuous voice stream to be identified;

inputting the frequency spectrum characteristics of the frame level to be identified into a trained frame level language classification model according to a specific step length and window length, and outputting a segment level language characteristic vector h _segment ；

The segment level language feature vector h is processed _segment And inputting the trained segment-level language classification model, and outputting posterior probability distribution of the language state corresponding to the segment-level language feature vector.

6. A system based on the multi-lingual continuous voice stream voice content recognition method of claim 1, the system comprising:

the segment level language feature extraction module is used for inputting the multi-language continuous voice stream to be recognized into the frame level language classification model and outputting segment level language feature vectors;

the posterior probability calculation module of the language state inputs the segment-level language feature vector into the segment-level language classification model, and outputs posterior probability distribution of the segment-level language state;

the language state path acquisition module is used for calculating an optimal language state path of the multilingual voice stream based on a Viterbi retrieval algorithm according to posterior probability distribution of the segment-level language states;

the language state interval segmentation module is used for segmenting the multi-language continuous voice stream to be identified according to the optimal language state path to obtain a language state interval; and

and the content recognition module is used for sending the segmented language state interval into a multilingual acoustic model and a corresponding multilingual decoder for decoding to obtain a content recognition result of the multilingual continuous voice stream.

7. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1-5 when executing the computer program.

8. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, causes the processor to perform the method of any of claims 1-5.