CN110634491B - Series connection feature extraction system and method for general voice task in voice signal - Google Patents

Abstract

The invention discloses a serial feature extraction system and method for general speech tasks in speech signals. After the emotional corpus is established, speech preprocessing is performed; the audio files after speech preprocessing are used as input to be connected with a speech characteristic extraction model; the speech characteristic extraction model The basic information and salient features of the speaker can be obtained through the voiceprint recognition layer and fed into the speech recognition layer together to obtain the speaker information and text information. Using personalized speech features and emotional features, emotion recognition can be used to obtain emotion-related features; the speech feature extraction model can also obtain the basic information and salient features of the speaker through the voiceprint recognition layer, and feed into the speech together with the features of speech recognition. In the recognition layer, the speaker information and the low-level descriptors are sent to the emotion recognition layer to output emotional features. The invention has the advantages of large data scale, accurate emotion expression and high integration.

Description

Concatenated feature extraction system and method for general speech tasks in speech signals

technical field

The invention relates to the field of signal processing and extraction, in particular to a feature extraction model for speech tasks.

Background technique

Voice is the most effective, natural and important form of communication for human beings. To realize the communication between humans and machines through voice, machines need to have enough intelligence to recognize human voices. With the development of machine learning, neural network and deep learning theory, the completion of speech recognition-related tasks is gradually improving, which is very helpful for computers to understand the content of speech. At present, speech recognition tasks mainly involve the following three recognition tasks:

1. Voiceprint recognition

Voiceprint recognition, also known as speaker recognition, is a form of biometric recognition. It analyzes the continuous speech signal of the speaker to extract discrete speech features, and automatically confirms the speech by matching the template in the database. By. It pays attention to the speaker itself, not the content of the speech. Due to the differences in pronunciation organs, accents, speaking rhythms, etc. between people, the speaker information can be extracted by analyzing the human voice, so as to achieve the purpose of identifying the person's identity.

2. Voice recognition

Speech recognition is a technology that allows machines to convert speech signals into corresponding text or commands through the process of recognition and understanding. The applications of speech recognition technology include voice dialing, voice navigation, indoor equipment control, voice document retrieval, simple dictation data entry, etc. Combining speech recognition technology with other natural language processing technologies such as machine translation and speech synthesis can create more complex applications.

3. Speech emotion recognition

The traditional human-computer interaction mainly relies on the keyboard and mouse. The computer only passively receives information, and cannot actively communicate with people, and there is no emotional communication between humans and computers. Computers are naturally unable to achieve natural and harmonious human-computer interaction. Emotion recognition can help to realize the communication and communication of emotions between simulated people, and make computers also have the ability to calculate emotions.

However, there are many defects or deficiencies in the practical application or design of the recognition tasks in the above 3. For example, voiceprint recognition, speech recognition, and emotion recognition task models are not universal, the input form is not uniform, there is no universal solution, the integration accuracy is not high, the recognition accuracy of a single emotion recognition task is not high, and so on.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a system and method for extracting series features for general speech tasks in speech signals with large data scale, accurate emotional expression and high integration.

In order to achieve the above purpose, the following technical solutions are adopted: the system described in the present invention mainly includes an emotion corpus, a speech preprocessing model, and a speech feature extraction model; the emotion corpus is a collection of real emotions constructed based on the form of natural language. Based on the greedy algorithm, statistical methods are introduced; after the emotional corpus is established, speech preprocessing is performed;

The speech preprocessing is divided into two stages, the spectrogram and the combination of low-level description and high-level statistical functions, and the corresponding basic features are extracted from them; the audio files after speech preprocessing are used as input to connect with the speech feature extraction model;

The voice feature extraction model takes the user's audio file as input, and includes a voiceprint recognition sub-module, a speech recognition sub-module, and a voice emotion recognition sub-module; the voice feature extraction model is divided into two working forms. The first layer is voiceprint recognition. The input of the voiceprint recognition layer is the user's audio data. Through this layer, the basic information and salient features of the speaker are obtained, and the basic information and salient features are fed into the speech recognition together with the features of speech recognition. The input of the speech recognition layer is the voiceprint feature and user audio data. Based on the speech recognition layer, the speaker information and text information are obtained. On the basis of extracting keywords, they are sent to the emotion recognition layer together with the low-level descriptors. Use personalized speech features and emotional features to perform emotion recognition, so as to obtain the relevant features of emotion;

Second, it contains a 2-layer structure, which adopts a combination of series and parallel. The first layer is voiceprint recognition. The input of this layer is the user's audio data. Through this layer, the basic information and salient features of the speaker are obtained, and the The features of speech recognition are fed into the second layer of speech recognition. The input of the speech recognition layer is the voiceprint feature and user audio data. Based on the speech recognition layer, the speaker information and text information are obtained. The speaker information is sent to the emotion recognition layer together with the low-level descriptors, and the output of emotion features is simultaneously performed. At this time, the input of emotion recognition model only comes from voiceprint features and speech signals.

Further, the spectrogram is the display image of the Fourier analysis of the speech signal, and the spectrogram is a three-dimensional spectrum, representing the graph of the change of the speech spectrum over time, the abscissa of the spectrogram is time, and the ordinate is frequency, the coordinate point value is the energy of speech data; the acquisition method is as follows: for a segment of speech signal x(t), first divide it into frames and change it to x(m,n), where n is the frame length, and m is the number of frames; Fourier transform, get X(m,n), get the periodogram Y(m,n), Y(m,n)=X(m,n)*X(m,n)', take 10*log10(Y( m,n)), transform m according to the time scale to get M, and n according to the frequency transform scale to get N; the two-dimensional image composed of M,N,10*log10(Y(m,n)) is the language spectrum Figure; the audio file takes 20ms as a frame, and slides with 10ms as the step to generate the spectrogram of each frame; the basic data of the spectrogram of each frame is vertically fused, and the 512-dimension is divided horizontally. On the basis of vertical segmentation, the average value of each column is calculated to obtain the spectral features of 1*512 dimensions.

Further, the emotion corpus contains four emotions: happy, angry, calm and sad; a statistical method is introduced on the basis of the traditional greedy algorithm, and for each type of emotional information statistics, the capacity of each type of data is guaranteed to be between 80% and 120%. In order to ensure the correctness of the centralized processing of the data, the recording files in this database are saved in wav format, the sampling rate of the audio files is 4400Hz, the precision is 16bit, and the recording is in mono; The mean of the sum of the last credibility products is used as the benchmark value, and then the correlation coefficient between each person's labeling result and the benchmark value is calculated separately as an indicator to measure its new credibility. After obtaining the new credibility index, the current labeling result is recalculated, that is, the weighted summation of all labeling results and weights to obtain the final labeling result; the specific formula is as follows.

分别是上一次和本次的可信度，n是标注人员的数量，Result_exp是本次标注人员的结果，Avg_i和

分别是利用上一次和本次可信度计算的计算标注结果。W _exp and

are the reliability of the previous and this time respectively, n is the number of labelers, Result _exp is the result of the labelers this time, Avg _i and

They are the calculation and annotation results of the previous and this credibility calculation respectively.

Further, when capturing the most primitive acoustic features, it is necessary to convert the speech signal into a speech feature vector, that is, combining the low-level descriptor LLD and the high-level statistical function HSF (High level Statistics Functions), the features can be directly calculated using the OpenSmile toolbox toolbox. Obtained; select the following LLD features: Mel Frequency Cepstral Coefficients MFCC1-14, Mel Band Log Power 1-7, Formant 1-3, Pitch Frequency F0, Jitter, Energy Feature, Energy Spectral Feature (alphaRatioUV), Line Spectrum pair frequencies 1-8.

Further, the voiceprint recognition sub-module includes all the personal characteristic factors in the early stage, the input adopts the spectrogram and its delta form, and the adopted model is a Time-Delay Neural Network (TDNN) and an i-vector. combination of models.

Further, the speech recognition sub-model is used to first input the atlas into the convolutional neural network, then through the Gated Recurrent Unit (GRU) network layer, enter the CTC (Connectionist TemporalClassification), obtain the phoneme sequence, and synthesize personalized voiceprint features, The similarity between 0 and 1 is obtained by comparing the similarity with the training sample.

Further, the speech emotion recognition sub-module, on the basis of combining the speaker recognition model or the speech recognition model, the speech emotion recognition sub-model utilizes the neural network to construct the hand-made HSF and the convolutional neural network to learn the multi-channel of speech emotion features. Joint representation, which focuses on specific parts and global information of records containing strong articulatory information.

The present invention further provides a tandem feature extraction method for general speech tasks in a speech signal, the extraction method comprising three parts: voiceprint recognition feature extraction, speech recognition feature extraction, and speech emotion recognition feature extraction;

1) The method of extracting voiceprint recognition features is:

S1-1, the input adopts the spectrogram and its delta form, and the adopted model is the combination of Time-Delay Neural Network (TDNN) and i-vector model;

S1-2, send the features of TDNN into the attention module to improve the computing power of the model;

S1-3, the attention module outputs important voiceprint features, and the voiceprint recognition model outputs part of the voiceprint features, the features are 1*512 dimensions, and the comparison results with everyone in the corpus are obtained through Softmax;

2) The method of extracting speech recognition features is:

The speech recognition sub-model first inputs the map into the convolutional neural network, and then passes through the Gated RecurrentUnit (GRU) network layer, enters the CTC (Connectionist Temporal Classification), obtains the phoneme sequence, integrates the personalized voiceprint features, and compares the similarity with the training samples. Get a similarity value between 0 and 1;

3) The method of extracting speech emotion recognition features is:

S3-1, the input information is spectrogram and LLD feature, and the LLD feature is calculated by a specific HSF to obtain a 642-dimensional HSF feature;

S3-2, send the spectrogram into the CNN model, go through 3 layers of convolution, 3 layers of pooling and 1 layer of full connection, and obtain 1*1024-dimensional feature output;

S3-3, project the above two outputs into the same feature space to obtain 1024+642=1666-dimensional features, and then feed them into the one-way 3-layer GRU model to extract the features of speech emotion, and finally obtain 1024-dimensional emotion feature.

Compared with the prior art, the present invention has the following advantages:

1. Use the common features extracted by the original speech signal processing module and the common model of the task to design a multi-channel network model. Each task can independently select several channels and cooperate to complete feature extraction, so as to realize one input and go through multiple channels. Solve multiple tasks.

2. Using one-time input, simultaneously, hierarchically and objectively display the results of voiceprint recognition, speech recognition and emotion recognition.

3. Improve the accuracy of voiceprint recognition, speech recognition and emotion recognition.

4. Different schemes can be freely selected in each sub-model, or the default combination method can be adopted.

5. The newly built speech emotion corpus can provide a stable and reliable data source for voiceprint recognition, speech recognition and emotion recognition tasks.

6. Improve the integration of voiceprint recognition, speech recognition and emotion recognition tasks.

7. Design an emotional voice database with large data scale, accurate emotional expression and high quality of recorded voices; it has the characteristics of multi-age and high-level speech emotions, and the involved voices are dense and highly recognizable, meeting the needs of rich and diverse emotions.

Description of drawings

FIG. 1 is a flow chart of the first working form of the speech feature extraction model in the present invention.

Fig. 2 is a flow chart of the second working form of the speech feature extraction model in the present invention.

FIG. 3 is a structural diagram of a voiceprint recognition sub-model in the present invention.

FIG. 4 is a structural diagram of a TDNN model in the present invention.

FIG. 5 is a structural diagram of a speech recognition sub-model in the present invention.

FIG. 6 is a structural diagram of the GRU model in the present invention.

FIG. 7 is a structural diagram of a speech emotion recognition sub-model in the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings:

The system of the invention mainly includes an emotion corpus, a speech preprocessing model, and a speech feature extraction model; the emotion corpus is a collection of real emotions constructed based on the form of natural language, and statistical methods are introduced on the basis of the traditional greedy algorithm when the database is established; the emotion corpus After the establishment, the speech preprocessing is performed;

The emotional corpus is characterized by a large scale of data, accurate emotional expression, and high quality of recorded voices. It has the characteristics of multi-age and high-level speech and emotion, the involved voice is strong, and the recognition degree is high, which meets the needs of rich and diverse emotions to a certain extent. Emotional speech is divided into natural speech, induced speech and performance speech according to different collection methods. This database is a collection of real emotions constructed based on natural speech, including four emotions: happiness, anger, calmness and sadness. The text that constitutes the speech database needs to cover as many language units as possible, and at the same time, the scale of the speech database is required not to be too large. Therefore, when establishing the emotional speech database, an improved greedy algorithm is proposed to combine traditional text screening methods with statistical methods. combine.

The traditional greedy algorithm uses the local optimal strategy to generate the global optimal solution. For the screening of the voice database content, the audio data with the best emotional expression is screened from each category each time. This method only considers text screening, The balance of the data set is not considered. On the basis of this method, the present invention introduces a statistical method, and for each type of emotional information statistics, the capacity of each type of data is guaranteed to be within the range of 80%-120%, to avoid the inclination of the data. model error.

The corpus form selects words with rich emotions, which have different comprehension forms in different contexts, and the discourse style meets the needs of rich and diverse emotions to a certain extent. In order to ensure the correctness of the centralized processing of the data, the recording files in this database are saved in wav format, the sampling rate of the audio files is 4400Hz, the precision is 16bit, and the audio files are recorded in mono.

Emotion annotation is performed on data based on multi-view emotion. Since emotion is divided into 4 categories, it is divided into 4 dimensions. Each dimension of each audio needs to be labeled, and its intensity is divided into "none" and "weak". , "Medium" and "Strong" four levels, and the scale table is composed of four levels of "0", "1", "2" and "3". The labeling process is divided into two parts: the training phase and the formal phase. In the training phase, 10 articles are selected for independent labeling. After that, several labelers compare the results and discuss the labeling rules. When the opinions are basically the same, a large-scale formal labeling is performed.

Calculate the mean value of the sum of the product of each dimension's annotation results and the previous credibility of all the annotators to use as the benchmark value, and then calculate the correlation coefficient between each person's annotation results and the benchmark value as an indicator to measure their new credibility. , as the number of tags increases, the credibility index is adjusted immediately. After obtaining a new credibility index, the current annotation result is recalculated, that is, the weighted summation of all the annotation results and weights is obtained to obtain the finalized annotation. result. The specific formula is as follows.

分别是上一次和本次的可信度，n是标注人员的数量，Result_exp是本次标注人员的结果，Avg_i和

分别是利用上一次和本次可信度计算的计算标注结果。W _exp and

are the reliability of the previous and this time respectively, n is the number of labelers, Result _exp is the result of the labelers this time, Avg _i and

They are the calculation and annotation results of the previous and this credibility calculation respectively.

Since the labeled data contains a lot of dimensional information whose label values are all 0, the correlation coefficient cannot be calculated when the label values are all zero. At this time, weight information approximately 0 is selected to replace the correlation coefficient. By adjusting the contribution rate of individual annotations to the integrated results, the calculation of weighted average is used to strengthen the contribution of values close to the mean.

The speech preprocessing is divided into two stages, the spectrogram and the combination of low-level description and high-level statistical functions, and the corresponding basic features are extracted from them; the audio files after speech preprocessing are used as input to connect with the speech feature extraction model;

The spectrogram is the display image of the Fourier analysis of the speech signal. The spectrogram is a three-dimensional spectrum, which represents the graph of the speech spectrum changing with time. The abscissa of the spectrogram is the time, the ordinate is the frequency, and the coordinate is the frequency. The point value is the speech data energy; the acquisition method is as follows: for a segment of speech signal x(t), first divide it into frames and change it into x(m,n), where n is the frame length, and m is the number of frames; perform fast Fourier transform, Get X(m,n), get the periodogram Y(m,n), Y(m,n)=X(m,n)*X(m,n)', take 10*log10(Y(m,n) )), transform m according to the time scale to obtain M, and n according to the frequency transform scale to obtain N; the two-dimensional image composed of M, N, 10*log10(Y(m,n)) is the spectrogram; audio The file takes 20ms as a frame, and slides with 10ms as a step to generate the spectrogram of each frame; the basic data of the spectrogram of each frame is vertically fused, and the 512-dimension is horizontally divided, and it is cut vertically. On the basis of the score, the average value of each column is calculated to obtain the spectral features of 1*512 dimensions.

The method of capturing the most primitive acoustic features requires 1) converting speech signals into speech feature vectors, that is, combining acoustic feature descriptors LLD (low-level acoustic feature descriptors) and high-level statistical functions (HSF). The present invention selects low-level descriptors based on a) their potential to guide emotional physiological changes in sound, b) their proven value in previous studies and their automatic extractability, and c) their theoretical significance. This ensemble is designed to provide a baseline for studying speech features and to eliminate differences caused by varying models, or even different implementations of the same parameters.

Based on this, the present invention selects the following LLD features: Mel frequency cepstral coefficient MFCC 1-14, Mel band logarithmic power 1-7, formant 1-3, fundamental frequency (F0), jitter, energy feature, Energy spectrum feature (alpha RatioUV), line spectrum versus frequency 1-8.

For HSF, different features have different expression forms and need to be designed separately. Based on this, the present invention extracts 14-dimensional MFCC features, Mel band 1-7 logarithmic power, and line spectrum vs. frequency 1-8 features, which are mainly embodied as static 21 HSF functions, such as maximum value, minimum value, arithmetic mean value, slope, and offset, are performed on them, and the complementarity of information is realized through feature fusion, thereby effectively improving the accuracy. At the same time, the center frequencies and their bandwidths of the 1st, 2nd, and 3rd formants are calculated and extracted. For the pitch frequency and jitter of the current frame, record its maximum (small) value, mean, variation range, standard deviation, median, quarterback number (upper, lower) and other 8-dimensional features. Calculate 20th percentile, 50th percentile, 80th percentile, range between 20th and 80th percentile, mean and standard deviation of the slope of the rising/falling speech signal for energy features, and get 8 Statistical Features. For the energy spectrum features, three functions, such as mean value, variation range, and standard deviation, are calculated, and three features are obtained.

The specific LLD and HSF involved are shown in Table 1, which involves a total of 642-dimensional features. These features can be calculated using the OpenSmile toolbox.

Table 1 List of specific LLDs and number of HSFs

Table 2 List of 21 common HSFs

The speech feature extraction model is a personalized model designed for individuals. The user's audio file is used as input, and the goal is to obtain the personalized features of the acoustic signal for a specific user. A multi-task-based Tandem tandem model is designed. This model contains 3 subsections. The model corresponds to three tasks respectively. The structure of these sub-models consists of multiple layers. The structure has two forms, and the selection of the form is determined by the requirements of the task. The voice feature extraction model is divided into two working forms. One, as shown in Figure 1, includes a 3-layer structure. The first layer is voiceprint recognition. The input of the voiceprint recognition layer is the user's audio data, and the speech is obtained through this layer. The basic information and salient features of the speaker are fed into the speech recognition layer together with the features of speech recognition. The input of the speech recognition layer is the voiceprint features and user audio data, and the speaker is obtained based on the speech recognition layer. Information and text information, on the basis of extracting keywords, they are sent to the emotion recognition layer together with the low-level descriptors, and the personalized speech features and emotional features are used for emotion recognition, so as to obtain the relevant features of emotion;

Second, as shown in Figure 2, it includes a 2-layer structure, which adopts a series-parallel combination. The first layer is voiceprint recognition. The input of this layer is the user's audio data. Sexual features, which are fed into the second layer of speech recognition along with the features of speech recognition. The input of the speech recognition layer is the voiceprint feature and user audio data, and the speaker information and text information are obtained based on the speech recognition layer. In addition, in At the same time, the speech recognition layer can send the speaker information together with the low-level descriptors to the emotion recognition layer, and output emotional features simultaneously. At this time, the input of the emotion recognition model only comes from the voiceprint features and speech signals.

The application categories of the above two forms are slightly different, and their focus is based on the generated results. The requirements of the first form focus on emotion recognition, while the requirements of the second form focus on speech recognition and emotion recognition. In the process of feature extraction, one of the structures can be selected according to the focus to complete related tasks.

As shown in Fig. 3, since the present invention is aimed at individual voiceprint recognition, speech recognition and emotion recognition, the voiceprint recognition model is an important link, which includes all the personal characteristic factors in the previous stage, and the input used here is is the spectrogram and its delta form, and the model adopted is the combination of the Time-Delay Neural Network (TDNN) and the i-vector model.

A TDNN is a feedforward neural network, where computations are performed by interconnected layers consisting of multiple clusters. As shown in Figure 4, where j-dimensional vector is input, D1-DN are delay vectors, and wi is connection weight. Assuming that the delay N is 2, the dimension of the input layer is J=10, and the delay N=2 means that two adjacent frame vectors, that is, a total of three frame vectors are spliced together as the input of the input layer. An important idea of time-delay neural networks is weight sharing, that is, connections at the same location have the same weight. That is to say, when the delay is 2, there are three groups of weights in the network, one group is used for the current frame, and the other two groups are used for the delay frames D1 and D2. The positions of different colors have different weights, and the same position shares the weight. The training method uses the traditional BP learning method.

The first few hidden layers are all learned from contexts with shorter strides. Since deep neurons have stronger learning ability, deep hidden layers learn from the context with wider stride. Different from the traditional forward propagation neural network, TDNN can capture the long-term dependence of feature pairs related to time series. It is precisely because of this feature of TDNN that TDNN is selected as the method for extracting speaker representation vectors. Multi-frame training data are spliced together in different network layers for learning, so as to obtain long-term features of 1*512 dimensions between speech frames.

The features of TDNN are sent to the attention module, which is a "selection mechanism" used to allocate limited information processing capabilities, which helps to quickly analyze target data, and cooperate with information screening and weight setting mechanisms to improve the computing power of the model.

For each vector xi in the sequence of input x, the attention weight α _i can be calculated according to Equation 5, where f( _{xi )} is the scoring function.

The output of the attention layer, attention_x, is the sum of the weights of the input sequence. as shown in Equation 6.

The model of this channel learns and outputs important voiceprint features, and the voiceprint recognition model outputs some voiceprint features, which are 1*512 dimensions. Alignment results with everyone in the speech library were obtained by Softmax.

i-vector is a low-dimensional vector, usually a few hundred dimensions. The i-vector can be regarded as a front-end feature, which contains both speaker voiceprint features and channel information, and removes some irrelevant information, such as background noise, channel interference, etc. In the training process of the mean supervector T of the i-vector, each sentence, that is, each recording, is used as a unit, and different recordings of the same speaker are regarded as different people. Given a speech of a speaker, the corresponding Gaussian mean hypervector can be defined as the following formula:

M=m+Tω (7)

Among them, M is the Gaussian mean supervector of the given speech; m is the Gaussian mean supervector of the general background model, which is independent of the specific speaker and channel; T is the global difference space matrix, low rank; ω is the global difference space factor, and its posterior mean is the i-vector vector, which a priori obeys the standard normal distribution. The user's voiceprint information is obtained through i-vector, and the comparison result of the voiceprint is obtained.

By assigning different weights to the training results obtained by the above two models, the decision information for the speaker, that is, the voiceprint ID, can be obtained. Here, in addition to the speaker information, all personal feature information identified by the speaker in the corpus is also obtained. This time, a personalized voiceprint feature was obtained. On this basis, perform the model for the following two speech tasks.

As shown in Figure 5, the speech recognition sub-model is used to first input the atlas into the convolutional neural network, and then go through the Gated Recurrent Unit (GRU) network layer and enter the CTC (Connectionist TemporalClassification) to obtain the phoneme sequence and synthesize personalized sound. The texture features are compared with the training samples to obtain a similarity value between 0 and 1.

Convolutional Neural Network (CNN) is a deep neural network consisting of alternately stacking convolutional layers and pooling layers. The neural unit of the current layer is connected to several feature maps of the previous layer through a set of weights, that is, the convolution kernel, and the feature map of the current layer is obtained by adding the bias. Each neural unit is only connected to the local area of the previous feature map, each neural unit extracts the features of the local area, and all neural units are combined to obtain global features. In order to obtain more comprehensive information from the feature parameters, multiple different convolution kernels are used in the same layer of network to operate to obtain multiple feature maps. The mapping relationship before and after the convolutional layer is as follows.

表示第m个卷积层第j个特征图的输入，

代表卷积核，

表示偏置，*表示卷积操作，M_j表示特征图的集合，f表示激活函数。in,

represents the input of the jth feature map of the mth convolutional layer,

represents the convolution kernel,

represents the bias, * represents the convolution operation, M _j represents the set of feature maps, and f represents the activation function.

The feature map after the convolution operation is down-sampled in the pooling layer. The pooling unit calculates the main information of the local area in the feature map, so redundant information is removed and the operation scale is reduced. CNN consists of 3 layers of convolution layers, 3 layers of pooling layers and 2 layers of fully connected layers, a total of 8 layers. The input picture of the first layer of convolution layer is 310*310*3, of which 310 is the length and width of the picture, 3 represents three channels of RGB. The image is passed through 64 3*3 convolution kernels, and 64 feature maps are generated after the convolution operation with a stride of 1, and then the Relu activation function is used to obtain 64 feature maps after the maximum pooling operation. The second layer volume The input source of the multi-layer layer is the output feature map of the first layer. The calculation process is the same as that of the first layer. The third layer is the same. Next is a fully connected layer. This layer has a total of 1024 neurons. Do Dropout operation to prevent the model from overfitting. The output of this layer is 1*1024 dimensional features.

The convolved features are fed into the Gated Recurrent Unit (GRU) model, which contains two gates: an update gate and a reset gate. The GRU model structure is shown in Figure 6, where z _t and r _t represent the update gate and the reset gate, respectively. The design formula of the GRU model is as follows.

z _t =σ(W _z ·[h _t-1 ,x _t ]) (9)

r _t =σ(W _r ·[h _t-1 ,x _t ]) (10)

z _t and r _t represent the update gate and the reset gate, respectively;

为第t层部分隐藏层输出；

output for the hidden layer of the t-th layer;

h _t is the vector of all hidden layers of the t-th layer.

The speech recognition features are learned from the GRU model, and the output features are 1*1024 dimensions.

Send the previous output features into CTC, which is based on neural network time series classification, and obtain the output sequence y through the input sequence x, such as the distribution p(I/x) of the output sequence, select the one with the highest probability. As the output sequence, as shown in Equation 13.

Obtain the sequence with the highest probability for representing Chinese through CTC, and output it. Accuracy is then measured by comparing the sequence with the text-correct phone sequence for similarity. Based on this, a similarity comparison matrix is constructed, which consists of 23 initials and 24 finals. Because it involves the measurement of pronunciation similarity, the tones of the finals are not considered. The designed similarity comparison matrix A is shown in Table 3. Show.

Table 3 Similarity comparison matrix

The degree of similarity between initials and finals is represented by a number between 0 and 100. The larger the number, the lower the similarity. After identifying the phonemes through the model, go to the confusion matrix to search, and the identified initials and finals correspond to the sample's initials and finals. is 1, and other positions are 0, generating matrix B, phoneme accuracy matrix C=A·B, the sum of the elements in C is the accuracy value of the phoneme, and the average value of the accuracy of all phonemes in a speech is the accuracy of the entire speech. Sexual evaluation value. Through the speech recognition model, the text information features are finally obtained.

The speech emotion recognition sub-module, on the basis of combining the speaker recognition model or the speech recognition model, the speech emotion recognition model uses a neural network to construct a hand-made HSF and a convolutional neural network to learn the multi-channel joint representation of speech emotion features, focusing on For specific parts of records containing strong pronunciation information and global information. As shown in Figure 7.

The specific process is as follows:

The input information is spectrogram and LLD feature. The LLD feature is calculated by a specific HSF to obtain a 642-dimensional HSF feature. The spectrogram is sent to the CNN model, which undergoes 3 layers of convolution, 3 layers of pooling and 1 layer of full connection. Get 1*1024-dimensional feature output, project the above two into the same feature space, get 1024+642=1666-dimensional feature, and then feed it into the one-way 3-layer GRU model to extract the features of speech emotion, and finally get 1024-dimensional emotional features, using this method not only ensures that the dimensions of the hand-crafted features are appropriate, but also obtains the global information of the speech.

The present invention also provides a feature extraction method according to the serial feature extraction system for general speech tasks in the speech signal, and the extraction method includes three parts: voiceprint recognition feature extraction, speech recognition feature extraction, and speech emotion recognition feature extraction;

1) The method of extracting voiceprint recognition features is:

S1-1, the input adopts the spectrogram and its delta form, and the adopted model is the combination of the Time-Delay Neural Network (TDNN) and the i-vector model;

S1-2, send the features of TDNN into the attention module to improve the computing power of the model;

S1-3, the attention module outputs important voiceprint features, and the voiceprint recognition model outputs part of the voiceprint features, the features are 1*512 dimensions, and the comparison results with everyone in the corpus are obtained through Softmax;

2) The method of extracting speech recognition features is:

The speech recognition sub-model first inputs the map into the convolutional neural network, and then passes through the Gated RecurrentUnit (GRU) network layer, enters the CTC (Connectionist Temporal Classification), obtains the phoneme sequence, integrates the personalized voiceprint features, and compares the similarity with the training samples. Get a similarity value between 0 and 1;

3) The method of extracting speech emotion recognition features is:

S3-1, the input information is spectrogram and LLD feature, and the LLD feature is calculated by a specific HSF to obtain a 642-dimensional HSF feature;

S3-2, send the spectrogram into the CNN model, go through 3 layers of convolution, 3 layers of pooling and 1 layer of full connection, and obtain 1*1024-dimensional feature output;

S3-3, project the above two outputs into the same feature space to obtain 1024+642=1666-dimensional features, and then feed them into the one-way 3-layer GRU model to extract the features of speech emotion, and finally obtain 1024-dimensional emotion feature.

The above-mentioned embodiments are only to describe the preferred embodiments of the present invention, and do not limit the scope of the present invention. On the premise of not departing from the design spirit of the present invention, those of ordinary skill in the art can Such deformations and improvements shall fall within the protection scope determined by the claims of the present invention.

Claims (8)

1. a tandem feature extraction system for general speech tasks in a speech signal, is characterized in that: the system mainly comprises an emotional corpus, a speech preprocessing model, a speech feature extraction model; the emotional corpus is based on the form of natural language to build real emotions Statistical methods are introduced based on the traditional greedy algorithm when establishing the database; speech preprocessing is performed after the emotional corpus is established; The speech preprocessing is divided into two stages, the spectrogram and the combination of low-level description and high-level statistical functions, and the corresponding basic features are extracted from them; the audio files after speech preprocessing are used as input to connect with the speech feature extraction model; The voice feature extraction model takes the user's audio file as input, and includes a voiceprint recognition sub-module, a speech recognition sub-module, and a voice emotion recognition sub-module; the voice feature extraction model is divided into two working forms. The first layer is voiceprint recognition. The input of the voiceprint recognition layer is the user's audio data. Through this layer, the basic information and salient features of the speaker are obtained, and the basic information and salient features are fed into the speech recognition together with the features of speech recognition. The input of the speech recognition layer is the voiceprint feature and user audio data. Based on the speech recognition layer, the speaker information and text information are obtained. On the basis of extracting keywords, they are sent to the emotion recognition layer together with the low-level descriptors. Use personalized speech features and emotional features to perform emotion recognition, so as to obtain the relevant features of emotion; Second, it contains a 2-layer structure, which adopts a combination of series and parallel. The first layer is voiceprint recognition. The input of this layer is the user's audio data. Through this layer, the basic information and salient features of the speaker are obtained, and the The features of speech recognition are fed into the second layer of speech recognition. The input of the speech recognition layer is the voiceprint feature and user audio data. Based on the speech recognition layer, the speaker information and text information are obtained. The speaker information is sent to the emotion recognition layer together with the low-level descriptors, and the output of emotion features is simultaneously performed. At this time, the input of emotion recognition model only comes from voiceprint features and speech signals. 2. in the speech signal according to claim 1, for the serial feature extraction system of general speech task, it is characterized in that: described spectrogram is the display image of the Fourier analysis of speech signal, and spectrogram is a kind of three-dimensional Spectrum, a graph representing the change of speech spectrum over time. The abscissa of the spectrogram is time, the ordinate is frequency, and the coordinate point value is the energy of speech data; the acquisition method is as follows: For a segment of speech signal x(t), first divide it into frames, Change to x(m,n), n is the frame length, m is the number of frames; perform fast Fourier transform to get X(m,n), get the periodogram Y(m,n), Y(m,n) =X(m,n)*X(m,n)', take 10*log10(Y(m,n)), transform m according to the time scale to get M, and n according to the frequency transform scale to get N; M, The two-dimensional image composed of N,10*log10(Y(m,n)) is the spectrogram; the audio file takes 20ms as a frame and slides with 10ms as the step to generate the spectrogram of each frame respectively; The basic data of the spectrogram of each frame is longitudinally fused, and the 512-dimension is used as the horizontal division. On the basis of the longitudinal division, the average value of each column is calculated to obtain the 1*512-dimension spectrogram feature. 3. the tandem feature extraction system for general speech tasks in the speech signal according to claim 1, is characterized in that: described emotion corpus records four kinds of emotions altogether: happy, angry, calm and sad; On the basis of the traditional greedy algorithm, statistical methods are introduced to count each type of emotional information; in order to ensure the correctness of the data set processing, the recording files in this database are saved in wav format, the sampling rate of the audio files is 4400Hz, and the precision is 16bit. Mono recording; Calculate the mean value of the sum of the product of each dimension's annotation results and the previous credibility of all the annotators to use as the benchmark value, and then calculate the correlation coefficient between each person's annotation results and the benchmark value as an indicator to measure their new credibility. , as the number of tags increases, the credibility index is adjusted immediately. After obtaining the new credibility index, the current annotation result is recalculated, that is, the weighted summation of all the annotation results and weights is obtained to obtain the finalized annotation. The result; the specific formula is as follows:

W _exp and

are the reliability of the previous and this time respectively, n is the number of labelers, Result _exp is the result of the labelers this time, Avg _i and

They are the calculation and annotation results of the previous and this credibility calculation respectively. 4. The tandem feature extraction system for general speech tasks in the speech signal according to claim 1, characterized in that: when capturing the most primitive acoustic features, the speech signal needs to be converted into a speech feature vector, that is, combined with low-level descriptors LLD and advanced statistical function HSF, features can be directly calculated using the OpenSmile toolbox; select the following LLD features: Mel Frequency Cepstral Coefficients MFCC 1-14, Mel Band Log Power 1-7, Formants 1-3 , fundamental frequency F0, jitter, energy characteristics, energy spectrum characteristics, line spectrum to frequency 1-8. 5. in the voice signal according to claim 1, it is characterized in that: the voiceprint recognition submodule comprises all personal characteristic factors in the early stage, and the input adopts spectrogram and its delta Form, the model adopted is the combination of time-delay neural network TDNN and i-vector model. 6. the tandem feature extraction system for general speech tasks in the speech signal according to claim 1, is characterized in that: described speech recognition sub-model is used to input spectrogram into convolutional neural network at first, then through GatedRecurrent Unit In the network layer, enter the CTC to obtain the phoneme sequence, synthesize the personalized voiceprint features, and compare the similarity with the training sample to obtain a similarity value between 0 and 1. 7. The tandem feature extraction system for general speech tasks in the speech signal according to claim 1, is characterized in that: described speech emotion recognition submodule, on the basis of combining speaker recognition model or speech recognition model, speech emotion The recognition submodel utilizes neural networks to build HSF and convolutional neural networks to learn multi-channel joint representations of speech emotion features, focusing on specific parts and global information that contain strong pronunciation information records. 8. a feature extraction method based on the tandem feature extraction system for general speech tasks in the described voice signal of claim 1, is characterized in that: described extraction method comprises voiceprint recognition feature extraction, speech recognition feature extraction, speech emotion recognition Three parts of feature extraction; 1) The method of extracting voiceprint recognition features is: S1-1, the input adopts the spectrogram and its delta form, and the model adopted is the combination of the time delay neural network TDNN and the i-vector model; S1-2, send the features of TDNN into the attention module to improve the computing power of the model; S1-3, the attention module outputs important voiceprint features, and the voiceprint recognition model outputs part of the voiceprint features, the features are 1*512 dimensions, and the comparison results with everyone in the corpus are obtained through Softmax; 2) The method of extracting speech recognition features is: The speech recognition sub-model first inputs the spectrogram into the convolutional neural network, and then passes through the GRU network layer, and then enters the CTC to obtain the phoneme sequence, synthesizes the personalized voiceprint features, and compares the similarity with the training sample to obtain a similarity between 0 and 1. degree value; 3) The method of extracting speech emotion recognition features is: S3-1, the input information is spectrogram and LLD feature, and the LLD feature is calculated by a specific HSF to obtain a 642-dimensional HSF feature; S3-2, send the spectrogram into the CNN model, go through 3 layers of convolution, 3 layers of pooling and 1 layer of full connection, and obtain 1*1024-dimensional feature output; S3-3, project the above two outputs into the same feature space to obtain 1024+642=1666-dimensional features, and then feed them into the one-way 3-layer GRU model to extract the features of speech emotion, and finally obtain 1024-dimensional emotion feature.

Images