CN114255736B - Rhythm annotation method and system - Google Patents

Abstract

The embodiment of the invention provides a rhythm marking method. The method comprises the following steps: extracting word-level prosody expressions from training voice data, and reconstructing the voice by using the word-level prosody expressions to obtain reconstructed voice with the word-level prosody expressions; performing first clustering on the reconstructed voice according to the text information of the word; and carrying out unsupervised second clustering on the results of the first clustering according to the prosody to obtain unsupervised prosody labels of each word. The embodiment of the invention also provides a rhythm marking system. The embodiment of the invention selects modeling and control of prosody at word level, and words contain rich semantic information, so that the words can be well associated with context. In addition, the method can mark the rhythm with diversity, has interpretability, good control capability and high expandability, and achieves more humanized rhythm marking. The speech synthesized by the TTS has richer rhythm, and the interactive experience of the user is improved.

Description

Rhythm annotation method and system

Technical Field

The present invention relates to the field of intelligent speech, and in particular to a prosody annotation method and system.

Background Art

In order to make intelligent voice interaction more humane, rhythm is usually added to the audio used to communicate with users. By adjusting the rhythm, the conversation can be made more rhythmic and the emotion of the language can be expressed.

Usually, we can add the modeling of rhythm space to the basic module of TTS (Text To Speech). Work related to rhythm control uses VAE (Variational Auto-Encoder) to sample rhythm from a latent space, or uses additional parameters to control certain quantities related to rhythm features such as fundamental frequency and energy. In rhythm modeling, most of them aim to improve naturalness and diversity, and most of them use explicit rhythm features. Thus, we can get speech with rhythm.

In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

Explicit prosodic features cannot fully represent the prosodic space and are cumbersome to control; implicit prosodic space is even more difficult to control because it is a continuous space and lacks interpretability. In addition, most related work is done at the phoneme level, which is convenient but does not conform to the principles of natural human speech, making the generated prosodic speech less humane.

Summary of the invention

In order to at least solve the problem that the rhythmic speech generated in the prior art is not humane enough.

In a first aspect, an embodiment of the present invention provides a prosody tagging method, comprising:

Extracting word-level prosodic expressions from training speech data, and reconstructing the speech using the word-level prosodic expressions to obtain reconstructed speech with word-level prosodic expressions;

Performing a first clustering on the reconstructed speech according to text information of words;

An unsupervised second clustering is performed on the results of the first clustering according to the rhythm to obtain an unsupervised rhythm tagging of each word.

In a second aspect, an embodiment of the present invention provides a text-to-speech method based on prosody annotation, comprising:

Inputting the text into a text-to-speech model, performing a first clustering based on the text information of the words and a second clustering based on the rhythm on the phoneme sequence determined by the text-to-speech model, and obtaining an unsupervised rhythmic annotation of each word in the text;

The unsupervised prosodic annotation of each word is at least used to perform prosodic synthesis on the prime sequence to obtain speech with prosody.

In a third aspect, an embodiment of the present invention provides a prosody annotation system, including:

A speech reconstruction program module is used to extract word-level prosodic expressions from training speech data, and reconstruct the speech using the word-level prosodic expressions to obtain reconstructed speech with word-level prosodic expressions;

A clustering program module, used for performing a first clustering on the reconstructed speech according to the text information of the words;

The rhythm tagging program module is used to perform unsupervised second clustering on the results of the first clustering according to rhythm to obtain unsupervised rhythm tagging of each word.

In a fourth aspect, an embodiment of the present invention provides a text-to-speech system based on prosody annotation, comprising:

An unsupervised prosodic tagging program module is used to input text into a text-to-speech model, perform a first clustering based on word text information and a second clustering based on prosody on a phoneme sequence determined by the text-to-speech model, and obtain an unsupervised prosodic tagging of each word in the text;

The speech synthesis program module is used to perform rhythm synthesis on the element sequence by at least using the unsupervised rhythm annotation of each word to obtain speech with rhythm.

In a fifth aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the rhythm tagging method of any embodiment of the present invention.

In a sixth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the prosody tagging method of any embodiment of the present invention are implemented.

The beneficial effects of the embodiments of the present invention are: the prosody modeling and control is performed at the word level, and the words contain rich semantic information, so they can be well associated with the context. In addition, a variety of prosody can be annotated, and the prosody is interpretable, well controlled, and highly scalable, so as to achieve more humanized prosody annotation. The TTS synthesized speech has richer prosody, and the user's interactive experience is improved.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a prosody marking method provided by an embodiment of the present invention;

FIG2 is a diagram of the architecture of a rhythm extraction and annotation system of a rhythm annotation method provided by an embodiment of the present invention;

FIG3 is a flow chart of a text-to-speech method based on prosody annotation provided by an embodiment of the present invention;

4 is a structural diagram of a prosody control model in the training and reasoning stages of a text-to-speech method based on prosody annotation provided by an embodiment of the present invention;

FIG5 is a schematic diagram of a log-likelihood curve of a leaf of a text-to-speech method based on prosody annotation provided by an embodiment of the present invention;

FIG6 is a schematic diagram of a subjective evaluation of the naturalness of a text-to-speech method based on prosody annotation provided by an embodiment of the present invention;

7 is a schematic diagram of synthesized speech with artificially specified prosody in a text-to-speech method based on prosody annotation provided by an embodiment of the present invention;

FIG8 is a schematic diagram of Mel-cepstrum distortion of GT labels and control labels of a text-to-speech method based on prosody annotation provided by an embodiment of the present invention;

FIG9 is a schematic diagram of subjective evaluation of controllability of a text-to-speech method based on prosody annotation provided by an embodiment of the present invention;

FIG10 is a schematic diagram of the structure of a prosody annotation system provided by an embodiment of the present invention;

11 is a schematic diagram of the structure of a text-to-speech system based on prosody annotation provided by an embodiment of the present invention;

FIG. 12 is a schematic diagram of the structure of an embodiment of an electronic device for prosody annotation and text-to-speech conversion based on prosody annotation provided by an embodiment of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

FIG. 1 is a flowchart of a prosody tagging method provided by an embodiment of the present invention, comprising the following steps:

S11: extracting word-level prosodic expressions from training speech data, and reconstructing the speech using the word-level prosodic expressions to obtain reconstructed speech with word-level prosodic expressions;

S12: performing a first clustering on the reconstructed speech according to text information of words;

S13: performing unsupervised second clustering on the results of the first clustering according to rhythm, to obtain unsupervised rhythm labeling of each word.

In this embodiment, it is found that words are the basic unit of prosody, because a word, in different words, has different prosody. This is used as the basic unit of prosody for processing. In short, this method is divided into the following stages, as shown in Figure 2. Word-level prosody embedding extraction, two-stage word-level prosody annotation, after obtaining word-level prosody annotation, TTS training with prosody annotation can be performed.

For step S11, as shown in FIG2(a), a word-level prosodic expression is extracted from the mel-spectrogram of the training data, and this prosodic expression is embedded in the TTS system for training.

Specifically, it includes: pre-building a text-to-speech model and a prosody extractor based on FastSpeech2; inputting the training speech data into the text-to-speech model, and determining the phoneme sequence of the training speech data using the encoder of the text-to-speech model;

The prosody extractor extracts the word-level prosodic expression of each word from the Mel-spectrogram of the training speech data, and reconstructs the phoneme sequence using the word-level prosodic expression to obtain a reconstructed speech with the word-level prosodic expression.

In this embodiment, in order to obtain word-level prosodic embedding, a TTS (Text To Speech) model based on FastSpeech2 is first constructed, and then a prosodic extractor is constructed. As shown in (a) of Figure 2, the prosodic extractor generates a hidden vector (called prosodic embedding e) for each word from the corresponding Mel spectrum segment. The generated prosodic embedding is then aligned with the phoneme sequence and connected to the encoder output. Therefore, the extractor is optimized to extract useful prosodic information expressions to better reconstruct the output speech, including the prosodic information and speech content of the word.

For step S12, there are two stages of prosody tagging. The intuitive idea is that words with very different phonetic contents, such as the long word "congratulations" and the short word "cat", are pronounced completely differently, so the same prosody tag should not be used. Therefore, in this method, a two-stage prosody tagging strategy is designed. First, a decision tree is used to classify words into different types according to their phonetic content, and then GMM (Gaussian Mixed Model) is used to cluster the prosody in each type of words.

The first clustering of the reconstructed speech according to the text information of the words includes: performing a first clustering of the reconstructed speech according to the text information of the words based on a decision tree. In this embodiment, the first clustering is a first stage decision tree clustering.

Inspired by the HMM (HiddenMarkov Model) state binding technology in ASR (Automatic Speech Recognition), a binary decision tree was constructed for word clustering, which contains a set of questions Q about its speech content, where all words in the root are clustered into l leaf nodes. For this purpose, 39 questions were designed based on preset professional knowledge, such as "Is the phoneme of the word greater than 4?", "Is the first phoneme of this word 'm' or 'n'?", and "Does this word end with a closed syllable?".

Each node in the decision tree contains a set of words whose prosodic embeddings can be modeled with a Gaussian distribution and the log-likelihood can be expressed as:

i is the node index, ε ⁽ⁱ⁾ is the set of all prosodic embeddings corresponding to the words in node i, and each non-leaf node i is associated with a question q that partitions the words in the node into its left or right children, resulting in an increase in the log-likelihood of the prosodic embeddings:

Δ _g LL ⁽ⁱ⁾ =LL ^{(i's left child under q)} +LL ^{(i's right child under q)} -LL ⁽ⁱ⁾

The initial tree contains only the root node, which is also a leaf node. Then, recursively perform the following steps: find the problem that maximizes the log-likelihood increase of all leaf nodes, and select a leaf node j whose increase is the largest among all leaf nodes, that is:

The selected nodes are segmented with the corresponding questions. This process continues until the increase in logarithmic likelihood is less than the threshold. Thus, the topological structure of the decision tree is obtained. In this method, as shown in (b) of Figure 2, the number of leaves l is 10, and its index is represented by letters from a to j.

In layman's terms, the above steps are to first set a set of questions, all of which are related to the text information of the word, such as "whether it starts with a consonant", "whether it contains the phoneme IY1", "whether the phonemes of the word are greater than 4", etc. A decision tree is used to iteratively cluster the data into 10 categories (leaf nodes) in the first cluster based on the criterion of maximizing the total likelihood of the data (the number of corresponding clusters will change as the question set is modified, and the number of clusters is not limited here).

For step S13, the second stage can use Gaussian mixture clustering. The word-level prosodic embedding extracted by the neural network contains the prosodic information and speech content of the word. However, the decision tree clusters words into l leaf nodes based only on the question of speech content, so it is assumed that the prosodic embeddings of words in leaf nodes are different only in prosody and similar in speech content. Therefore, the clustering within the leaf node is controlled by prosody, not speech content.

Then, GMM-based clustering is performed on the rhythm embedding in each leaf node i:

where k is the Gaussian component index and m is the number of components. The prosody of each word is labeled with the index of the Gaussian component that maximizes the posterior probability of its prosodic embedding e:

In the above steps represents the mean vector of the kth Gaussian component in leaf node i, represents the covariance matrix of the kth Gaussian component in this node, Represents the weight of the kth Gaussian component in this node. Specifically, m can be set to 5, so the Gaussian component ID ranges from 0 to 4. Therefore, all words in the training set are labeled with m*l=5*10=50 prosody tags, which is the prosody annotation of each word, which is a combination of 10 leaf IDs and 5 Gaussian component IDs. As shown in Figure 2(b), the prosody tags are from a0 to j4. The above prosody extraction and labeling are completely unsupervised, where only audio information is used. In addition, the labeling system is driven by both data and knowledge.

From this implementation, it can be seen that the prosody modeling and control is done at the word level, and words contain rich semantic information, so they can be well associated with the context. In addition, a variety of prosody can be annotated, and it has interpretability, good controllability and high scalability, achieving more humanized prosody annotation.

FIG3 is a flowchart of a text-to-speech method based on prosody annotation provided by an embodiment of the present invention, comprising the following steps:

S21: inputting the text into a text-to-speech model, performing a first clustering based on the text information of the words and a second clustering based on the rhythm on the phoneme sequence determined by the text-to-speech model, and obtaining an unsupervised rhythmic annotation of each word in the text;

S22: Performing rhythm synthesis on the prime sequence using at least the unsupervised rhythm annotation of each word to obtain rhythmic speech.

In this embodiment, since unsupervised prosodic annotations are obtained for each word, they can be applied to text-to-speech tasks to perform prosodic control of speech with prosodic annotations.

For step S21, a TTS model is trained with derived word-level prosodic tags, as shown in Figure 4. In the training phase, the TTS model guides the determined phoneme sequence into the prosodic tag predictor, and in the inference phase, the prosodic tag can be predicted from the input text by the prosodic predictor. That is, the prosodic tagging part described above, and the unsupervised prosodic tagging of each word is obtained.

For step S22, the unsupervised prosodic annotation of each word is used to perform prosodic synthesis on the element sequence to obtain a speech with prosody. As an implementation method, considering a more humanistic design, a user is provided with an inputtable manually specified prosodic annotation to perform prosodic synthesis on the element sequence to obtain a speech with prosody that better meets the user's expectations.

Specifically, as shown in Figure 4, (a) represents the training process in this step, and (b) represents the flowchart in the synthesis stage after the training is completed. In the synthesis stage, the model can be allowed to automatically predict the prosody annotation based on the context, or the annotation can be manually specified for any word and input into the TTS model to synthesize the desired prosody. The prosody predictor predicts the prosody marker of each word based on the phoneme hidden state corresponding to each word (i.e., the encoder output sequence h). The prosody predictor contains a bi-GRU (Bi-directional Gated Recurrent Unit), which converts the phoneme hidden state into a vector for each word, two convolution blocks, and a softmax layer. The convolution block here consists of a 1D convolution layer, a ReLU activation layer, a normalization layer, and a random dropout layer. The predictor is optimized by the cross entropy loss L _PP and the prosody label. Therefore, the overall loss of model training is defined as:

where _LFastSpeech2 is the text-to-speech model loss of FastSpeech2 and α is the relative weight between these two terms.

It can be seen from this implementation that the prosody annotation based on the definite diversity, interpretability and good controllability is applied to TTS, so that the speech synthesized by TTS has richer prosody and improves the user's interactive experience.

The method is experimentally demonstrated using LJSpeech, a single-speaker dataset containing 24 hours of experimental recordings. 242 utterances are set aside as a test set. All sounds are downsampled to 16kHz. Feature extraction is performed using 800-point window length, 200-point frame shift, 1024 FFT points, and 320 mel-bins. Phoneme alignment is obtained using a HMM-GMM ASR model trained on librispeech (a dataset), where GMM (Gaussian mixture model). The vocoder used in this work is MelGAN. The coefficient α is set to 1.0. The prosodic embedding is 128-dimensional.

To determine the performance of the decision tree in prosodic labeling, the method shows the average prosodic embedding number curve of each leaf node and the overall prosodic embedding log-likelihood curve of all leaf nodes ∑ _{i∈leaf nodes} LL ⁽ⁱ⁾ as the decision tree grows in Figure 5. As the number of leaf nodes increases, the average prosodic embedding number of each leaf node decreases, while the overall log-likelihood of prosodic embedding increases. Considering performance and complexity, the tree growth is stopped when the number of leaves reaches 10.

To determine the naturalness of the predicted prosody, a TTS model with a prosody predictor is trained using the derived word-level prosody markers. In the inference phase, word-level prosody can be predicted by the prosody predictor from the input text or can be manually specified. In this section, a test set is synthesized where the prosody is predicted and sampled. Then, the naturalness is evaluated using the MUSHRA test, where 30 listeners are asked to rate each utterance between 0 and 100. The proposed model is compared with two baselines: the typical FastSpeech2 model original Raw_FSP and a TTS model where the phoneme-level prosody is modeled using a mixture density network PLP_MDN. In addition, the real mel-spectrograms of the recordings are reconstructed by a vocoder and then provided as GT in the listening test. The results are shown in Figure 6. It can be observed that the proposed system outperforms the other two models in terms of naturalness due to the word-level prosody modeling. However, the WLP samples are still slightly worse than the GT.

Among them, GT refers to the real recording (ground truth). In the experiment, to ensure that the listener is not affected by the sound quality, the GT sample is actually the audio synthesized by MelGAN after extracting the mel spectrum from the real recording. Raw_FSP refers to the audio synthesized by the original FastSpeech2 model without prosody modeling, and PLP_MDN (phone-level prosody mixture density network) refers to the audio synthesized by using the phoneme-level mixture density network as the model for prosody modeling. WLP_predict (word-level prosody, predicted) is the audio synthesized by completely predicting the prosody through the prosody predictor in this method.

In order to evaluate the word-level prosody controllability of the TTS model of the proposed method, the ground truth word prosody is first labeled for the test set using the proposed prosody labeling system. Then, the test set is synthesized 5 times, where the prosody labels of the words in leaf d are manually assigned as d0 to d4, while the prosody labels of other words are predicted and sampled. Figure 7 shows an example, where the word "responsibility" between the yellow dotted lines is manually controlled by d0 to d4, respectively. It can be observed that the 5 prosody of the word are all different, showing the controllability of the prosody labeling.

In addition, it is necessary to confirm that the same prosodic label leads to similar prosody. Therefore, this method evaluates the prosodic similarity between the recording and the synthesized speech with different assigned prosodic labels for all words in the leaf d in the test set. In theory, when the assigned prosodic label is equal to the reference prosodic label, the prosody of the word in the synthesized speech should be most similar to the recording.

This method evaluates the prosodic similarity from both objective and subjective aspects. First, the average Mel inverted distortion (MCD) of all words with a baseline prosodic label dt is calculated, where t ranges from 0 to 4 for recordings and synthesized speech with a specific prosodic label. The results are shown in Figure 8. It can be found that all diagonal values are the lowest among their column values, which indicates that in the synthesized speech, the same prosodic label will produce similar prosody, and the label used for control can also better show the prosodic characteristics of this label in the training set.

At the same time, the prosody similarity was evaluated with a subjective listening test in which 30 listeners provided recorded and synthesized speech with different prosody labels for each group and were asked to select the prosodic word of the corresponding synthesized speech that was most similar to the recording. The proportion of selection was described in the form of a confusion matrix in Figure 9. Similar to the objective evaluation results, the proportion of synthesized speech with the same prosody label as the benchmark, i.e., the diagonal value was the highest in its column, further verifying the controllability of the prosody label.

In summary, this paper proposes a new unsupervised word-level prosodic tagging method in two stages, firstly classifying words into different types according to their phonetic content using decision trees, and then clustering the prosody using GMMs in each type of words. In addition, a TTS system with derived word-level prosodic tags is trained for controllable speech synthesis, where the prosody can be predicted from the input text or manually specified. Experiments on LJSpeech show that the proposed model achieves better naturalness than the typical FastSpeech2 model with predicted prosody. In addition, subjective and objective evaluations of the controllability of prosody show that prosody can be effectively controlled by specifying word-level prosodic tags.

FIG. 10 is a schematic diagram showing the structure of a prosody annotation system provided by an embodiment of the present invention. The system can execute the prosody annotation method described in any of the above embodiments and is configured in a terminal.

The prosody annotation system 10 provided in this embodiment includes: a speech reconstruction program module 11 , a clustering program module 12 and a prosody annotation program module 13 .

Among them, the speech reconstruction program module 11 is used to extract word-level prosodic expressions from the training speech data, and reconstruct the speech using the word-level prosodic expressions to obtain reconstructed speech with word-level prosodic expressions; the clustering program module 12 is used to perform a first clustering of the reconstructed speech according to the text information of the words; the prosodic annotation program module 13 is used to perform an unsupervised second clustering of the results of the first clustering according to the prosody to obtain unsupervised prosodic annotations for each word.

Furthermore, the speech reconstruction program module is used to:

Pre-built FastSpeech2-based text-to-speech model and prosody extractor;

Inputting the training speech data into the text-to-speech model, and determining a phoneme sequence of the training speech data using an encoder of the text-to-speech model;

The prosody extractor extracts the word-level prosodic expression of each word from the Mel-spectrogram of the training speech data, and reconstructs the phoneme sequence using the word-level prosodic expression to obtain a reconstructed speech with the word-level prosodic expression.

The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the prosody tagging method in any of the above method embodiments;

As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows:

Extracting word-level prosodic expressions from training speech data, and reconstructing the speech using the word-level prosodic expressions to obtain reconstructed speech with word-level prosodic expressions;

Performing a first clustering on the reconstructed speech according to text information of words;

An unsupervised second clustering is performed on the results of the first clustering according to the rhythm to obtain an unsupervised rhythm tagging of each word.

FIG11 is a schematic diagram of the structure of a text-to-speech system based on prosody annotation provided in one embodiment of the present invention. The system can execute the text-to-speech method based on prosody annotation described in any of the above embodiments and be configured in a terminal.

The prosody tagging system 20 provided in this embodiment includes: an unsupervised prosody tagging program module 21 and a speech synthesis program module 22 .

Among them, the unsupervised prosodic annotation program module 21 is used to input the text into the text-to-speech model, perform a first clustering based on the text information of the word and a second clustering based on the prosody on the phoneme sequence determined by the text-to-speech model, and obtain the unsupervised prosodic annotation of each word in the text; the speech synthesis program module 22 is used to perform prosodic synthesis on the phoneme sequence using at least the unsupervised prosodic annotation of each word to obtain speech with rhythm.

Furthermore, the speech synthesis program module is also used for:

The unsupervised prosodic annotation of each word and the input manually specified prosodic annotation are used to perform prosodic synthesis on the element sequence to obtain speech with prosody.

The embodiment of the present invention further provides a non-volatile computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the text-to-speech method based on prosody annotation in any of the above method embodiments;

As an implementation mode, the non-volatile computer storage medium of the present invention stores computer executable instructions, and the computer executable instructions are configured as follows:

Inputting the text into a text-to-speech model, performing a first clustering based on the text information of the words and a second clustering based on the rhythm on the phoneme sequence determined by the text-to-speech model, and obtaining an unsupervised rhythmic annotation of each word in the text;

The unsupervised prosodic annotation of each word is at least used to perform prosodic synthesis on the prime sequence to obtain speech with prosody.

As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions/modules corresponding to the method in the embodiment of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by the processor, the prosody annotation method and the text-to-speech method based on prosody annotation in any of the above method embodiments are executed.

FIG12 is a schematic diagram of the hardware structure of an electronic device for a prosody annotation method and a text-to-speech method based on prosody annotation provided in another embodiment of the present application. As shown in FIG12 , the device includes:

One or more processors 1210 and a memory 1220 , and FIG12 takes one processor 1210 as an example. The apparatus of the prosody annotation method and the text-to-speech method based on prosody annotation may further include: an input device 1230 and an output device 1240 .

The processor 1210, the memory 1220, the input device 1230 and the output device 1240 may be connected via a bus or other means, and FIG12 takes the connection via a bus as an example.

The memory 1220, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions/modules corresponding to the prosody annotation method and the text-to-speech method based on prosody annotation in the embodiments of the present application. The processor 1210 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1220, that is, the prosody annotation method and the text-to-speech method based on prosody annotation in the above method embodiments are implemented.

The memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data, etc. In addition, the memory 1220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1220 may optionally include a memory remotely arranged relative to the processor 1210, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input device 1230 can receive input digital or character information. The output device 1240 can include a display device such as a display screen.

The one or more modules are stored in the memory 1220 , and when executed by the one or more processors 1210 , perform the prosody annotation method and the text-to-speech method based on prosody annotation in any of the above method embodiments.

The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the prosody annotation method and the text-to-speech method based on prosody annotation of any embodiment of the present invention.

The electronic device of the embodiment of the present application exists in various forms, including but not limited to:

(1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

(2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

(3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

(4) Other electronic devices with data processing functions.

In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (6)

1. A prosody marking method, comprising: Extracting word-level prosodic expressions from training speech data, and reconstructing the speech using the word-level prosodic expressions to obtain reconstructed speech with word-level prosodic expressions; Performing a first clustering on the reconstructed speech according to text information of words; Performing an unsupervised second clustering on the results of the first clustering according to prosody to obtain an unsupervised prosodic labeling of each word; The extracting word-level prosodic expressions from the training speech data and reconstructing the speech using the word-level prosodic expressions comprises: Pre-built FastSpeech2-based text-to-speech model and prosody extractor; Determining a phoneme sequence of the training speech data using an encoder of the text-to-speech model; The prosody extractor extracts the word-level prosodic expression of each word from the Mel-spectrogram of the training speech data, and reconstructs the phoneme sequence using the word-level prosodic expression to obtain reconstructed speech with the word-level prosodic expression. 2. The method according to claim 1, wherein the first clustering of the reconstructed speech according to the text information of the words comprises: The reconstructed speech is subjected to a first clustering based on a decision tree according to text information of words. 3. A prosody annotation system, comprising: A speech reconstruction program module is used to extract word-level prosodic expressions from training speech data, and reconstruct the speech using the word-level prosodic expressions to obtain reconstructed speech with word-level prosodic expressions; A clustering program module, used for performing a first clustering of the reconstructed speech according to the text information of the words; A prosody tagging program module, used for performing an unsupervised second clustering of the results of the first clustering according to the prosody to obtain an unsupervised prosody tagging of each word; The speech reconstruction program module is used for: Pre-built FastSpeech2-based text-to-speech model and prosody extractor; Determining a phoneme sequence of the training speech data using an encoder of the text-to-speech model; The prosody extractor extracts the word-level prosodic expression of each word from the Mel-spectrogram of the training speech data, and reconstructs the phoneme sequence using the word-level prosodic expression to obtain a reconstructed speech with the word-level prosodic expression. 4. The system according to claim 3, wherein the clustering program module is used to: The reconstructed speech is subjected to a first clustering based on a decision tree according to text information of words. 5. An electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the method described in claim 1 or 2. 6. A storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method according to claim 1 or 2 are implemented.