CN116206592A - A voice cloning method, device, equipment and storage medium - Google Patents

Abstract

The present invention provides a voice cloning method, device, equipment and storage medium. The method includes: receiving text to be synthesized; converting the text to be synthesized into a phoneme sequence to be synthesized; inputting the phoneme sequence to be synthesized into adaptive acoustics model, using the adaptive acoustic model to convert the sequence of phonemes to be synthesized into the acoustic features to be synthesized of the target object; wherein, the adaptive acoustic model is to adapt the basic acoustic model based on the target voice sample data of the target object Obtained by training, when performing adaptive training on the basic acoustic model, update the parameters of the conditional normalization layer of the decoder in the basic acoustic model; use a vocoder to synthesize the acoustic features to be synthesized The audio data to be synthesized of the target object. The invention reduces the training time of the self-adaptive model during voice cloning, and improves the efficiency and quality of voice cloning.

Description

A voice cloning method, device, equipment and storage medium

technical field

The invention relates to the technical field of artificial intelligence, in particular to a voice cloning method, device, equipment and storage medium.

Background technique

With the advancement of technology, intelligent voice technology is also constantly developing, such as: voice cloning technology, voice cloning refers to the use of smart devices to synthesize specified content into audio data pronounced by a specified person, for example: a smart speaker at home can emit the voice of the owner .

The general voice cloning technology needs to use the recorded audio of the target speaker in the actual scene to extract the acoustic features and voiceprint features of the target speaker for model training. Finally, the acoustic features of the target speaker can be decoded by the vocoder to obtain the target speaker. Clone voice of speaker. However, when performing model training, a large number of model parameters need to be updated, which requires a long model training time, and will also result in a decrease in the quality and naturalness of the synthesized audio.

Contents of the invention

In view of this, an embodiment of the present invention provides a voice cloning method, device, device, and storage medium, so as to eliminate or improve one or more defects existing in the prior art.

One aspect of the present invention provides a voice cloning method, the method comprising the following steps:

Receive the text to be synthesized;

Converting the text to be synthesized into a phoneme sequence to be synthesized;

The phoneme sequence to be synthesized is input into an adaptive acoustic model, and the phoneme sequence to be synthesized is converted into an acoustic feature to be synthesized of a target object by using the adaptive acoustic model; wherein, the adaptive acoustic model is based on the target The target speech sample data of the object is obtained by performing adaptive training on the basic acoustic model, and updating the parameters of the conditional normalization layer of the decoder in the basic acoustic model when performing adaptive training on the basic acoustic model;

The acoustic features to be synthesized are synthesized into target synthesized audio data of the target object by using a vocoder.

In some embodiments of the present invention, the training method of the adaptive acoustic model includes:

collecting the target voice sample data of the target object;

Converting the target speech sample data into target text;

Extracting target voiceprint vectors and target acoustic features in the target voice sample data;

converting the target text into a target phoneme sequence, and aligning the target voice sample data with the target text to obtain a target duration sequence;

Perform adaptive training on the basic acoustic model based on the target phoneme sequence, the target duration sequence, the target acoustic feature and the target voiceprint vector, and update the conditional normalization of the decoder in the basic acoustic model Layer parameters until the preset requirements are met.

In some embodiments of the present invention, the method also includes:

When the basic acoustic model is adaptively trained based on the target voice sample data of the target object, the neural network algorithm is used to reduce the noise of the target voice sample data, and the basic acoustic model is automatically trained by using the target voice sample data after noise reduction. and adapting the training to obtain the adaptive acoustic model.

In some embodiments of the present invention, the method also includes:

The noise reduction model is used to perform secondary noise reduction on the target synthesized audio data, and the noise-reduced target synthesized audio data is obtained and output.

In some embodiments of the present invention, the method also includes:

The sampling frequency of the target speech sample data is the same as the sampling frequency of the training data of the basic acoustic model.

In some embodiments of the present invention, the basic acoustic model further includes a reference encoder, and the reference encoder is used to learn the rules of the acoustic features in the audio data.

Another aspect of the present invention provides a voice cloning device, the device comprising:

The text receiving module is used for receiving the text to be synthesized;

A phoneme conversion module, configured to convert the text to be synthesized into a phoneme sequence to be synthesized;

An acoustic feature prediction module, configured to input the phoneme sequence to be synthesized into an adaptive acoustic model, and use the adaptive acoustic model to convert the phoneme sequence to be synthesized into an acoustic feature to be synthesized of a target object; wherein the adaptive The acoustic model is obtained by performing adaptive training on the basic acoustic model based on the target voice sample data of the target object. When performing adaptive training on the basic acoustic model, the condition of the decoder in the basic acoustic model is normalized The parameters of the layer are updated;

A speech synthesis module, configured to use a vocoder to synthesize the acoustic features to be synthesized into target synthesized audio data of the target object.

In some embodiments of the present invention, the device further includes a model training module for training the adaptive acoustic model by using the following method:

collecting the target voice sample data of the target object;

Converting the target speech sample data into target text;

Extracting target voiceprint vectors and target acoustic features in the target voice sample data;

converting the target text into a target phoneme sequence, and aligning the target voice sample data with the target text to obtain a target duration sequence;

Perform adaptive training on the basic acoustic model based on the target phoneme sequence, the target duration sequence, the target acoustic feature and the target voiceprint vector, and update the conditional normalization of the decoder in the basic acoustic model Layer parameters until the preset requirements are met.

Another aspect of the present invention provides a voice cloning device, including a processor and a memory, where computer instructions are stored in the memory, and the processor is used to execute the computer instructions stored in the memory, when the computer instructions are When the processor executes, the device implements the above voice cloning method.

Another aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above voice cloning method is implemented.

The voice cloning method, device, equipment and storage medium provided by the present invention only need to collect a small amount of audio data of the target object, and can realize the training and acquisition of the adaptive acoustic model of the target object on the basis of the basic acoustic model. Furthermore, the adaptive acoustic model can be used to convert the specified content into the acoustic characteristics of the target object, and then the audio data of the target object can be cloned by using the vocoder. In the adaptive training of the acoustic model, by introducing the conditional normalization layer into the decoder of the acoustic model, the time required for adaptive training can be reduced without losing the naturalness of the synthesized audio, and the voice cloning is reduced. The training time of the adaptive model in the process increases the speed of voice cloning and ensures the audio quality of voice cloning.

Additional advantages, objects, and features of the present invention will be set forth in part in the following description, and will be partly apparent to those of ordinary skill in the art after studying the following text, or can be learned from the practice of the present invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and appended drawings.

It will be understood by those skilled in the art that the objects and advantages that can be achieved by the present invention are not limited to the above specific ones, and the above and other objects that can be achieved by the present invention will be more clearly understood from the following detailed description.

Description of drawings

The drawings described here are used to provide further understanding of the present invention, constitute a part of the application, and do not limit the present invention. The components in the figures are not drawn to scale, merely illustrating the principles of the invention. For ease of illustration and description of some parts of the present invention, corresponding parts in the figures may be exaggerated, ie, may be made larger relative to other components in an exemplary device actually manufactured in accordance with the present invention. In the attached picture:

Fig. 1 is a schematic flow chart of the voice cloning method provided in one embodiment of this specification;

Fig. 2 is a schematic diagram of the training process of the basic acoustic model in an embodiment of the present specification;

Fig. 3 is a schematic flow chart of voice cloning in another embodiment of the specification;

Fig. 4 is a schematic diagram of the module structure of an embodiment of the voice cloning device provided in this specification;

Fig. 5 is a block diagram of the hardware structure of the voice cloning server in one embodiment of this specification.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with the embodiments and accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

Here, it should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only the structures and/or processing steps closely related to the solution according to the present invention are shown in the drawings, and the related Other details are not relevant to the invention.

It should be emphasized that the term "comprising/comprising" when used herein refers to the presence of a feature, element, step or component, but does not exclude the presence or addition of one or more other features, elements, steps or components.

Here, it should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection, but also to an indirect connection with an intermediate.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

The general speech cloning technology is based on speech synthesis model, which is divided into acoustic model and vocoder. Firstly, a basic acoustic model is trained on multi-speaker data; then, the acoustic features and voiceprint features of the target speaker are extracted using the recorded audio of the target speaker in the actual scene, and adaptive training is performed on the basic acoustic model (Update a large number of network model parameters), infer the corresponding acoustic features of the text to be cloned through the adaptive acoustic model; finally, the acoustic features of the target speaker can be decoded by the vocoder to obtain the cloned voice of the target speaker. Voice cloning technology can synthesize the voice of the target speaker. For example, voice cloning technology can be applied to the dubbing industry to quickly synthesize the voice of the target voice actor, reducing the cost and time of dubbing. Of course, the voice cloning technology can also be applied to other scenarios according to actual needs, and this embodiment of this specification does not specifically limit it.

However, in the real application scenarios of speech cloning technology, in order to achieve a good cloning effect, there are often more requirements on the recordings of the target speaker, such as: the number of recordings, the signal-to-noise ratio of the recordings, and the basic model. The adaptation training takes a long time, and the acoustic model updates more parameters. These limitations will greatly reduce the user experience. In order to improve the naturalness and expressiveness of synthesized audio, the basic acoustic model is often obtained on the premise of high-quality speech synthesis data and a more complex network structure. Adapting to the length of training will be a long-term challenge for the field. In order to solve the reason for the long adaptive training, practitioners often directly reduce the adaptation time of the basic acoustic model, but this will also directly reduce the naturalness of the cloned audio.

The embodiment of this specification provides a voice cloning method, which can pre-train an adaptive acoustic model of the target object based on the basic acoustic model, and add a conditional normalization layer to the basic acoustic model. When training the adaptive acoustic model, only It is necessary to update the parameters of the conditional normalization layer of the decoder in the basic acoustic model. When performing speech cloning, use the trained adaptive acoustic model to synthesize the text to be synthesized into the audio data of the target object's pronunciation. The entire process only needs to update a small number of specified model parameters, and the specified text can be quickly and accurately synthesized into the audio of the target object, which greatly reduces the model training time of voice cloning, thereby improving the efficiency of voice cloning without affecting the audio of voice cloning The quality improves the user experience.

Figure 1 is a schematic flow chart of the voice cloning method provided in an embodiment of this specification, as shown in Figure 1, in an embodiment of the voice cloning method provided in this specification, the method can be applied to computers, tablet computers, servers, smart In terminal devices such as mobile phones and smart wearable devices, the method may include the following steps:

Step 102, receiving text to be synthesized.

In the specific implementation process, voice cloning technology refers to the use of smart devices to synthesize specified content into the audio data pronounced by the target object. The text to be synthesized can be understood as the audio content that needs to be cloned into the target object, which can be text or pictures, or It can be voice, and the voice is converted into corresponding text through voice recognition. For example, if user A wants to use a smart device to synthesize the text "today's weather is fine" into the audio pronounced by user B, the text "today's weather is fine" can be understood as the text to be synthesized. Of course, the text to be synthesized can be in different languages, such as Chinese or English, which can be determined according to actual needs, and is not specifically limited in this embodiment of the specification.

Step 104, converting the text to be synthesized into a phoneme sequence to be synthesized.

In the specific implementation process, after receiving the text to be synthesized, phoneme conversion can be performed. The phoneme sequence in the embodiment of this specification can be understood as the combination of the smallest phonetic unit, and front-end equipment TTS software (text-to-speech software) can be used. The text to be synthesized is converted into the corresponding phoneme sequence to be synthesized.

Step 106: Input the phoneme sequence to be synthesized into an adaptive acoustic model, and use the adaptive acoustic model to convert the phoneme sequence to be synthesized into an acoustic feature to be synthesized of the target object; wherein, the adaptive acoustic model is based on The target speech sample data of the target object is obtained by performing adaptive training on the basic acoustic model. When performing adaptive training on the basic acoustic model, the parameters of the conditional normalization layer of the decoder in the basic acoustic model to update.

In the specific implementation process, the target object can be understood as the object who wants to clone the voice, for example, if the user wants to use the voice cloning device to synthesize the audio pronounced by user B, user B can be called the target object. The target voice sample data of the target object can be used in advance to train and build an adaptive acoustic model. When the audio data of the target object needs to be cloned, the corresponding phoneme sequence to be synthesized into the text to be synthesized is input into the adaptive acoustic model, and the Synthesize the audio data of the target object. Referring to the records of the above-mentioned embodiments, the adaptive acoustic model can be obtained by adaptive training on the basis of the basic acoustic model, the basic acoustic model can be an acoustic model obtained by training with a large amount of voice sample data, and the basic acoustic model can learn the sounds of different objects Features, and then synthesize the specified content into the acoustic features of the specified object. The basic acoustic model can be an autoregressive speech synthesis model such as: Tactotron2, TransformerTTS, Deep Voice 3, or a non-autoregressive speech synthesis model such as: FastSpeech, FastSpeech2 according to actual needs, which is not specifically limited in this embodiment.

In some embodiments of this specification, FastSpeech2 may be used for the basic acoustic model, and a conditional normalization layer (Conditional Layer Normalization, CLN) is added to the basic acoustic model. When it is necessary to clone a target object such as the voice of a certain user, the audio data of the target object, that is, the target voice sample data, can be collected, and the basic acoustic model can be adaptively trained using the target voice sample data of the target object. When the model is adaptively trained, it is only necessary to update the parameters of the conditional normalization layer in the decoder of the basic acoustic model, and train to obtain an adaptive acoustic model. By setting the conditional normalization layer, the number of model parameter updates can be controlled during the training process of the adaptive acoustic model, reducing the model training time of voice cloning, improving the model training efficiency of voice cloning, and for better deployment in the later stage.

Fig. 2 is a schematic diagram of the training process of the basic acoustic model in an embodiment of the present specification. As shown in Fig. 2, the basic acoustic model in the embodiment of the present specification may also include a reference encoder, and the reference encoder (reference encoder) may be used to Model the acoustic features extracted from the audio data, learn the rules of the acoustic features, and then enable the basic acoustic model to extract the user's pronunciation characteristics, such as: stress, pause, language style, etc., and then improve the rhythm of the final synthesized speech And emotional effects, etc., make the voice of the clone closer to the scene of real people speaking.

As shown in Figure 2, the training process of the basic acoustic model can include:

Collecting sample training audio data, extracting training voiceprint vectors and training acoustic features in the sample training audio data;

converting said sample training audio data into training text;

Converting the training text into a training phoneme sequence, aligning the sample training audio data with the training text to obtain a training duration sequence;

Setting model parameters of the basic acoustic model, wherein the basic acoustic model includes an encoder, a duration predictor, and a reference encoder, and the decoder includes a conditional normalization layer;

inputting the training phoneme sequence into the encoder, the training duration sequence into the duration predictor, and the training acoustic features into the reference encoder;

The output of the encoder, the duration predictor, the reference encoder and the voiceprint vector are input into a decoder to perform model training on the basic acoustic model.

As shown in Figure 2, the basic acoustic model can be understood as an intelligent learning model that inputs audio and text data and outputs predicted acoustic features. The sample training audio data for basic acoustic model training and the training text corresponding to the sample audio data can be collected in advance from the audio sample database, and feature extraction is performed on the sample training audio data and training text, and the basic acoustic model is analyzed by using the extracted features. For model training, the specific process is as follows:

Step 1: Feature extraction for audio and text. First, the trained voiceprint recognition model can be used to extract the 256-dimensional voiceprint vector (x-vector) from the audio, that is, the sample training audio data, and continue to extract the acoustic features (320-dimensional mel-spectrum, mel-spectrum) from the audio. ; Secondly, convert the recording text, that is, the training text corresponding to the sample training audio data, into a corresponding phoneme sequence (phoneme); finally, use the trained MFA (MontrealForced Aligner, voice alignment tool) model for the audio and text to perform phoneme pronunciation duration The alignment of is to get the duration sequence (duration).

Step 2: Pass the phoneme sequence through the encoder, pass the duration sequence through the duration predictor, pass the mel spectrum through the reference encoder, and finally send the output of the above parts together with the voiceprint vector to the decoder containing the conditional normalization layer , the corresponding acoustic features are predicted.

Step 3: Calculate the MSE loss between the output of the decoder and the real Mel spectrum, and learn the distribution and regularity of the training sample data.

After a large amount of training data is used to train and obtain the basic acoustic model, the basic acoustic model can be saved. When the specified object needs to be voice cloned, the corresponding audio data can be collected and the basic acoustic model can be adaptively trained. Clone an adaptive acoustic model of the sound of the specified object.

In some embodiments of this specification, the training method of the adaptive acoustic model includes:

collecting the target voice sample data of the target object;

Converting the target speech sample data into target text;

Extracting target voiceprint vectors and target acoustic features in the target voice sample data;

converting the target text into a target phoneme sequence, and aligning the target voice sample data with the target text to obtain a target duration sequence;

Perform adaptive training on the basic acoustic model based on the target phoneme sequence, the target duration sequence, the target acoustic feature and the target voiceprint vector, and update the conditional normalization of the decoder in the basic acoustic model Layer parameters until the preset requirements are met.

In the specific implementation process, FIG. 3 is a schematic flow chart of voice cloning in another embodiment of this specification. As shown in FIG. 3, the target speaker is the target object, and the audio data recorded by the target object can be collected, that is, the target voice sample data, Generally, the target speech sample data does not require a large number of samples like the training of the basic acoustic model. The target speech sample data can be a small amount of audio data, such as: 10 recordings, and only a small amount of audio data of the target object can be used. Train an adaptive acoustic model capable of cloning the acoustic signature of the target object.

In addition, in some embodiments of the present specification, when the target speech sample data of the target object is collected, the sampling frequency of the target speech sample data is the same as the sampling frequency of the training data of the basic acoustic model. The basic acoustic model is trained based on a large amount of training data. By collecting the audio of the target object at the same sampling frequency, the basic acoustic model can quickly and accurately learn the sound characteristics of the target object and improve the audio quality of the voice clone.

After the target speech sample data of the target object is collected, ASR (Automatic Speech Recognition, automatic speech recognition) recognition may be performed on the target speech sample data, and the target speech sample data is converted into a target text. Then perform feature extraction on the target speech sample data and the target text, such as: extracting the target voiceprint vector and target acoustic features in the target speech sample data, converting the target text into a target phoneme sequence, and performing the target speech sample data and the target text Alignment, obtaining the target duration sequence, and the process of feature extraction can refer to the process of feature extraction in the basic acoustic model in Figure 2 above, and will not be repeated here.

As shown in Figure 3, after the feature extraction is completed, the basic acoustic model is adaptively trained based on the extracted target voiceprint vector, target acoustic feature, target duration sequence, and target phoneme sequence. The parameters of the CLN layer are updated until the preset requirements are met, such as: update 500 steps to obtain the adaptive acoustic model of the target speaker.

When adapting to the acoustic model of the target object, the conditional normalization layer in the decoder is updated to achieve only updated 70k parameters. Through experiments, it is found that when the adaptive step is 500, faster and better engraving effects can be achieved. . Table 1 refers to the flow process of Fig. 3 in one embodiment of this description, and collects the related tests carried out by 10 people's test sets (10 voices per person), and Table 1 is the relevant data of the cloning time, naturalness and similarity of the test, wherein , the full score for the similarity and naturalness of the cloned voice is 5 points. As shown in Table 1, the embodiment of this specification starts from the training to obtain the adaptive acoustic model of the target object, and the entire voice cloning takes only 6.2 minutes, and the similarity and naturalness of the voice are relatively good.

Table 1

Time required for voice cloning process Clone Speech Naturalness Clone phonetic similarity 6.2 minutes 3.92 3.87

In the embodiment of this specification, on the basis of the basic acoustic model, only a small amount of audio data of the target object is needed to quickly train and obtain an adaptive acoustic model capable of synthesizing the acoustic features of the target object, and only a small number of parameters are required in the adaptive process The update greatly reduces the time for training the adaptive model in voice cloning, improves the speed of model training in voice cloning, and can quickly obtain adaptive acoustic models for different target objects, laying the foundation for the popularization of voice cloning technology.

Step 108, using a vocoder to synthesize the acoustic features to be synthesized into target synthesized audio data of the target object.

In the specific implementation process, after using the audio data of the target object to train the adaptive acoustic model, when using the adaptive acoustic model, you can specify the sound of the target object, and then the adaptive acoustic model can convert the text to be synthesized into the target The acoustic characteristics of the object. After using the adaptive acoustic model to convert the text to be synthesized into the acoustic features to be synthesized of the target object, a vocoder can be used to synthesize the acoustic features to be synthesized into target synthesized audio data of the pronunciation of the target object.

For example, if user A wants to clone and synthesize user B's voice with his smart device at home, he can collect user B's audio data in advance, refer to the description of the above embodiment, and perform adaptive training on the basic acoustic model to obtain user B's adaptive acoustic model . It is possible to load the adaptive acoustic model obtained through training to the smart device. User A can input the content he wants the smart device to output, such as: input "Happy Birthday", and the adaptive acoustic model in the smart device can output "Happy Birthday" Convert it into the acoustic features of user B, and then use the vocoder to synthesize the corresponding audio data from the acoustic features output by the adaptive acoustic model, and the sound of the audio data is the voice of user B.

In addition, after training the adaptive acoustic model of the target object, if you need to clone the audio data of the target object, you can directly use the trained adaptive acoustic model of the target object to predict the acoustic characteristics of the phoneme sequence to be synthesized. Retrain the adaptive acoustic model of the target object.

The voice cloning method provided by the embodiment of this specification only needs to collect a small amount of audio data of the target object, and can realize training on the basis of the basic acoustic model to obtain the adaptive acoustic model of the target object. Furthermore, the adaptive acoustic model can be used to convert the specified content into the acoustic characteristics of the target object, and then the audio data of the target object can be cloned by using the vocoder. When performing adaptive training on the basic acoustic model, by introducing the conditional normalization layer into the decoder of the acoustic model, the time required for adaptive training can be reduced without losing the naturalness of the synthesized audio, and the speech The training time of the adaptive model during the cloning process improves the speed of voice cloning and ensures the audio quality of voice cloning.

As shown in Figure 3, in some embodiments of this specification, the method further includes:

When the basic acoustic model is adaptively trained based on the target voice sample data of the target object, the neural network algorithm is used to reduce the noise of the target voice sample data, and the basic acoustic model is automatically trained by using the target voice sample data after noise reduction. and adapting the training to obtain the adaptive acoustic model.

In the specific implementation process, when the basic acoustic model is adaptively trained, after the target speech sample data of the target object is collected, a neural network algorithm such as RNN (Recurrent Neural Network, cyclic neural network) can be used to analyze the target speech sample data. Perform noise reduction processing, and then perform feature extraction on the target voice sample data after noise reduction, and use the extracted features to perform adaptive training on the basic acoustic model to obtain an adaptive acoustic model. Generally, in a real usage scenario, the audio of the target speaker has a certain amount of noise, and the signal-to-noise ratio of the recorded audio of the target object is low. When the basic acoustic model is adaptively trained, a large number of model parameters need to be updated, which will lead to Composite audio quality and naturalness will be reduced.

In the embodiment of this specification, by using neural network noise reduction before performing adaptive acoustic model training, the quality of the audio data of the target object can be improved, and it is no longer necessary to impose high requirements on the environment and quality of the audio data recorded by the target object, reducing the The difficulty and requirements of sample collection are reduced, and the quality of voice cloning is improved at the same time.

In addition, as shown in Figure 3, in some embodiments of this specification, the method further includes:

The noise reduction model is used to perform secondary noise reduction on the target synthesized audio data, and the noise-reduced target synthesized audio data is obtained and output.

In the specific implementation process, after the acoustic features output by the adaptive acoustic model are synthesized into the corresponding audio data, a noise reduction model such as: speexdsp can be used to denoise the synthesized audio data, and the target synthesized audio for noise reduction can be obtained and output data. Among them, speexdsp can be used for additional functions such as echo suppression and noise cancellation. It is found through experiments that denoising the final synthesized audio will improve the sound quality of the final synthesized audio.

In the embodiment of this specification, noise reduction is performed twice on the audio data of the target object and the audio data generated by cloning, and noise reduction is performed on the recorded audio and the cloned audio in a specific method, which can improve the sound quality of the cloned audio, thereby improving the voice quality. The quality of the clone.

As shown in Figure 3, the voice cloning method provided by the embodiment of this specification uses 10 sentences of audio recorded by the target speaker in a real usage scenario, and updates the specified network layer on the basis of the trained basic acoustic model, so that The adaptation time of the model is shortened, and two different noise reductions are performed to ensure the sound quality and naturalness of the cloned audio.

First of all, in real usage scenarios, users are in different environments, and some users' recordings will inevitably contain some noise, which will inevitably affect the quality of the final synthesized audio to a certain extent. In the pre-processing of the embodiment of this specification, the main operations include sampling rate conversion, loudness normalization, and noise reduction. The sampling rate of the user recording should be consistent with the sampling rate of the training data of the basic acoustic model. The intensity normalization can be processed with the audio tool sox, and the noise reduction is based on the neural network.

Second, feature extraction. This step is mainly divided into, speech to text, text to phoneme sequence; alignment operation: MFA for alignment; voiceprint vector extraction; acoustic feature extraction. First, the user uses the mobile phone to record 10 smooth audio sentences in a quiet environment, and uses the ASR tool to transcribe the voice and convert it into corresponding text. Secondly, use the Chinese dictionary to convert the recognized text into the corresponding phoneme sequence; then, use the alignment tool MFA to align the phoneme sequence with the corresponding audio, and the result is the pronunciation time corresponding to each phoneme; again, use The trained voiceprint recognition model extracts the speaker information in the audio, and outputs it as a 256-dimensional vector (x-vector); finally, extracts the acoustic features of the audio, and here uses a 320-dimensional mel spectrum.

Finally, the adaptive training of the model. The embodiment of this specification is improved on Fastspeech2. The main change is the addition of a conditional normalization layer and a reference encoder. The purpose of the conditional normalization layer is to control the number of model parameter updates during the adaptive process, and for better deployment in the later stage. The reference encoder is The purpose is to extract the user's pronunciation characteristics from the user's recording.

The voice cloning method provided in the embodiment of this specification, through the improved acoustic model, only needs to update the parameters specified in the acoustic model when performing adaptive learning of voice cloning, which can achieve faster and better engraving effect, and improve The speed and effect of voice cloning are improved, and the neural network is used to reduce noise before self-adaptation. After the audio is synthesized, speedxsp is used to perform a noise reduction process, which can improve the sound quality of the synthesized audio of voice cloning.

Each embodiment of the above-mentioned method in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. For relevant parts, please refer to part of the descriptions of the method embodiments.

Based on the voice cloning method described above, one or more embodiments of this specification further provide a voice cloning device. The device may include devices (including distributed systems), software (applications), modules, components, servers, clients, etc. that use the methods described in the embodiments of this specification combined with necessary implementation hardware. Based on the same innovative idea, the devices in one or more embodiments provided by the embodiments of this specification are as described in the following embodiments. Since the implementation of the device to solve the problem is similar to the method, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetition will not be repeated. As used below, the term "unit" or "module" may be a combination of software and/or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

Specifically, FIG. 4 is a schematic diagram of the module structure of an embodiment of the voice cloning device provided in this specification. As shown in FIG. 4, the device provided in this specification may include:

Text receiving module 41, is used for receiving text to be synthesized;

A phoneme conversion module 42, configured to convert the text to be synthesized into a phoneme sequence to be synthesized;

An acoustic feature prediction module 43, configured to input the phoneme sequence to be synthesized into an adaptive acoustic model, and convert the phoneme sequence to be synthesized into an acoustic feature to be synthesized of a target object by using the adaptive acoustic model; The adaptive acoustic model is obtained by performing adaptive training on the basic acoustic model based on the target speech sample data of the target object. When performing adaptive training on the basic acoustic model, the conditional normalization of the decoder in the basic acoustic model The parameters of the chemical layer are updated;

The speech synthesis module 44 is configured to use a vocoder to synthesize the acoustic features to be synthesized into target synthesized audio data of the target object.

In some embodiments of this specification, the device further includes a model training module, which is used to train the adaptive acoustic model by using the following method:

collecting the target voice sample data of the target object;

Converting the target speech sample data into target text;

Extracting target voiceprint vectors and target acoustic features in the target voice sample data;

converting the target text into a target phoneme sequence, and aligning the target voice sample data with the target text to obtain a target duration sequence;

Perform adaptive training on the basic acoustic model based on the target phoneme sequence, the target duration sequence, the target acoustic feature and the target voiceprint vector, and update the conditional normalization of the decoder in the basic acoustic model Layer parameters until the preset requirements are met.

The speech cloning device provided in the embodiment of this specification only needs to collect a small amount of audio data of the target object, and can realize training on the basis of the basic acoustic model to obtain the adaptive acoustic model of the target object. Furthermore, the adaptive acoustic model can be used to convert the specified content into the acoustic characteristics of the target object, and then the audio data of the target object can be cloned by using the vocoder. When performing adaptive learning on the basic acoustic model, by introducing the conditional normalization layer into the decoder of the acoustic model, it is possible to reduce the time required for adaptive training without losing the naturalness of the synthesized audio, reducing the speech The training time of the adaptive model during the cloning process improves the speed of voice cloning and ensures the audio quality of voice cloning.

In some embodiments of this specification, there is also provided a language synthesis device, including a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory, when the computer When the instructions are executed by the processor, the device implements the voice cloning method recorded in the above embodiments, such as:

Receive the text to be synthesized;

Converting the text to be synthesized into a phoneme sequence to be synthesized;

The phoneme sequence to be synthesized is input into an adaptive acoustic model, and the phoneme sequence to be synthesized is converted into an acoustic feature to be synthesized of a target object by using the adaptive acoustic model; wherein, the adaptive acoustic model is based on the target The target speech sample data of the object is obtained by performing adaptive training on the basic acoustic model, and updating the parameters of the conditional normalization layer of the decoder in the basic acoustic model when performing adaptive training on the basic acoustic model;

The acoustic features to be synthesized are synthesized into target synthesized audio data of the target object by using a vocoder.

It should be noted that the above-mentioned apparatus and equipment may also include other implementation manners according to the description of the method embodiment. For specific implementation manners, reference may be made to descriptions of related method embodiments, and details are not repeated here.

The method embodiments provided in the embodiments of this specification may be executed in mobile terminals, computer terminals, servers or similar computing devices. Taking running on a server as an example, FIG. 5 is a block diagram of the hardware structure of the voice cloning server in an embodiment of this specification. The computer terminal can be the voice cloning server or the voice cloning device in the above embodiments. As shown in Figure 5, the server 10 may include one or more (only one is shown in the figure) processors 100 (the processor 100 may include but not limited to processing devices such as microprocessor MCU or programmable logic device FPGA, etc.), A non-volatile memory 200 for storing data, and a transmission module 300 for communication functions. Those of ordinary skill in the art can understand that the structure shown in FIG. 5 is only a schematic diagram, which does not limit the structure of the above-mentioned electronic device. For example, the server 10 may also include more or fewer components than those shown in FIG. 5 , for example, may also include other processing hardware, such as a database or multi-level cache, GPU, or have a configuration different from that shown in FIG. 5 .

The non-volatile memory 200 can be used to store software programs and modules of application software, such as program instructions/modules corresponding to the voice cloning method in the embodiment of this specification, and the processor 100 can run the software stored in the non-volatile memory 200 Programs and modules to perform various functional applications and resource data updates. The non-volatile memory 200 may include high-speed random access memory, and may also include non-volatile memories, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some examples, the non-volatile memory 200 may further include a memory that is remotely located relative to the processor 100, and these remote memories may be connected to a computer terminal through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission module 300 is used to receive or transmit data via a network. The specific example of the above-mentioned network may include a wireless network provided by the communication provider of the computer terminal. In one example, the transmission module 300 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 300 may be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet in a wireless manner.

Corresponding to the above method, the present invention also provides an apparatus, the apparatus includes computer equipment, the computer equipment includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the instructions in the memory. Stored computer instructions, when the computer instructions are executed by the processor, the device implements the steps of the aforementioned method.

An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned method for deploying an edge computing server can be implemented. The computer readable storage medium may be a tangible storage medium such as random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

Those of ordinary skill in the art should understand that each exemplary component, system and method described in conjunction with the embodiments disclosed herein can be implemented by hardware, software or a combination of the two. Whether it is implemented in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the invention are the programs or code segments employed to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted over transmission media or communication links by data signals carried in carrier waves.

It is to be understood that the invention is not limited to the specific arrangements and processes described above and shown in the drawings. For conciseness, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the sequence of steps after understanding the spirit of the present invention.

In the present invention, features described and/or exemplified for one embodiment can be used in the same or similar manner in one or more other embodiments, and/or can be combined with features of other embodiments or replace other Features of the implementation.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, various modifications and changes may be made to the embodiments of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A voice cloning method, characterized in that the method comprises: Receive the text to be synthesized; Converting the text to be synthesized into a phoneme sequence to be synthesized; The phoneme sequence to be synthesized is input into an adaptive acoustic model, and the phoneme sequence to be synthesized is converted into an acoustic feature to be synthesized of a target object by using the adaptive acoustic model; wherein, the adaptive acoustic model is based on the target The target speech sample data of the object is obtained by performing adaptive training on the basic acoustic model, and updating the parameters of the conditional normalization layer of the decoder in the basic acoustic model when performing adaptive training on the basic acoustic model; The acoustic features to be synthesized are synthesized into target synthesized audio data of the target object by using a vocoder. 2. method according to claim 1, is characterized in that, the training method of described adaptive acoustic model comprises: collecting the target voice sample data of the target object; Converting the target speech sample data into target text; Extracting target voiceprint vectors and target acoustic features in the target voice sample data; converting the target text into a target phoneme sequence, and aligning the target voice sample data with the target text to obtain a target duration sequence; Perform adaptive training on the basic acoustic model based on the target phoneme sequence, the target duration sequence, the target acoustic feature and the target voiceprint vector, and update the conditional normalization of the decoder in the basic acoustic model Layer parameters until the preset requirements are met. 3. The method according to claim 1, wherein the method further comprises: When the basic acoustic model is adaptively trained based on the target voice sample data of the target object, the neural network algorithm is used to reduce the noise of the target voice sample data, and the basic acoustic model is automatically trained by using the target voice sample data after noise reduction. and adapting the training to obtain the adaptive acoustic model. 4. method according to claim 3, is characterized in that, described method also comprises: The noise reduction model is used to perform secondary noise reduction on the target synthesized audio data, and the noise-reduced target synthesized audio data is obtained and output. 5. The method according to claim 2, characterized in that the method further comprises: The sampling frequency of the target speech sample data is the same as the sampling frequency of the training data of the basic acoustic model. 6 . The method according to claim 1 , wherein the basic acoustic model further includes a reference encoder, and the reference encoder is used to learn the rules of the acoustic features in the audio data. 7. A voice cloning device, characterized in that the device comprises: The text receiving module is used for receiving the text to be synthesized; A phoneme conversion module, configured to convert the text to be synthesized into a phoneme sequence to be synthesized; An acoustic feature prediction module, configured to input the phoneme sequence to be synthesized into an adaptive acoustic model, and use the adaptive acoustic model to convert the phoneme sequence to be synthesized into an acoustic feature to be synthesized of a target object; wherein the adaptive The acoustic model is obtained by performing adaptive training on the basic acoustic model based on the target voice sample data of the target object. When performing adaptive training on the basic acoustic model, the condition of the decoder in the basic acoustic model is normalized The parameters of the layer are updated; A speech synthesis module, configured to use a vocoder to synthesize the acoustic features to be synthesized into target synthesized audio data of the target object. 8. The device according to claim 7, wherein the device also includes a model training module for training the adaptive acoustic model using the following method: collecting the target voice sample data of the target object; Converting the target speech sample data into target text; Extracting target voiceprint vectors and target acoustic features in the target voice sample data; converting the target text into a target phoneme sequence, and aligning the target voice sample data with the target text to obtain a target duration sequence; Perform adaptive training on the basic acoustic model based on the target phoneme sequence, the target duration sequence, the target acoustic feature and the target voiceprint vector, and update the conditional normalization of the decoder in the basic acoustic model Layer parameters until the preset requirements are met. 9. A voice cloning device, comprising a processor and a memory, wherein computer instructions are stored in the memory, and the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor When executed, the device implements the steps of the method according to any one of claims 1-6. 10. A computer-readable storage medium, on which a computer program is stored, wherein, when the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are realized.

Images