CN115019767A - Singing voice synthesis method and device - Google Patents

Abstract

The invention provides a singing voice synthesis method and a singing voice synthesis device, wherein the method comprises the following steps: acquiring the sound spectrum data of a target song; extracting phonemes, pitches and audios from the voice signals; processing the phoneme by using an encoder to obtain an implicit spectrum characterization vector; predicting acoustic parameters based on the implicit spectral characterization vector and the pitch by using a predictor; extracting and decomposing a pitch track in the audio to obtain the trill characteristics corresponding to each syllable in the target song; predicting the probability of the trill of each syllable based on the trill characteristics; when the probability of the trill of the syllable meets a preset threshold value, synthesizing the analog trill corresponding to the syllable and marking the syllable; and inputting the voice spectrum data, the acoustic parameters and the simulated trill corresponding to the marked syllables into a preset voice model to generate a synthetic singing voice corresponding to the target song. By applying the method provided by the invention, the prosody of the singing voice is improved through the implicit spectral feature vector, and meanwhile, the expressive force of the singing voice is improved by adding the vibrato, so that the synthesized singing voice is more natural.

Description

Method and device for synthesizing singing voice

technical field

The present invention relates to the field of computer technology, in particular to a method and device for synthesizing singing voices.

Background technique

Singing voice synthesis is a virtual singing voice technology that is synthesized based on the sound spectrum of information such as lyrics, rhythm, and pitch. Existing singing voice synthesis technologies, such as XiaoiceSing, ByteSing, Durian-SC and other singing models, usually directly use information such as notes, phonemes, note durations, and pitches in the score to predict acoustic features. At the same time, the prior art also applies the singing model to predict the energy characteristics of the song, and adds the energy characteristics when synthesizing the singing voice to improve the overall rhythm of the song.

In the process of singing, singers usually use the technique of vibrato to improve the expressiveness of singing, but adding energy features in the prior art cannot simulate the effect of real singers singing vibrato, and the energy features of the prior art are The high-dimensional amplitude spectrum features are calculated and obtained, and a large amount of information will be lost when the high-dimensional energy features are compressed into one-dimensional energy features, and the synthesized singing rhythm performance is not enough, and the expressiveness is also poor.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a singing voice synthesis method, by which, while improving the singing voice rhythm through the implicit spectral representation vector, adding vibrato to improve the expressive power of the singing voice, making the synthesized singing voice more natural.

The present invention also provides a singing voice synthesis device to ensure the realization and application of the above method in practice.

A singing voice synthesis method, comprising:

Obtain the sound spectrum data of the target song;

extracting the phoneme, pitch and audio corresponding to the target song in the sound spectrum data;

Applying a preset encoder to process the phoneme to obtain an implicit spectral representation vector corresponding to the target song, where the implicit spectral representation vector is used to characterize the energy information of the target song;

Applying a preset predictor to predict the acoustic parameter corresponding to the target song based on the implicit spectral representation vector and the pitch;

Extract the pitch track in the audio, and decompose the pitch track to obtain the vibrato feature corresponding to each syllable in the target song;

Based on the vibrato feature corresponding to each of the syllables, predict the probability of vibrato in each of the syllables;

When the probability of vibrato in any syllable meets a preset threshold, synthesizing the analog vibrato corresponding to the syllable, and marking the syllable;

The sound spectrum data, the acoustic parameters, and the simulated vibrato corresponding to each marked syllable are input into a preset sound model to generate a synthetic singing voice corresponding to the target song.

In the above method, optionally, the application of the preset encoder to process the phoneme to obtain an implicit spectral representation vector corresponding to the target song, including:

Inputting the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, where the initial representation vector is a phoneme-level representation sequence;

Perform level expansion on the initial representation vector to obtain an implicit spectral representation vector corresponding to the initial representation vector, where the implicit spectral representation vector is a frame-level representation sequence.

In the above method, optionally, the application preset predictor predicts the acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the pitch, including:

inputting the implicit spectral representation vector to the first test layer of the predictor, to obtain a spectral representation vector output by the first test layer, where the first test layer is a one-dimensional convolution layer;

Splicing the spectral representation vector, the implicit spectral representation vector and the pitch to obtain splicing data;

The spliced data is input into the second test layer of the tester, and the acoustic parameters output by the second test layer are obtained, and the second test layer is a one-dimensional convolution layer.

In the above-mentioned method, optionally, the decomposing the pitch trajectory to obtain the vibrato feature corresponding to each syllable in the target song, including:

Applying a preset triangular filter to perform a convolution operation on the pitch track to obtain a smooth pitch track;

Decomposing the smooth pitch track to obtain the vibrato corresponding to each syllable;

A preset Hilbert transformation method is applied to transform the vibrato corresponding to each syllable to obtain the vibrato feature corresponding to each syllable.

The above method, optionally, the synthesizing the analog vibrato corresponding to the syllable, including:

Select the amplitude, frequency and phase in the vibrato feature corresponding to the syllable;

A preset vibrato simulation algorithm is applied to calculate the amplitude, frequency, phase and probability of vibrato in the syllable to generate a simulated vibrato corresponding to the syllable.

A singing voice synthesis device, comprising:

The first acquisition unit is used to acquire the sound spectrum data of the target song;

an extraction unit for extracting the phoneme, pitch and audio corresponding to the target song in the sound spectrum data;

a processing unit, configured to process the phoneme by applying a preset encoder to obtain an implicit spectral representation vector corresponding to the target song, where the implicit spectral representation vector is used to characterize the energy information of the target song;

a first prediction unit, configured to apply a preset predictor to predict the acoustic parameter corresponding to the target song based on the implicit spectral representation vector and the pitch;

a decomposition unit, used to extract the pitch track in the audio, and decompose the pitch track to obtain the vibrato feature corresponding to each syllable in the target song;

The second prediction unit is used to predict the probability of vibrato in each of the syllables based on the vibrato feature corresponding to each of the syllables;

The first synthesis unit is used to synthesize the analog vibrato corresponding to the syllable when the probability of vibrato in any syllable meets a preset threshold, and mark the syllable;

The second synthesis unit is configured to input the sound spectrum data, the acoustic parameters and the simulated vibrato corresponding to each marked syllable into a preset sound model to generate a synthesized singing voice corresponding to the target song.

The above device, optionally, the processing unit includes:

The first input subunit is used for inputting the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, where the initial representation vector is a phoneme-level representation sequence;

The expansion subunit is configured to perform level expansion on the initial characterization vector to obtain an implicit spectral characterization vector corresponding to the initial characterization vector, where the implicit spectral characterization vector is a frame-level characterization sequence.

In the above device, optionally, the first prediction unit includes:

The second input subunit is used to input the implicit spectral representation vector to the first test layer of the predictor, and obtain the spectral representation vector output by the first test layer, and the first test layer is one-dimensional convolutional layer;

a splicing subunit, used for splicing the spectral representation vector, the implicit spectral representation vector and the pitch to obtain splicing data;

The third input subunit is used for inputting the spliced data into the second test layer of the tester to obtain acoustic parameters output by the second test layer, where the second test layer is a one-dimensional convolution layer.

The above-mentioned device, optionally, the decomposition unit includes:

an operation subunit, used for applying a preset triangular filter to perform a convolution operation on the pitch track to obtain a smooth pitch track;

a decomposition subunit, used for decomposing the smooth pitch track to obtain the vibrato corresponding to each syllable;

The transformation subunit is configured to perform transformation operations on the vibrato corresponding to each of the syllables by applying a preset Hilbert transformation method to obtain the vibrato feature corresponding to each of the syllables.

The above-mentioned device, optionally, the first synthesis unit includes:

Selecting a subunit for selecting amplitude, frequency and phase in the vibrato feature corresponding to the syllable;

The synthesis subunit is configured to apply a preset vibrato simulation algorithm to calculate the amplitude, frequency, phase and probability of vibrato in the syllable, and generate a simulated vibrato corresponding to the syllable.

A storage medium, the storage medium comprising stored instructions, wherein when the instructions are executed, a device where the storage medium is located is controlled to execute the above singing voice synthesis method.

An electronic device includes a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and configured to perform the above-described singing voice synthesis method by one or more processors.

Compared with the prior art, the present invention includes the following advantages:

The present invention provides a method for synthesizing singing voice, which includes: acquiring sound spectrum data of a target song; extracting phonemes, pitches and audios corresponding to the target song in the sound spectrum data; processing to obtain an implicit spectral representation vector corresponding to the target song, the implicit spectral representation vector is used to characterize the energy information of the target song; a preset predictor is applied based on the implicit spectral representation vector and the pitch, predict the acoustic parameters corresponding to the target song; extract the pitch trajectory in the audio, and decompose the pitch trajectory to obtain the vibrato feature corresponding to each syllable in the target song; based on each of the The vibrato feature corresponding to the syllable is to predict the probability of vibrato in each syllable; when the probability of vibrato in any syllable meets the preset threshold, the simulated vibrato corresponding to the syllable is synthesized, and the syllable is marked; The sound spectrum data, the acoustic parameters, and the simulated vibrato corresponding to each marked syllable are input to a preset sound model to generate a synthesized singing voice corresponding to the target song. By applying the method provided by the present invention, the rhythm of the singing voice is improved through the implicit spectral representation vector, and the expressiveness of the singing voice is improved by adding vibrato, so that the synthesized singing voice is more natural.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

Fig. 1 is the method flow chart of a kind of singing voice synthesis method provided in the embodiment of the present invention;

2 is an overall network structure diagram of a singing model provided by an embodiment of the present invention;

Fig. 3 is another method flow chart of a kind of singing voice synthesis method provided by the embodiment of the present invention;

4 is a device structure diagram of a singing voice synthesis device provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that any such relationship exists between these entities or operations. The terms "comprising", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Include other elements not expressly listed, or which are inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The present invention may be used in numerous general purpose or special purpose computing device environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet-type devices, multi-processor devices, distributed computing environments including any of the above, and the like.

An embodiment of the present invention provides a singing voice synthesis method. The method can be applied to various system platforms, and the execution body of the method can be a computer terminal or a processor of various mobile devices. The method flowchart of the method is shown in FIG. 1 . , including:

S101: Acquire sound spectrum data of a target song.

In this embodiment of the present invention, the sound spectrum data includes information such as phoneme, pitch, and audio.

S102: Extract the phoneme, pitch and audio corresponding to the target song in the sound spectrum data.

In the embodiment of the present invention, a phoneme is the smallest unit of speech divided according to the natural attributes of speech, and is analyzed according to the pronunciation action in the syllable, and one action constitutes a phoneme. For example, the Chinese characters "kai (kai)" and "xin (xin)" have three phonemes respectively. Pitch refers to sounds with different pitches.

S103: Use a preset encoder to process the phoneme to obtain an implicit spectral representation vector corresponding to the target song, where the implicit spectral representation vector is used to represent energy information of the target song.

In the embodiment of the present invention, the encoder is a part of the acoustic model, and the phoneme is input into the encoder of the acoustic model, and the representation vector of the phoneme, that is, the implicit spectral representation vector, is obtained by the encoder.

S104: Apply a preset predictor to predict the acoustic parameter corresponding to the target song based on the implicit spectral representation vector and the pitch.

In the embodiment of the present invention, the predictor is a part of the acoustic model. After processing the implicit spectral representation vector, it is combined with the pitch input predictor, and the predictor outputs acoustic parameters. Among them, the acoustic parameters include MGC (Mel-Generalized Cepstral, Mel generalized cepstral), BAP (BandAperiodicity, with aperiodic signal) and pitch. Wherein, with aperiodicity refers to the power ratio between the speech signal and the aperiodic component of the signal.

S105: Extract the pitch track in the audio, and decompose the pitch track to obtain the vibrato feature corresponding to each syllable in the target song.

In this embodiment of the present invention, after decomposing the pitch track, it includes pitch in addition to vibrato, obtains the vibrato from the treble track, and obtains the vibrato features corresponding to the syllables in the vibrato, including amplitude extent, frequency rate and phase phase.

Further, decompose the pitch track to obtain the vibrato feature corresponding to each syllable in the target song, including:

Applying a preset triangular filter to perform a convolution operation on the pitch track to obtain a smooth pitch track;

Decomposing the smooth pitch track to obtain the vibrato corresponding to each syllable;

A preset Hilbert transformation method is applied to transform the vibrato corresponding to each syllable to obtain a vibrato feature corresponding to each syllable, where the vibrato feature includes amplitude, frequency and phase.

It should be noted that the real treble trajectory pitchtrajectory is extracted from the audio data, and then a triangular filter is used to perform a convolution operation with the extracted treble trajectory to obtain a smooth treble trajectory. The treble trajectory is decomposed into pitch pitchintonation and vibrato, pitch intonation It is the pitchcontour after smoothing, and the vibrato shows the performance skills of the singer. The vibrato is further decomposed into three features: amplitude extent, frequency rate and phase phase through the Hilbert transform.

S106: Based on the vibrato feature corresponding to each of the syllables, predict the probability of vibrato appearing in each of the syllables.

In this embodiment of the present invention, the probability of vibrato appearing in each syllable is an average probability, that is, the average probability of vibrato appearing on each syllable. Trills are basically absent in shorter-pronounced syllables, and only appear on longer-pronounced syllables, so the longer the syllable, the higher the probability of vibrato.

Optionally, each vibrato feature can be input into the vibrato similarity annotation network to predict the probability of vibrato in each syllable.

S107: When the probability of vibrato in any syllable meets a preset threshold, synthesize the analog vibrato corresponding to the syllable, and mark the syllable.

Further, synthesizing the analog vibrato corresponding to the syllable may specifically include:

Select the amplitude, frequency and phase in the vibrato feature corresponding to the syllable;

A preset vibrato simulation algorithm is applied to calculate the amplitude, frequency, phase and probability of vibrato in the syllable to generate a simulated vibrato corresponding to the syllable.

Among them, the calculation formula of the analog vibrato corresponding to each byte is as follows:

分别是预测的颤音的extent,rate,phase和likeliness。where p _mean is the calculated mean vibrato probability over that syllable,

are the extent, rate, phase and likeliness of the predicted vibrato, respectively.

It should be noted that, because it is difficult to directly extract whether there is vibrato in Pitchtrajectory as a tag of vibrato possibility in singing voice data through an algorithm, the present invention proposes a method for constructing simulated vibrato data and tags to obtain vibrato tags and tags. The corresponding pitchtrajectory.

First, extract the real pitchtrajectory from the audio data, and then perform a convolution operation with the extracted pitchtrajectory through a triangular filter to obtain a smooth pitchtrajectory, and then we pass a vibrato simulation algorithm (for example: randomly select the extent of the vibrato, and multiply the Construct vibrato data with 6Hz sine wave noise), and then randomly select a segment of the smoothed pitchtrajectory, add vibrato to this segment, so as to obtain a pitchtrajectory containing vibrato, and know the position of the randomly selected segment, so that Just know the vibrato tag. The randomly selected clip is vibrato, so the vibrato label of the frame corresponding to this clip is 1, that is, vibrato occurs, and the frames outside this clip do not contain vibrato, and the vibrato label is 0.

With the simulated pitchtrajectory containing the vibrato and the corresponding labels of whether the vibrato is present, a neural network that predicts the probability of vibrato, that is, a neural network that predicts the probability of vibrato, can be trained with these fake data. After the network training is completed, the network can be used to mark the training data of the pitchmodel, that is, we can extract the real pitch trajectory of the pitchmodel training data, and then predict whether there is a vibrato in the position of each frame in the pitchtrajectory, thus obtaining the training pitchmodel Labels for the vibrato likelihood of the training data.

S108: Input the sound spectrum data, the acoustic parameters, and the simulated vibrato corresponding to each marked syllable into a preset sound model to generate a synthesized singing voice corresponding to the target song.

In the embodiment of the present invention, vibrato and acoustic parameters are added in the singing voice synthesis process to improve the expressive power of the synthesized singing voice and make the synthesized singing voice more natural.

It should be noted that the sound model is a neural network model for synthesizing singing voices. After multiple machine learnings, the model simulates the corresponding vocal synthesis target song according to various input parameters (such as: sound spectrum data, acoustic parameters and simulated vibrato). Synthesized vocals.

In the method provided by the embodiment of the present invention, the sound spectrum data of the target is acquired, and the phoneme, pitch and audio therein are extracted. The phoneme is processed by the encoder to obtain the implicit spectral representation vector, and the implicit spectral representation vector and pitch are further processed by the predictor to obtain one-dimensional acoustic parameters, which can be used in the process of synthesizing singing voices. Increase the rhythm of the song. After the acoustic parameters are obtained, the vibrato feature corresponding to each syllable is obtained from the audio frequency, and the possibility of vibrato appearing in each syllable is predicted through the vibrato feature, that is, the probability of vibrato appearing. When vibrato occurs in any syllable, that is, when the probability of vibrato is greater than the threshold, the simulated vibrato is synthesized and marked, and the label of the possibility of vibrato of this syllable is set through the mark. Finally, after inputting the spectral data, acoustic parameters and simulated vibrato into the sound model, the singing voice of the target song is synthesized.

By applying the method provided by the embodiment of the present invention, while improving the rhythm of the singing voice through the implicit spectral representation vector, adding vibrato to improve the expressive power of the singing voice, making the synthesized singing voice more natural.

Referring to Figure 2, Figure 2 is the overall network structure diagram of the singing model, which includes three parts, the first part is (a) is the acoustic model, the input of the acoustic model contains phonemes, and pitch, and then the acoustic model passes through the encoder's input. The output predicts the implicit spectral representation vector, and the predicted implicit spectral representation vector and the representation obtained by the encoder are concatenated and sent to the decoder to predict the acoustic features (MGC, BAP, Pitch); in Figure 1 (b ) is the pitch model of pitchmodel. The input of pitchmodel is phoneme and syllable and the corresponding singer id. In pitchmodel, pitch is decomposed into information such as intonation, phase, extent, rate, etc., and we also model the possibility of vibrato. , that is, the likeliness in the figure; (c) in Figure 1 is the vibrato likelihood labeling network, which is used to label the training data with vibrato likelihood labels. Among them, "c" in Figure 2 means connection, "+" means adding, "f" is the formula for generating analog vibrato, "FC" is the fully connected layer, "LR" is the length regulator, and "Rep" is the abbreviation of representation , Vibrato likeliness is the possibility of vibrato, that is, the probability of vibrato.

In the method provided by the embodiment of the present invention, a preset encoder is used to process the phoneme, and an implicit spectral representation vector corresponding to the target song is obtained, including:

Inputting the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, where the initial representation vector is a phoneme-level representation sequence;

Perform level expansion on the initial representation vector to obtain an implicit spectral representation vector corresponding to the initial representation vector, where the implicit spectral representation vector is a frame-level representation sequence.

It should be noted that, as shown in Figure 2, the process of obtaining the implicit spectral representation vector may be: the input of the encoder of the acoustic model is a phoneme, and after obtaining the representation vector of the phoneme through the encoder, according to the phoneme duration information in the song score, the The output of the encoder is expanded to the frame level, that is, from the sequence of representations at the phoneme level to the sequence of representations at the frame level.

Further, with reference to FIG. 3 , a preset predictor is applied to predict the acoustic parameters corresponding to the target song based on the implicit spectral representation vector and the acoustic spectral data, including:

S301: Input the implicit spectral representation vector to the first test layer of the predictor, and obtain the spectral representation vector output by the first test layer.

Wherein, the first test layer is a one-dimensional convolution layer.

It should be noted that the predictor includes two layers of one-dimensional convolution, the output layer is a linear transformation, and predicts a spectral representation vector with a dimension of 256 per frame.

S302: Splicing the spectral representation vector, the implicit spectral representation vector and the pitch to obtain splicing data.

S303: Input the spliced data into the second test layer of the tester, and obtain the acoustic parameters output by the second test layer.

Among them, the second test layer is a one-dimensional convolutional layer.

It should be noted that the output of the encoder that predicts the spectral representation and the acoustic model and the pitchembedding are spliced together and sent to the acoustic model tester for acoustic parameter prediction. The acoustic parameters used in the present invention are MGC (Mel-Generalized Cepstral), BAP (BandAperiodicity, with aperiodicity) and Pitch. Among them, the output of the encoder is the implicit spectral representation vector. As shown in Table 1, Table 1 shows the dimensions of different characteristics of acoustic parameters in the acoustic model:

Table 1

It should also be noted that, in order to enable the spectral representation vector to represent the energy information very well, the present invention adds a decoder to the acoustic model, so that the predicted general representation vector can return the energy of the audio corresponding to the input text through the decoder. spectrum, because the singing data all appear in pairs, that is, a text input corresponds to an audio output, so the spectrum representation obtained by the spectrum representation predictor can be used to reply the corresponding energy spectrum through a decoder, which is the ( The part in the dashed box in part a), this part of the model only needs to be used in the training process, not in the inference. The spectral representation predicted by the spectral representation predictor can be directly used for acoustic parameter prediction during inference.

Based on the method provided by the above-mentioned embodiment, the present invention displays features such as Phase, Extent, Rate, Likeliness and the like for predicting the vibrato through the vibrato model, and then improves the expressiveness of the singing model through these features; A method to improve the expressiveness of singing rhythm with a latent spectrogram representation with high expressiveness (ie, the bottle neck feature of energy).

The specific implementation process of each of the above-mentioned embodiments and the derivatives thereof are all within the protection scope of the present invention.

Corresponding to the method described in FIG. 1 , an embodiment of the present invention also provides a singing voice synthesis device for implementing the method in FIG. 1 . The singing voice synthesis device provided by the embodiment of the present invention can be applied to a computer terminal or various In the mobile device, a schematic diagram of its structure is shown in Figure 4, which specifically includes:

The first obtaining unit 401 is used to obtain the sound spectrum data of the target song;

Extraction unit 402, for extracting the phoneme, pitch and audio corresponding to the target song in the sound spectrum data;

A processing unit 403, configured to process the phoneme by applying a preset encoder to obtain an implicit spectral representation vector corresponding to the target song, where the implicit spectral representation vector is used to characterize the energy information of the target song;

a first predicting unit 404, configured to apply a preset predictor to predict the acoustic parameter corresponding to the target song based on the implicit spectral representation vector and the pitch;

Decomposition unit 405 is used to extract the pitch track in the audio, and decompose the pitch track to obtain the vibrato feature corresponding to each syllable in the target song;

The second prediction unit 406 is configured to predict the probability of vibrato appearing in each of the syllables based on the vibrato feature corresponding to each of the syllables;

The first synthesizing unit 407 is used to synthesize the analog vibrato corresponding to the syllable when the probability of vibrato appearing in any syllable meets a preset threshold, and mark the syllable;

The second synthesis unit 408 is configured to input the sound spectrum data, the acoustic parameters and the simulated vibrato corresponding to each marked syllable into a preset sound model to generate a synthesized singing voice corresponding to the target song.

In the device provided by the embodiment of the present invention, the sound spectrum data of the target is acquired, and the phoneme, pitch and audio therein are extracted. The phoneme is processed by the encoder to obtain the implicit spectral representation vector, and the implicit spectral representation vector and pitch are further processed by the predictor to obtain one-dimensional acoustic parameters, which can be used in the process of synthesizing singing voices. Increase the rhythm of the song. After the acoustic parameters are obtained, the vibrato feature corresponding to each syllable is obtained from the audio frequency, and the possibility of vibrato appearing in each syllable is predicted through the vibrato feature, that is, the probability of vibrato appearing. When vibrato occurs in any syllable, that is, when the probability of vibrato is greater than the threshold, the simulated vibrato is synthesized and marked, and the label of the possibility of vibrato of this syllable is set through the mark. Finally, after inputting the spectral data, acoustic parameters and simulated vibrato into the sound model, the singing voice of the target song is synthesized.

By applying the device provided by the embodiment of the present invention, the rhythm of the singing voice is improved by the implicit spectral representation vector, and the expressive power of the singing voice is improved by adding vibrato, so that the synthesized singing voice is more natural.

In the apparatus provided by the embodiment of the present invention, the processing unit 403 includes:

The first input subunit is used for inputting the phoneme into the encoder to obtain an initial representation vector corresponding to the target song output by the encoder, where the initial representation vector is a phoneme-level representation sequence;

The expansion subunit is configured to perform level expansion on the initial characterization vector to obtain an implicit spectral characterization vector corresponding to the initial characterization vector, where the implicit spectral characterization vector is a frame-level characterization sequence.

In the apparatus provided by the embodiment of the present invention, the first prediction unit 404 includes:

The second input subunit is used to input the implicit spectral representation vector to the first test layer of the predictor, and obtain the spectral representation vector output by the first test layer, and the first test layer is one-dimensional convolutional layer;

a splicing subunit, used for splicing the spectral representation vector, the implicit spectral representation vector and the pitch to obtain splicing data;

The third input subunit is used for inputting the spliced data into the second test layer of the tester to obtain acoustic parameters output by the second test layer, where the second test layer is a one-dimensional convolution layer.

In the device provided by the embodiment of the present invention, the decomposition unit 405 includes:

an operation subunit, used for applying a preset triangular filter to perform a convolution operation on the pitch track to obtain a smooth pitch track;

a decomposition subunit, used for decomposing the smooth pitch track to obtain the vibrato corresponding to each syllable;

The transformation subunit is configured to perform transformation operations on the vibrato corresponding to each of the syllables by applying a preset Hilbert transformation method to obtain the vibrato feature corresponding to each of the syllables.

In the device provided by the embodiment of the present invention, the first synthesis unit 407 includes:

Selecting a subunit for selecting amplitude, frequency and phase in the vibrato feature corresponding to the syllable;

The synthesis subunit is configured to apply a preset vibrato simulation algorithm to calculate the amplitude, frequency, phase and probability of vibrato in the syllable, and generate a simulated vibrato corresponding to the syllable.

For the specific working process of each unit and sub-unit in the singing voice synthesis device disclosed in the above embodiments of the present invention, reference may be made to the corresponding content in the singing voice synthesis method disclosed in the above embodiments of the present invention, which will not be repeated here.

An embodiment of the present invention further provides a storage medium, where the storage medium includes stored instructions, wherein when the instructions are executed, a device where the storage medium is located is controlled to execute the above singing voice synthesis method.

An embodiment of the present invention further provides an electronic device, the schematic structural diagram of which is shown in FIG. 5 , and specifically includes a memory 501 and one or more instructions 502 , wherein one or more instructions 502 are stored in the memory 501 and are processed through the memory 501 . The one or more instructions 502 are configured to be executed by the one or more processors 503 to:

Obtain the sound spectrum data of the target song;

extracting the phoneme, pitch and audio corresponding to the target song in the sound spectrum data;

Applying a preset encoder to process the phoneme to obtain an implicit spectral representation vector corresponding to the target song, where the implicit spectral representation vector is used to characterize the energy information of the target song;

Applying a preset predictor to predict the acoustic parameter corresponding to the target song based on the implicit spectral representation vector and the pitch;

Extract the pitch track in the audio, and decompose the pitch track to obtain the vibrato feature corresponding to each syllable in the target song;

Based on the vibrato feature corresponding to each of the syllables, predict the probability of vibrato in each of the syllables;

When the probability of vibrato in any syllable meets a preset threshold, synthesizing the analog vibrato corresponding to the syllable, and marking the syllable;

The sound spectrum data, the acoustic parameters, and the simulated vibrato corresponding to each marked syllable are input into a preset sound model to generate a synthetic singing voice corresponding to the target song.

Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for related parts. The systems and system embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, It can be located in one place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two.

In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functions in the foregoing description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A singing voice synthesizing method, comprising:

acquiring the sound spectrum data of a target song;

extracting phonemes, pitches and audios corresponding to the target songs in the sound spectrum data;

processing the phonemes by using a preset encoder to obtain an implicit spectral characterization vector corresponding to the target song, wherein the implicit spectral characterization vector is used for characterizing energy information of the target song;

predicting acoustic parameters corresponding to the target song based on the implicit spectral characterization vector and the pitch by applying a preset predictor;

extracting a pitch track in the audio, decomposing the pitch track, and obtaining a vibrato characteristic corresponding to each syllable in the target song;

predicting the probability of the trill of each syllable based on the trill characteristic corresponding to each syllable;

when the probability of the vibrato of any syllable meets a preset threshold value, synthesizing the analog vibrato corresponding to the syllable and marking the syllable;

and inputting the sound spectrum data, the acoustic parameters and the simulated trills corresponding to the marked syllables into a preset sound model to generate a synthetic singing voice corresponding to the target song.

2. The method of claim 1, wherein the applying a preset encoder to process the phoneme to obtain an implicit spectral feature vector corresponding to the target song comprises:

inputting the phonemes into the encoder, and obtaining initial characterization vectors corresponding to the target songs output by the encoder, wherein the initial characterization vectors are characterization sequences at a phoneme level;

and performing level expansion on the initial characterization vector to obtain an implicit spectrum characterization vector corresponding to the initial characterization vector, wherein the implicit spectrum characterization vector is a frame-level characterization sequence.

3. The method of claim 1 or 2, wherein the applying a preset predictor to predict the acoustic parameters corresponding to the target song based on the implicit spectral characterization vector and the pitch comprises:

inputting the implicit spectral feature vector into a first test layer of the predictor to obtain a spectral feature vector output by the first test layer, wherein the first test layer is a one-dimensional convolutional layer;

splicing the spectrum characterization vector, the implicit spectrum characterization vector and the pitch to obtain spliced data;

and inputting the splicing data into a second test layer of the tester to obtain the acoustic parameters output by the second test layer, wherein the second test layer is a one-dimensional convolutional layer.

4. The method of claim 1, wherein the decomposing the pitch track to obtain the trill feature for each syllable in the target song comprises:

performing convolution operation on the pitch track by applying a preset triangular filter to obtain a smooth pitch track;

decomposing the smooth pitch track to obtain the trills corresponding to each syllable;

and performing transformation operation on the vibrato corresponding to each syllable by applying a preset Hilbert transformation method to obtain the vibrato characteristic corresponding to each syllable.

5. The method according to claim 1 or 4, wherein the synthesizing of the analog vibrato corresponding to the syllable comprises:

selecting amplitude, frequency and phase in the trill characteristics corresponding to the syllables;

and calculating the amplitude, the frequency, the phase and the probability of the trill of the syllable by applying a preset trill simulation algorithm to generate the simulated trill corresponding to the syllable.

6. A singing voice synthesizing apparatus, comprising:

a first acquisition unit for acquiring the sound spectrum data of a target song;

the extraction unit is used for extracting phonemes, pitches and audios corresponding to the target songs from the sound spectrum data;

the processing unit is used for processing the phonemes by using a preset encoder to obtain an implicit spectral characterization vector corresponding to the target song, and the implicit spectral characterization vector is used for characterizing energy information of the target song;

the first prediction unit is used for applying a preset predictor to predict the acoustic parameters corresponding to the target song based on the implicit spectral characterization vector and the pitch;

the decomposition unit is used for extracting a pitch track in the audio, decomposing the pitch track and obtaining the trill characteristics corresponding to each syllable in the target song;

a second prediction unit, configured to predict a probability of a vibrato occurring for each of the syllables based on a vibrato characteristic corresponding to each of the syllables;

the first synthesis unit is used for synthesizing the simulated vibrato corresponding to the syllable and marking the syllable when the probability of the vibrato appearing on any syllable meets a preset threshold value;

and the second synthesis unit is used for inputting the sound spectrum data, the acoustic parameters and the simulated trill corresponding to each marked syllable into a preset sound model to generate the synthetic singing voice corresponding to the target song.

7. The apparatus of claim 6, wherein the processing unit comprises:

a first input subunit, configured to input the phoneme to the encoder, and obtain an initial characterization vector corresponding to the target song output by the encoder, where the initial characterization vector is a phoneme-level characterization sequence;

and the expansion subunit is used for performing level expansion on the initial characterization vector to obtain an implicit spectrum characterization vector corresponding to the initial characterization vector, wherein the implicit spectrum characterization vector is a frame-level characterization sequence.

8. The apparatus according to claim 6 or 7, wherein the first prediction unit comprises:

the second input subunit is configured to input the implicit spectral feature vector to a first test layer of the predictor, so as to obtain a spectral feature vector output by the first test layer, where the first test layer is a one-dimensional convolutional layer;

a splicing subunit, configured to splice the spectrum characterization vector, the implicit spectrum characterization vector, and the pitch to obtain splicing data;

and the third input subunit is used for inputting the splicing data into a second test layer of the tester to obtain the acoustic parameters output by the second test layer, and the second test layer is a one-dimensional convolutional layer.

9. The apparatus of claim 6, wherein the decomposition unit comprises:

the operation subunit is used for performing convolution operation on the pitch track by applying a preset triangular filter to obtain a smooth pitch track;

the decomposition subunit is used for decomposing the smooth pitch track to obtain the trills corresponding to each syllable;

and the transformation subunit is used for performing transformation operation on the vibrato corresponding to each syllable by applying a preset Hilbert transformation method to obtain the vibrato characteristic corresponding to each syllable.

10. The apparatus of claim 6 or 9, wherein the first synthesis unit comprises:

a selecting subunit, configured to select an amplitude, a frequency, and a phase of a vibrato feature corresponding to the syllable;

and the synthesis subunit is used for calculating the amplitude, the frequency, the phase and the probability of the trill of the syllable by applying a preset trill simulation algorithm to generate the simulated trill corresponding to the syllable.

Images