CN115148186A - Speech synthesis method, apparatus, readable medium and electronic device - Google Patents

Abstract

The present disclosure relates to a speech synthesis method, apparatus, readable medium and electronic device. The method includes: determining the tone annotation information of the target text to be processed, wherein the tone annotation information includes the tone change type of each text unit in the target text, wherein the text unit is composed of at least one unit text , the continuous tone-shift type of the text unit is used to indicate the pitch change trend of the text unit; determine the prosodic labeling information of the target text and the phoneme sequence corresponding to the target text; according to the tone labeling information, the prosody labeling information and the phoneme sequence to generate synthesized audio corresponding to the target text. As a result, when performing speech synthesis for the target text, the feature of continuous tone change is introduced, so that the continuous tone change mode can be directly controlled during speech synthesis, which improves the controllability of the continuous tone change phenomenon in speech synthesis, and further The naturalness of the synthesized speech.

Description

Speech synthesis method, apparatus, readable medium and electronic device

technical field

The present disclosure relates to the field of computer technology, and in particular, to a speech synthesis method, apparatus, readable medium, and electronic device.

Background technique

Speech synthesis technology can convert any text into corresponding audio, which usually includes two parts, one part is to analyze the text to obtain linguistically related information, and the other part is to generate sound waveforms based on the analysis results. In the related art, there is usually a lack of learning of the feature of link-to-speech pitch change, so that the pitch of the synthesized speech cannot be effectively controlled, resulting in an unnaturally synthesized audio.

SUMMARY OF THE INVENTION

This section is provided to introduce in a simplified form concepts that are described in detail in the detailed description that follows. This section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

In a first aspect, the present disclosure provides a speech synthesis method, the method comprising:

Determine the tone annotation information of the target text to be processed, wherein the tone annotation information includes the tone change type of each text unit in the target text, wherein the text unit is composed of at least one unit text, and the continuous tone of the text unit is The reading pitch type is used to indicate the pitch change trend of the text unit;

Determine the prosodic annotation information of the target text and the phoneme sequence corresponding to the target text;

According to the tone annotation information, the prosodic annotation information and the phoneme sequence, a synthesized audio corresponding to the target text is generated.

In a second aspect, the present disclosure provides a speech synthesis device, the device comprising:

The first determination module is used to determine the tone annotation information of the target text to be processed, wherein the tone annotation information includes the tone change type of each text unit in the target text, wherein the text unit is composed of at least one unit. Text composition, the type of tone change of the text unit is used to indicate the pitch change trend of the text unit;

a second determination module, configured to determine the prosodic annotation information of the target text and the phoneme sequence corresponding to the target text;

A generating module, configured to generate synthetic audio corresponding to the target text according to the tone annotation information, the prosody annotation information and the phoneme sequence.

In a third aspect, the present disclosure provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing apparatus, implements the steps of the method described in the first aspect of the present disclosure.

In a fourth aspect, the present disclosure provides an electronic device, comprising:

a storage device having at least one computer program stored thereon;

At least one processing device is configured to execute the at least one computer program in the storage device to implement the steps of the method in the first aspect of the present disclosure.

Through the above technical solution, the tone labeling information of the target text to be processed is determined, wherein the tone labeling information includes the tone change type of each text unit in the target text, wherein the text unit is composed of at least one unit text, and the continuous reading of the text unit The pitch type is used to indicate the pitch change trend of the text unit, determine the prosody annotation information of the target text and the phoneme sequence corresponding to the target text, and generate synthetic audio corresponding to the target text according to the tone annotation information, prosodic annotation information and phoneme sequence. As a result, in addition to using prosody features in speech synthesis for the target text, the link-to-tone shift feature is further introduced, so that the link-to-tone shift mode can be directly controlled during speech synthesis, which improves the accuracy of speech synthesis in speech synthesis. The controllability of the tone-shifting phenomenon of continuous reading, and then the naturalness of the synthesized speech.

Other features and advantages of the present disclosure will be described in detail in the detailed description that follows.

Description of drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the attached image:

1 is a flowchart of a speech synthesis method provided according to an embodiment of the present disclosure;

2 is a block diagram of a speech synthesis apparatus provided according to an embodiment of the present disclosure;

FIG. 3 shows a schematic structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for the purpose of A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this regard.

As used herein, the term "including" and variations thereof are open-ended inclusions, ie, "including but not limited to". The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "a" and "a plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, they should be understood as "one or a plurality of". multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of these messages or information.

All actions of obtaining signals, information or data in this disclosure are carried out under the premise of complying with the corresponding data protection laws and policies of the country where they are located and authorized by the corresponding device owner.

It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the user should be informed of the type, scope of use, and use scenario of the personal information involved in the present disclosure in an appropriate manner in accordance with relevant laws and regulations, and the user's authorization should be obtained. .

For example, in response to receiving an active request from the user, prompt information is sent to the user to explicitly prompt the user that the requested operation will require the acquisition and use of the user's personal information. Therefore, the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers, or storage media that perform operations of the technical solutions of the present disclosure according to the prompt information.

As an optional but non-limiting implementation manner, in response to receiving the user's active request, the method of sending prompt information to the user may be, for example, a pop-up window, and the prompt information may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

It can be understood that the above process of notifying and obtaining user authorization is only illustrative, and does not limit the implementation of the present disclosure, and other methods that satisfy relevant laws and regulations can also be applied to the implementation of the present disclosure.

At the same time, it can be understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the corresponding laws and regulations and relevant regulations.

FIG. 1 is a flowchart of a speech synthesis method provided according to an embodiment of the present disclosure. As shown in FIG. 1 , the method provided by the present disclosure may include steps 11 to 13 .

In step 11, the tone annotation information of the target text to be processed is determined.

In the present disclosure, the tone annotation information includes the tone transition type of each text unit in the target text, wherein the text unit is composed of at least one unit text, and the tone transition type of the text unit is used to indicate the pitch change trend of the text unit .

Among them, the unit text is the smallest unit that constitutes the target text. For example, if the target text is Chinese text, the unit text is a single character, that is, a single syllable, and accordingly, the text unit is composed of at least one character.

The transposition type of text unit can refer to the following definitions:

If the text unit consists of one unit text, the tone transition type of the text unit is one of the following: the first type used to indicate that the pitch is the original tone of the text unit, the first type used to indicate that the pitch is the original tone of the text unit The pitch onset of the tone changes to the second type of flat;

If a text unit is composed of two unit texts, the text unit's continuation transposition type is one of the following: the third type used to indicate that the pitch is lowered from high flat to low flat, the third type used to indicate pitch from high flat to low flat Type 4 for pitch down to mid-flat, type 5 for indicating pitch from mid-flat to high-flat, sixth type for indicating pitch from low-flat to high-flat, use Type 7 for indicating pitch rising from low flat to low rising, and Type 8 for indicating pitch rising from low flat to medium flat;

If the text unit consists of more than two unit texts, the text unit's continuation shift type is one of the following: the third type, the fourth type, the seventh type, the eighth type, which is used to indicate the pitch from the middle The ninth type of flat pitch raised to a high flat key and then lowered to a low flat key, the tenth type used to indicate pitch from medium flat to high flat and then lowered to medium flat, used to indicate pitch from low The eleventh type, where the pitch is raised to high flat and then lowered to low flat, and the twelfth type, which is used to indicate that the pitch is raised from low flat to high flat and then lowered to medium flat.

By way of example, the type of tone shift and its pitch change trend of text units can refer to the following table:

Among them, H stands for high level, M for medium level, L for low level, R for low rise, and the symbol "-" is used to separate two adjacent unit texts. Taking the target text as Chinese text as an example, the unit text is a single character, that is, a single syllable. Correspondingly, H-L represents a two-syllable tone, its first syllable is a high-level tone, the second syllable is a low-level tone, and the overall The pitch curve changes from high flat to low flat.

In the above table, [M]{n} means that n M are connected, that is, n are connected horizontally. For example, M{2} is M-M, M{3} is M-M-M, and the rest are the same.

It should be noted that, taking the third type as an example, if the text unit is composed of 4 unit texts, the pitch change trend is H-M-M-L, and the pitch can be gradually reduced from high-level to low-level, that is to say, Although the two in the middle are all mid-flat, the pitch of the former can be slightly higher than that of the latter.

In step 12, the prosodic annotation information of the target text and the phoneme sequence corresponding to the target text are determined.

In the present disclosure, the phoneme sequence corresponding to the text to be synthesized may be obtained through a Grapheme-to-Phoneme (G2P) model.

For example, the G2P model can use a Recurrent Neural Network (RNN) and a Long Short-Term Memory (LSTM) to realize the transformation from grapheme to phoneme.

The prosodic annotation information is used to reflect content related to prosody, and it may include but not limited to prosodic boundary information. Among them, the prosodic boundary (break index, which can be abbreviated as BRK), which can also be called a break index, is used to describe the form of information organization and clauses in the speech flow. Optionally, the prosodic boundary information may include, but is not limited to, sentence boundaries, intonation phrase boundaries, prosodic phrase boundaries, and prosodic word boundaries.

In step 13, a synthetic audio corresponding to the target text is generated according to the tone annotation information, the prosody annotation information and the phoneme sequence.

Through the above technical solution, the tone labeling information of the target text to be processed is determined, wherein the tone labeling information includes the tone change type of each text unit in the target text, wherein the text unit is composed of at least one unit text, and the continuous reading of the text unit The pitch type is used to indicate the pitch change trend of the text unit, determine the prosody annotation information of the target text and the phoneme sequence corresponding to the target text, and generate synthetic audio corresponding to the target text according to the tone annotation information, prosodic annotation information and phoneme sequence. As a result, in addition to using prosody features in speech synthesis for the target text, the link-to-tone shift feature is further introduced, so that the link-to-tone shift mode can be directly controlled during speech synthesis, which improves the accuracy of speech synthesis in speech synthesis. The controllability of the tone-shifting phenomenon of continuous reading, and then the naturalness of the synthesized speech.

In order to make those skilled in the art better understand the speech synthesis method provided by the present disclosure, the above steps are described in detail below.

First, the related content of the tone annotation information used in the present disclosure will be explained.

As above, the tone annotation information includes the tone transition type of each text unit in the target text, wherein the text unit is composed of at least one unit text, and the tone transition type of the text unit is used to indicate the pitch change trend of the text unit. The manner of determining the tone annotation information of the target text will be described in detail below.

In a possible implementation manner, the tone annotation information of the target text may be determined by manual annotation. That is to say, the tone annotation operation for the target text can be received, and the tone annotation information of the target text can be generated according to the tone annotation operation.

That is to say, the annotator can directly annotate the tone of the target text, and annotate the tone-translation features expected to be heard from the synthesized audio into the target text. Among them, the labeling operation for tones can be performed in two steps.

In the first step, text units may be divided based on the content of the target text, that is, the target text may be divided into multiple text units by generating boundaries.

In the second step, each text unit is marked with its continuation transposition type.

In another possible implementation manner, the determination of the tone annotation information of the target text may be realized by a pre-trained tone annotation model. Correspondingly, the tone annotation information of the target text can be obtained in the following ways:

Input the target text into the tone annotation model, and obtain the output result of the tone annotation model;

According to the output result, the tone annotation information of the target text is determined.

The tone labeling model is obtained by training based on the second training text with tone labeling information, and the output result includes boundary information for dividing the target text into multiple text units and the corresponding tone transition type for each text unit.

The second training text may be text extracted from real speech. For such speech, the annotator can mark appropriate positions in the text by listening to the speech to obtain the tone annotation information of the second training text. Such annotation mainly depends on the auditor's sense of hearing. For the labeling idea, please refer to the description of the type of tone shift in the previous section.

In a possible implementation, the above output result can be directly used as the tone annotation information of the target text.

In another possible implementation, determining the tone annotation information of the target text according to the output result, which may include the following steps:

determine whether a correction instruction for the output result has been received;

According to the correction instruction, the output result is corrected, and the corrected output result is determined as the tone annotation information of the target text.

Wherein, the modification instruction is used to instruct to modify at least one of the boundary information in the output result and the type of tone shift of continuous reading.

That is to say, the annotator can also correct the output result of the model through the correction instruction, so as to improve the efficiency of determining the tone annotation information, and also ensure the high accuracy of the annotation.

In the above manner, the tone annotation information corresponding to the real speech of the second training text can be obtained. Therefore, the second training text is used as the input of the neural network model, and the tone annotation information corresponding to the second training text is used as the target output of the model, and the neural network model is trained. Tone annotation model for tone annotation information. In this way, inputting a piece of text into the tone labeling model can automatically obtain the tone labeling information output by the tone labeling model corresponding to the piece of text, and no manual labeling is required, which is beneficial to improve the efficiency of determining the tone labeling information.

Returning to FIG. 1 , in step 13, according to the tone annotation information, the prosodic annotation information and the phoneme sequence, a synthesized audio corresponding to the target text is generated.

In a possible implementation, step 13 may include the following steps:

According to the tone label information, determine the sequence of consecutive tone-shift labels corresponding to the target text;

Determine the prosodic label sequence corresponding to the target text according to the prosodic labeling information;

Generate acoustic feature information corresponding to the target text by using the pre-trained speech synthesis model according to the sequence of transposition tags, prosodic tags and phoneme sequences;

Speech synthesis is performed on the acoustic feature information using a vocoder to generate synthesized audio corresponding to the target text.

Wherein, the idea of determining the link-shifting label is that the unit texts of the same text unit share the same linking-shifting label, that is, the linking-shifting label of each unit text constituting a text unit is consistent with the linking-shifting label of the text unit . Furthermore, according to the order of appearance of the text units in the target text, the sequence of read-through transposition tags can be obtained.

As before, the prosodic annotation information may include prosodic boundary information, and correspondingly, the prosodic label may include a prosodic boundary label.

The prosodic annotation information generally marks a certain position in the text, for example, a certain position in the text is a prosodic phrase boundary. In order to facilitate subsequent speech synthesis and ensure that the prosodic labeling information can correspond to the phonemes of the text to be synthesized one by one, the prosodic label at the phoneme level may be further determined based on the prosodic labeling information.

The idea of determining the prosodic label is to generate label content according to the label information for phoneme positions where prosodic label information exists, and to substitute specified alternative content for phoneme locations without prosodic label information. For example, for the phoneme sequence {A1, A2, A3, A4, A5, A6}, it is assumed that the prosodic annotation information includes prosodic boundary information, and the annotation content is that there is a prosodic phrase boundary at A2 and an intonation phrase boundary at A5, and it is specified that If the prosodic phrase boundary is represented by 3, the intonation phrase boundary is represented by 4, and the unmarked phrase is represented by N2, the determined prosodic boundary label is {N2, 3, N2, N2, 4, N2}.

When generating the acoustic feature information corresponding to the target text, the continuous tone-shifting label sequence, the prosodic label sequence and the phoneme sequence can be input into the pre-trained speech synthesis model to obtain the acoustic feature information corresponding to the target text. For example, the acoustic feature information may be a Mel spectrum (Mel spectrum), a linear spectrum, or the like.

The above-mentioned speech synthesis model may include an encoding network, an attention network, and a decoding network. Among them, the encoding network is used to generate the text representation sequence according to the splicing vector corresponding to the sequence of read-linking transposition tags, the prosodic tag sequence and the phoneme sequence; the attention network is used to generate the semantic representation according to the text representation sequence; the decoding network is used to generate the semantic representation according to the semantic Characterization, output the acoustic feature information corresponding to the target text.

The input of the encoding network (Encoder) of the speech synthesis model is the vector representation of the target text, which can include the first vector obtained after the phoneme sequence is vectorized (embedding), the second vector obtained after the sequence of read-through tone-shifting labels is vectorized, and The prosodic label sequence is the third vector after vectorization, and the above-mentioned ones are concatenated to form a concatenated vector, which is used as the input of the encoding network. After that, the encoding network corresponds to the text representation sequence (TE, textembedding) of the output target text. The text representation sequence output by the encoding network passes through the attention network to generate a context vector C, which is used as the semantic representation of the target text. The semantic representation generated by the attention network enters the decoding network, which outputs the acoustic feature information corresponding to the target text.

Illustratively, a speech synthesis model is trained by:

Obtaining a first training sample, wherein each first training sample includes a training phoneme sequence corresponding to the first training text, a training sequence of tone-shifting labels, a sequence of training prosody labels, and training acoustic feature information corresponding to the first training text;

The model is trained by taking the splicing vector corresponding to the training phoneme sequence, the training read-connection transposition label sequence and the training prosodic label sequence as the input of the model, and the training acoustic feature information as the target output of the model, so as to obtain the trained speech synthetic model.

Illustratively, the above-mentioned speech synthesis model may use the Tacotron model.

The first training text corresponds to audio, and the acoustic feature information of the audio is determined as the training acoustic feature information.

In the present disclosure, the training phoneme sequence of the first training text can be determined in a manner similar to that of determining the phoneme sequence of the target text in step 12, and the continuous tone-shifting label sequence and the prosodic label sequence of the target text can be determined by the above In a similar manner, the training read-through pitch-shifting label sequence and the training prosodic label sequence of the first training text are determined. The above content will not be repeated here.

The training purpose of the speech synthesis model is to make the audio synthesized by the model output infinitely close to the actual audio of the first training sample, that is, to make the acoustic feature information output by the model infinitely close to the training acoustic feature information. Therefore, the loss value of the model can be calculated based on the training acoustic feature information and the acoustic feature information output by the model during training, and the internal parameters of the current model can be adjusted by using the loss value. After that, the adjusted model is used in the next training, and so on and so forth, until the conditions for stopping training are met, and the trained speech synthesis model can be obtained.

The trained speech synthesis model obtained through the above training steps can be used in speech synthesis scenarios. which is:

Generate acoustic feature information corresponding to the target text by using the pre-trained speech synthesis model according to the sequence of transposition tags, prosodic tags and phoneme sequences;

Speech synthesis is performed on the acoustic feature information using a vocoder to generate synthesized audio corresponding to the target text.

After obtaining the acoustic feature information of the target text, the acoustic feature information can be input into a vocoder (eg, Wavenet vocoder, Griffin-Lim vocoder) for speech synthesis, so as to obtain the corresponding text to be synthesized. Synthesized audio.

FIG. 2 is a block diagram of a speech synthesis apparatus provided according to an embodiment of the present disclosure. As shown in Figure 2, the device 20 includes:

The first determination module 21 is configured to determine the tone annotation information of the target text to be processed, wherein the tone annotation information includes the tone-translation type of each text unit in the target text, wherein the text unit is composed of at least one The unit text is composed, and the tone change type of the text unit is used to indicate the pitch change trend of the text unit;

The second determination module 22 is used to determine the prosodic labeling information of the target text and the phoneme sequence corresponding to the target text;

The generating module 23 is configured to generate synthetic audio corresponding to the target text according to the tone labeling information, the prosody labeling information and the phoneme sequence.

Optionally, the generating module 23 includes:

a first determination submodule, configured to determine a sequence of consecutive tone-shifting labels corresponding to the target text according to the tone labeling information;

a second determination submodule, configured to determine the prosodic label sequence corresponding to the target text according to the prosodic labeling information;

The first generation sub-module is used to generate acoustic feature information corresponding to the target text by using a pre-trained speech synthesis model according to the read-through tone-changing label sequence, the prosodic label sequence and the phoneme sequence;

The synthesis sub-module is configured to use a vocoder to perform speech synthesis on the acoustic feature information to generate synthesized audio corresponding to the target text.

Optionally, the speech synthesis model includes an encoding network, an attention network and a decoding network; wherein:

The encoding network is configured to generate a text representation sequence according to the splicing vector corresponding to the read-through pitch tag sequence, the prosodic tag sequence and the phoneme sequence;

The attention network is used to generate semantic representations according to the text representation sequence;

The decoding network is configured to output acoustic feature information corresponding to the target text according to the semantic representation.

Optionally, the speech synthesis model is obtained by the following modules:

an acquisition module, configured to acquire a first training sample, wherein each of the first training samples includes a training phoneme sequence corresponding to the first training text, a training tone-shifting label sequence, a training prosody label sequence, and the first training The training acoustic feature information corresponding to the text;

A training module is used to use the splicing vector corresponding to the training phoneme sequence, the training read-through pitch tag sequence and the training prosody tag sequence as the input of the model, and the training acoustic feature information as the target of the model Model training is performed in the manner of output, so as to obtain the speech synthesis model that has been trained.

Optionally, the first determining module 21 includes:

A processing submodule, configured to input the target text into the tone annotation model, and obtain the output result of the tone annotation model, wherein the tone annotation model is obtained by training based on the second training text with tone annotation information, and the tone annotation model is obtained by training. The output result includes boundary information for dividing the target text into a plurality of text units and the respective corresponding link-translation type of each text unit;

The third determination sub-module is configured to determine the tone annotation information of the target text according to the output result.

Optionally, the third determination submodule includes:

a fourth determination sub-module, configured to determine whether to receive a correction instruction for the output result, where the correction instruction is used to instruct to modify at least one of the boundary information in the output result, and the tone change type of continuous reading;

A modification sub-module, configured to modify the output result according to the modification instruction, and determine the modified output result as the tone annotation information of the target text.

Optionally, if the text unit is constituted by a unit text, the tone transition type of the text unit is one of the following: the first type used to indicate that the pitch is the original tone of the text unit, the first type used to indicate that the pitch is The second type of pitch onset change of the original tone of the text unit to a flat tone;

If a text unit is composed of two unit texts, the text unit's continuation transposition type is one of the following: the third type used to indicate that the pitch is lowered from high flat to low flat, the third type used to indicate pitch from high flat to low flat Type 4 for pitch down to mid-flat, type 5 for indicating pitch from mid-flat to high-flat, sixth type for indicating pitch from low-flat to high-flat, use Type 7 for indicating pitch rising from low flat to low rising, and Type 8 for indicating pitch rising from low flat to medium flat;

If the text unit is composed of more than two unit texts, the text unit's transposition type is one of the following: the third type, the fourth type, the seventh type, the eighth type , the ninth type used to indicate that the pitch is raised from mid-flat to high-flat and then lowered to low-flat, and the tenth type used to indicate that the pitch is raised from mid-flat to high-flat and then lowered to mid-flat , the eleventh type used to indicate that the pitch is raised from low flat to high flat and then lowered to low flat, the tenth type used to indicate pitch raised from low flat to high flat and then lowered to medium flat Two types.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

3, which shows a schematic structural diagram of an electronic device (600) suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include, but is not limited to, such as mobile phones, notebook computers, digital broadcast receivers PDA (Personal Digital Assistant), PAD (Tablet Computer), PMP (Portable Multimedia Player), mobile terminals such as in-vehicle terminals (such as in-vehicle navigation terminals), etc., and stationary terminals such as digital TVs, desktop computers, etc. Fig. The electronic device shown in 3 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 3 , electronic device 600 may include processing means (eg, central processing unit, graphics processor, etc.) 601 that may be loaded into random access according to a program stored in read only memory (ROM) 602 or from storage means 608 Various appropriate actions and processes are executed by the programs in the memory (RAM) 603 . In the RAM 603, various programs and data necessary for the operation of the electronic device 600 are also stored. The processing device 601 , the ROM 602 , and the RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

Typically, the following devices can be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 607 of a computer, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. Communication means 609 may allow electronic device 600 to communicate wirelessly or by wire with other devices to exchange data. Although FIG. 3 shows electronic device 600 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication device 609 , or from the storage device 608 , or from the ROM 602 . When the computer program is executed by the processing apparatus 601, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the client and server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium (eg, a communications network) interconnected. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks), as well as any currently known or future development network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: determines the tone annotation information of the target text to be processed, wherein the tone annotation information Including the tone transition type of each text unit in the target text, wherein the text unit is composed of at least one unit text, and the tone transition type of the text unit is used to indicate the pitch change trend of the text unit; determine the The prosodic labeling information of the target text and the phoneme sequence corresponding to the target text; according to the tone labeling information, the prosodic labeling information and the phoneme sequence, a synthetic audio corresponding to the target text is generated.

Computer program code for performing operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and This includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider to via Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The modules involved in the embodiments of the present disclosure may be implemented in software or hardware. Wherein, the name of the module does not constitute a limitation of the module itself under certain circumstances, for example, the first acquisition module may also be described as "a module for acquiring at least two Internet Protocol addresses".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logical Devices (CPLDs) and more.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, there is provided a speech synthesis method, the method comprising:

Determine the tone annotation information of the target text to be processed, wherein the tone annotation information includes the tone change type of each text unit in the target text, wherein the text unit is composed of at least one unit text, and the continuous tone of the text unit is The reading pitch type is used to indicate the pitch change trend of the text unit;

Determine the prosodic annotation information of the target text and the phoneme sequence corresponding to the target text;

According to the tone annotation information, the prosodic annotation information and the phoneme sequence, a synthesized audio corresponding to the target text is generated.

According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein the synthesized audio corresponding to the target text is generated according to the tone annotation information, the prosodic annotation information and the phoneme sequence, include:

According to the tone labeling information, determine the sequence of consecutive tone-shifting labels corresponding to the target text;

determining the prosodic label sequence corresponding to the target text according to the prosodic labeling information;

Generate acoustic feature information corresponding to the target text by using a pre-trained speech synthesis model according to the read-through tone-shifting label sequence, the prosodic label sequence and the phoneme sequence;

Speech synthesis is performed on the acoustic feature information using a vocoder to generate synthesized audio corresponding to the target text.

According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein the speech synthesis model includes an encoding network, an attention network and a decoding network; wherein:

The encoding network is configured to generate a text representation sequence according to the splicing vector corresponding to the read-through pitch tag sequence, the prosodic tag sequence and the phoneme sequence;

The attention network is used to generate semantic representations according to the text representation sequence;

The decoding network is configured to output acoustic feature information corresponding to the target text according to the semantic representation.

According to one or more embodiments of the present disclosure, a speech synthesis method is provided, and the speech synthesis model is obtained by:

Acquiring a first training sample, wherein each of the first training samples includes a training phoneme sequence corresponding to the first training text, a training tone-shifting label sequence, a training prosody label sequence, and a training acoustic corresponding to the first training text characteristic information;

The model is carried out by using the splicing vector corresponding to the training phoneme sequence, the training read-through pitch tag sequence and the training prosody tag sequence as the input of the model, and the training acoustic feature information as the target output of the model training to obtain the speech synthesis model that has been trained.

According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein the determining the tone annotation information of the target text to be processed includes:

The target text is input into the tone annotation model, and an output result of the tone annotation model is obtained, wherein the tone annotation model is obtained by training based on the second training text with tone annotation information, and the output result includes a method for adding tone annotation information. Described target text is divided into the boundary information of a plurality of text units and each text unit is respectively corresponding to the contiguous tone change type;

According to the output result, the tone annotation information of the target text is determined.

According to one or more embodiments of the present disclosure, a speech synthesis method is provided, wherein the determining tone annotation information of the target text according to the output result includes:

determining whether a correction instruction for the output result is received, the correction instruction being used to instruct to change at least one of the boundary information in the output result, and the type of tone change;

According to the correction instruction, the output result is corrected, and the corrected output result is determined as the tone annotation information of the target text.

According to one or more embodiments of the present disclosure, a speech synthesis method is provided. If a text unit is composed of a unit text, the link-to-speech change type of the text unit is one of the following: used to indicate that the pitch is text a first type of the original tone of the unit, a second type for indicating a pitch change from the pitch origin of the original tone of the text unit to a flat tone;

If a text unit is composed of two unit texts, the text unit's continuation transposition type is one of the following: the third type used to indicate that the pitch is lowered from high flat to low flat, the third type used to indicate pitch from high flat to low flat Type 4 for pitch down to mid-flat, type 5 for indicating pitch from mid-flat to high-flat, sixth type for indicating pitch from low-flat to high-flat, use Type 7 for indicating pitch rising from low flat to low rising, and Type 8 for indicating pitch rising from low flat to medium flat;

If the text unit is composed of more than two unit texts, the text unit's transposition type is one of the following: the third type, the fourth type, the seventh type, the eighth type , the ninth type used to indicate that the pitch is raised from mid-flat to high-flat and then lowered to low-flat, and the tenth type used to indicate that the pitch is raised from mid-flat to high-flat and then lowered to mid-flat , the eleventh type used to indicate that the pitch is raised from low flat to high flat and then lowered to low flat, the tenth type used to indicate pitch raised from low flat to high flat and then lowered to medium flat Two types.

According to one or more embodiments of the present disclosure, there is provided a speech synthesis apparatus, the apparatus comprising:

The first determination module is used to determine the tone annotation information of the target text to be processed, wherein the tone annotation information includes the tone change type of each text unit in the target text, wherein the text unit is composed of at least one unit. Text composition, the type of tone change of the text unit is used to indicate the pitch change trend of the text unit;

a second determination module, configured to determine the prosodic annotation information of the target text and the phoneme sequence corresponding to the target text;

A generating module, configured to generate synthetic audio corresponding to the target text according to the tone annotation information, the prosody annotation information and the phoneme sequence.

According to one or more embodiments of the present disclosure, there is provided a computer-readable medium on which a computer program is stored, and when the program is executed by a processing apparatus, implements the steps of the speech synthesis method provided by any embodiment of the present disclosure.

According to one or more embodiments of the present disclosure, there is provided an electronic device, comprising:

a storage device having at least one computer program stored thereon;

At least one processing device is configured to execute the at least one computer program in the storage device to implement the steps of the speech synthesis method provided by any embodiment of the present disclosure.

The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, and should also cover, without departing from the above-mentioned disclosed concept, the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of its equivalent features. For example, a technical solution is formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) with similar functions.

Additionally, although operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, although the above discussion contains several implementation-specific details, these should not be construed as limitations on the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or logical acts of method, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

Claims (10)

1. A method of speech synthesis, the method comprising:

determining tone marking information of a target text to be processed, wherein the tone marking information comprises continuous reading tone variation types of text units in the target text, the text units are composed of at least one unit text, and the continuous reading tone variation types of the text units are used for indicating pitch variation trends of the text units;

determining prosody labeling information of the target text and a phoneme sequence corresponding to the target text;

and generating synthetic audio corresponding to the target text according to the tone marking information, the rhythm marking information and the phoneme sequence.

2. The method of claim 1, wherein generating synthetic audio corresponding to the target text according to the tone labeling information, the prosody labeling information, and the phoneme sequence comprises:

determining a continuous reading tonal modification label sequence corresponding to the target text according to the tone marking information;

determining a prosody tag sequence corresponding to the target text according to the prosody labeling information;

generating acoustic feature information corresponding to the target text by utilizing a pre-trained speech synthesis model according to the continuous reading tonal modification label sequence, the prosodic label sequence and the phoneme sequence;

and performing voice synthesis on the acoustic feature information by using a vocoder to generate synthetic audio corresponding to the target text.

3. The method of claim 2, wherein the speech synthesis model comprises an encoding network, an attention network, and a decoding network; wherein:

the coding network is used for generating a text representation sequence according to the splicing vectors corresponding to the continuous reading tonal modification label sequence, the prosodic label sequence and the phoneme sequence;

the attention network is used for generating semantic representation according to the text representation sequence;

and the decoding network is used for outputting acoustic characteristic information corresponding to the target text according to the semantic representation.

4. The method of claim 2, wherein the speech synthesis model is obtained by:

acquiring first training samples, wherein each first training sample comprises a training phoneme sequence, a training continuous reading tone-changing label sequence and a training prosody label sequence which correspond to a first training text, and training acoustic feature information which corresponds to the first training text;

and performing model training by taking the splicing vectors corresponding to the training phoneme sequence, the training continuous reading tone-changing label sequence and the training prosody label sequence as the input of a model and taking the training acoustic characteristic information as the target output of the model to obtain the trained speech synthesis model.

5. The method according to claim 1, wherein the determining tonal marking information of the target text to be processed comprises:

inputting the target text into a tone marking model, and obtaining an output result of the tone marking model, wherein the tone marking model is obtained by training based on a second training text with tone marking information, and the output result comprises boundary information for dividing the target text into a plurality of text units and a continuous reading tone-changing type corresponding to each text unit;

and determining the tone marking information of the target text according to the output result.

6. The method according to claim 5, wherein the determining the tone labeling information of the target text according to the output result comprises:

determining whether a revision instruction for the output result is received, the revision instruction being indicative of altering at least one of boundary information, a type of read-through transpose in the output result;

and correcting the output result according to the correction instruction, and determining the corrected output result as the tone marking information of the target text.

7. The method according to any one of claims 1 to 6,

if the text unit is composed of a unit text, the continuous reading tonal modification type of the text unit is one of the following types: a first type for indicating that a pitch is an original tone of the text unit, a second type for indicating that the pitch is changed from a pitch start point of the original tone of the text unit to a flat tone;

if a text unit consists of two units of text, the type of the continuous reading tone of the text unit is one of the following: a third type for indicating a decrease in pitch from high to low, a fourth type for indicating a decrease in pitch from high to medium, a fifth type for indicating an increase in pitch from medium to high, a sixth type for indicating an increase in pitch from low to high, a seventh type for indicating an increase in pitch from low to low, and an eighth type for indicating an increase in pitch from low to medium;

if a text unit is composed of more than two units of text, the type of the continuous reading key of the text unit is one of the following: the third type, the fourth type, the seventh type, the eighth type, a ninth type for indicating that the pitch is raised from a middle level to a high level and then lowered to a low level, a tenth type for indicating that the pitch is raised from a middle level to a high level and then lowered to a middle level, an eleventh type for indicating that the pitch is raised from a low level to a high level and then lowered to a low level, and a twelfth type for indicating that the pitch is raised from a low level to a high level and then lowered to a middle level.

8. A speech synthesis apparatus, characterized in that the apparatus comprises:

the system comprises a first determining module, a second determining module and a processing module, wherein the first determining module is used for determining tone marking information of a target text to be processed, the tone marking information comprises continuous reading tone variation types of text units in the target text, the text units are composed of at least one unit text, and the continuous reading tone variation types of the text units are used for indicating pitch variation trends of the text units;

the second determining module is used for determining prosody labeling information of the target text and a phoneme sequence corresponding to the target text;

and the generating module is used for generating synthetic audio corresponding to the target text according to the tone marking information, the rhythm marking information and the phoneme sequence.

9. A computer-readable medium, on which a computer program is stored, characterized in that the program, when being executed by processing means, carries out the steps of the method of any one of claims 1 to 7.

10. An electronic device, comprising:

a storage device having at least one computer program stored thereon;

at least one processing device for executing the at least one computer program in the storage device to carry out the steps of the method according to any one of claims 1 to 7.

Images