JP2023038957A - Voice synthesis system and method for synthesizing voice - Google Patents

Abstract

[Problem] Emotional speech synthesis allows for easily obtaining speech that mimics human emotions. [Solution] The system stores a speech synthesis model that generates acoustic features of text based on emotional parameters and linguistic features of the text, and a conversion model that associates speech conditions with statistics of emotional parameters of a target speech sample. The target speech sample is a speech sample of a target speaker whose emotion is to be mimicked. The system acquires input current speech conditions. The system generates current emotional parameters from the current speech conditions using the conversion model. The system generates acoustic features of the target utterance using a speech synthesis model based on the current emotional parameters and linguistic features of the target utterance. The system generates synthetic speech based on the acoustic features. [Selected Figure] Figure 1A

Description

The present invention relates to speech synthesis technology.

Among the human interfaces in which humans interact with machines, there are dialogue systems that allow users to have voice conversations with machines. The use of the dialogue system is not limited to accepting people's talks, but it can also be considered to support people's lives by appropriately working with them from the dialogue system.

In order for the dialogue system to appropriately work on humans through conversation, it is important not only what the dialogue system says but also what kind of emotions are expressed in the voice. When a person asks someone to do something, he/she speaks to the other person while appropriately controlling the emotions put into the voice. In a dialogue system as well, it is desirable to add emotion to such speech.

Emotional speech synthesis technology has existed so far, which generates emotions by putting emotions into voice. This technology synthesizes emotional speech according to a specified multidimensional vector representing emotion. The input used for specifying the emotion is called an emotion parameter.

In order for the dialogue system to work more appropriately with speech, it is necessary to investigate how people who are doing tasks similar to the use of the dialogue system speak and generate speech that imitates the emotion of the spoken speech. It is considered effective.

As a technique for realizing this method, Patent Document 1 discloses a technique for recognizing an emotion in an input voice from a microphone as one category and specifying that category as the emotion of the output voice of speech synthesis. there is As a result, by preparing a voice sample uttered by a human being, it is possible to generate a voice that imitates the emotion of the uttered voice.

Japanese Patent Application No. 2006-113546

In the conventional emotion speech synthesis technology, in order to determine the emotion, the developer had to generate an emotion voice, listen to it, and adjust the emotion parameters until the desired voice was obtained. There was a problem that it took a long time for the developer to make this adjustment by trial and error.

Alternatively, if you want to generate speech with emotions that mimic humans, you can recognize one emotion category in the input speech and designate that category as the emotion in the output speech for speech synthesis. However, since human beings express various emotions in speech, it is difficult to adequately express human emotions with a configuration in which emotions are selected from a small number of categories.

In addition, when a plurality of people speak in the same situation, the changes in the speech are diverse, and the emotions are also diverse. When thinking about using a machine to replace the actions that humans have performed with their voices, it is necessary to synthesize speech while taking into account such variations for each speaker. However, methods that determine emotion based on a single voice cannot meet this need.

A speech synthesis system according to one aspect of the present invention includes one or more arithmetic devices and one or more storage devices. The one or more storage devices include: a speech synthesis model that generates acoustic features of the text based on the emotional parameters and the linguistic features of the text; , to store Said target speech sample is a speech sample of a target speaker whose emotions are to be mimicked. The one or more arithmetic devices acquire the input current utterance conditions, generate current emotion parameters from the current utterance conditions using the conversion model, and generate the current emotion parameters and linguistic feature amounts of the target utterance. The speech synthesis model is used to generate acoustic features of the target utterance, and synthetic speech is generated based on the acoustic features.

By using one aspect of the present invention, it is possible to perform speech synthesis that more appropriately reproduces the emotions of speech spoken by humans.

1 illustrates an example logical configuration of a speech synthesis system according to an embodiment of the present specification; 4 shows another logical configuration example of a speech synthesis system according to an embodiment of the present specification; An example of the configuration of a speech synthesis corpus is shown. An example of a target speech database is shown. 4 shows a configuration example of a target speech statistic database. A configuration example of a condition-class conversion model is shown. 1 is a block diagram schematically showing a hardware configuration example of a speech synthesis system; FIG. 4 is a flow chart of an example of a learning procedure for a speech synthesis model; 4 shows a flow chart of an example of processing for generating a model that associates conditions and emotional parameters from a target speech database. 4 is a flow chart illustrating an example of clustering-based modeling; 10 is a flow chart showing an example of mean value-based modeling; 4 is a flowchart illustrating an example of machine learning-based modeling; 10 is a flowchart illustrating an example method of speech synthesis; 4 schematically shows a logical configuration example of a speech synthesis system according to a second embodiment; FIG. 10 is a diagram illustrating a procedure for obtaining statistics from emotion parameters obtained from a machine learning model; FIG. 11 shows a logical configuration example of a speech synthesis system according to a third embodiment; FIG. 4 shows a configuration example of an output speech statistic database; . FIG. 11 is a flowchart showing an example of conversion parameter update processing for emotion parameters; FIG. FIG. 11 is a diagram for explaining a function of adjusting emotion parameters for speech synthesis by a developer in the fourth embodiment; FIG. 11 is a diagram for explaining a function of adjusting emotion parameters for speech synthesis by a developer in the fourth embodiment; FIG. 11 is a diagram for explaining a function of adjusting emotion parameters for speech synthesis by a developer in the fourth embodiment; FIG. 16 is a flow chart showing an example of a method for adjusting conversion parameters of emotion parameters in the fifth embodiment; FIG.

Several embodiments of the present invention will be described below with reference to the drawings. For convenience, the following embodiments are divided into a plurality of sections or embodiments when necessary, but unless otherwise specified, they are not unrelated to each other, and one Some or all of them are related to modifications, details, supplementary explanations, and the like. Each embodiment may be implemented individually, or may be implemented in combination.

In addition, in the following embodiments, when referring to the number of elements (including the number, numerical value, amount, range, etc.), unless otherwise specified or clearly limited to a specific number in principle , is not limited to the specific number, and may be greater than or less than the specific number.

Furthermore, in the following embodiments, its constituent elements (including element steps, etc.) are, of course, not necessarily essential unless otherwise specified or clearly considered essential in principle. stomach.

Similarly, in the following embodiments, when referring to the shape, positional relationship, etc., of components, etc., unless otherwise specified or in principle clearly considered otherwise, the shape, etc. shall include those that are similar or similar to This also applies to numerical values and ranges.

Also, in the following description, the expression "xxx table" may be used, but the information may be data of any structure. Also, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table. good. Also, in the following description, the function may be described using the expression "xxx part", but the function may be realized by executing one or more computer programs by one or more arithmetic units. .

A speech synthesis system according to an embodiment of the present specification described below synthesizes and outputs human voice including emotion. This makes it possible to work on people more appropriately and support their activities more appropriately.

A variety of emotions put into speech by humans can be represented, for example, by multidimensional vectors. Each element of the multidimensional vector corresponds to, for example, Arousal (activity), Pleasantness (emotional valence), and Dominance (dominance). Multi-dimensional vectors can accommodate many emotional variations.

In addition, when a plurality of people speak in the same situation, the changes in the speech are diverse, and the emotions are also diverse. When thinking about using a machine to replace the vocal approach that humans have performed, synthesis should take into account such variations for each speaker.

Moreover, even if the emotion parameter output by the emotion recognizer is directly specified to the speech synthesizer, the emotion of the generated speech is not always what the developer desires. This is because the speaker who is the sound source for speech synthesis and the speaker who is the target to imitate the emotion may be different. This difference in speakers may make speech by speech synthesis unnatural.

A speech synthesis system according to one embodiment herein includes a speech synthesis model and a speech synthesizer. The speech synthesis model receives linguistic features and emotion parameters as inputs and outputs acoustic features. A linguistic feature is a feature obtained from a text that indicates the content of speech. The emotion parameter is a parameter that indicates the emotion to be put on the voice. The acoustic feature quantity is the feature quantity of speech (sound). A speech synthesizer generates synthesized speech from the acoustic features from the speech synthesis model.

A speech synthesis system according to an embodiment of the present specification uses an emotion recognizer to obtain language features and emotion parameters from speech data in a speech synthesis model training corpus (speech synthesis corpus). As described above, the speech synthesis system uses emotion parameters as inputs in addition to linguistic features in learning a speech synthesis model.

The speech synthesis model outputs acoustic features from input language features and emotion parameters. The parameters of the speech synthesis model are updated (learning or training of the speech synthesis model) based on the error between the acoustic feature quantity extracted from the speech data in the training corpus and the acoustic feature quantity output from the speech synthesis model. This makes it possible to obtain a speech synthesis model capable of controlling emotional parameters.

The speech synthesis system of one embodiment herein maintains a target speech database that stores target speech samples for which the speech synthesis system mimics emotions, and uses an emotion recognizer for each of the target speech samples to identify emotion parameters. get The speech synthesis system calculates emotion parameter statistics and stores them in the database.

The speech synthesis system generates a model that relates the statistics of the emotion samples to the conditions (utterance conditions) that indicate the situation in which the person to be spoken to or the dialogue system is placed. Synthetic speech with more universal emotions can be generated by using the statistics of emotion parameters.

The speech synthesis system obtains the conditions indicating the current situation in which the speech target person or the dialogue system is placed, and generates emotion parameters using the above model. The speech synthesis system inputs the linguistic feature quantity of the target utterance content and the generated emotion parameter to the trained speech synthesis model, and obtains the acoustic feature quantity accompanied by the emotion imitating the target speech. A speech synthesizer generates synthesized speech to be uttered from the acoustic features.

A speech synthesis system according to one embodiment of the present specification obtains distributions of emotion parameters in the training corpus and the target speech database, and based on these differences, modifies the statistics of the emotion parameters obtained from the target speech. As a result, differences in emotional expressions between the training corpus and the target speech can be absorbed, and speech with more natural emotional expressions can be synthesized.

<First embodiment>
In this embodiment, an emotion recognizer is applied to a speech synthesis corpus, which is a training corpus, and target speech. The training of the speech synthesis model uses emotion parameters obtained from the speech synthesis corpus as training data. During speech synthesis by the speech synthesis model, emotion parameters obtained from the target speech are used as inputs to the speech synthesis model.

The speech synthesis corpus stores, for example, uttered voice data of voice actors who specialize in uttering, and texts indicating the content of utterances. The target speech is the human speech that the speech synthesis system aims to imitate. For example, voice data and text of speech by people in specific occupations (for example, nursing care and customer service) are stored in the database. As a result, it is possible to realize easy-to-hear utterances that express emotions suitable for the utterance target person or the situation of the dialogue system.

FIG. 1A illustrates an example logical configuration of a speech synthesis system according to one embodiment herein. The speech synthesis system 100 includes a speech synthesis corpus 1010 , an acoustic feature quantity converter 1040 , a language feature quantity converter 1050 , an emotion recognizer 1060 , a speech synthesis model learning unit 1070 and a speech synthesis model 1080 .

The speech synthesis system 100 further includes a target speech database (DB) 1090, an emotion recognizer 1130, a cluster generator 1140, a target speech statistics calculator 1150, a target speech statistics database 1160, and a condition-to-class conversion model 1170. Furthermore, speech synthesis system 100 includes emotion parameter acquisition section 1190 , emotion parameter conversion section 1200 , language feature quantity conversion section 1220 , and speech synthesizer 1230 .

The speech synthesis system 100 receives a condition (utterance condition) 1180 indicating the situation in which the dialogue system is placed, and calculates an emotion parameter corresponding to the condition 1180 . The emotion parameter to be calculated is a value based on the statistics of the emotion parameter of existing samples. The speech synthesis system 100 generates synthetic speech 1240 from text 1210 indicating the generated emotional parameters and the utterance content. As a result, it is possible to utter a voice that accompanies emotion in accordance with the situation.

In the configuration example shown in FIG. 1A, the emotion parameter acquisition unit 1190 generates emotion parameters from the input condition 1180 using the condition-class conversion model 1170 and the target speech statistics database 1160 . Condition-to-class transformation model 1170 and target speech statistics database 1160 are models that associate input conditions with emotion parameter statistics of target speech samples. As described below, in another configuration, speech synthesis system 100 uses machine learning models to generate emotion parameters from input conditions 1180 .

FIG. 1B illustrates another logical configuration example of the speech synthesis system 100 according to one embodiment herein. The speech synthesis system 100 shown in FIG. 1B includes a condition-emotion parameter conversion model 1165 instead of the condition-class conversion model 1170 and the target speech statistics database 1160 of the configuration example shown in FIG. 1A. Also, a normalization unit 1145 is included instead of the cluster generation unit 1140 .

The condition-emotion parameter conversion model 1165 is a machine learning model that outputs emotion parameters for the input conditions 1180 . A conditional to affective parameter conversion model 1165 is trained using the affective parameters of the target speech sample. Therefore, the condition-emotional parameter conversion model 1165 is a model that associates the input condition with the emotion parameter statistics of the target speech sample. Note that both configurations shown in FIGS. 1A and 1B may be included in one speech synthesis system, and either one of the modes may be selectable.

FIG. 2 shows an example configuration of the speech synthesis corpus 1010 . The speech synthesis corpus 1010 stores data for training the speech synthesis model 1080 . Speech synthesis corpus 1010 includes speech sample column 1020 and text column 1030 . Each record shows the audio data of a particular utterance and the text of that utterance. Note that the speech synthesis corpus 1010 may hold conversion results of at least a part of the acoustic feature quantity conversion unit 1040 , the language feature quantity conversion unit 1050 , and the emotion recognizer 1060 .

FIG. 3 shows a configuration example of the target speech database 1090. As shown in FIG. The target speech database 1090 stores information on utterances of target speakers. Specifically, target speech database 1090 includes speaker ID column 1100 , speech sample column 1110 , text column 1120 , and condition column 1125 . Each record indicates information of one utterance.

A speaker ID column 1100 indicates the ID of the speaker. The voice sample column 1110 shows the voice data of the utterance. A text field 1120 indicates the text of the uttered content. A condition column 1125 indicates the condition under which the voice was uttered.

The condition field 1125 can contain any condition that may alter the emotion of a person's speech in the intended use of the text-to-speech system 100 . For example, the location and time period in which the utterance was performed can be mentioned. Examples of locations are dining rooms, rooms, common rooms, and the like. In addition, a condition indicating for what purpose the target speaker spoke to the other party may be included.

For example, the purpose of encouraging the other party, the purpose of caring for the other party, greetings, etc. can be treated as different conditions. A condition may be expressed by one variable or may be expressed by multiple variables. In the multi-variable representation, one situation in which an utterance is made is represented as a set of multiple condition variables, ie, a tuple. The situation includes the location, time zone, purpose of the utterance, etc. of the utterance. Note that the target speech database 1090 may store the conversion result of the emotion recognizer 1130 in the emotion recognizer 1060, or the text may be omitted.

FIG. 4 shows a configuration example of the target speech statistics database 1160 shown in FIG. 1A. The target speech statistics database 1160 stores statistical information of target speech (records) stored in the target speech database 1090 . Specifically, the target speech statistic database 1160 includes a class ID column 1161 and an emotion parameter column 1162 .

A class ID column 1161 indicates the ID of the class to which each target speech in the target speech database 1090 belongs. The emotion parameter column 1162 shows statistics of the emotion parameter of the target voice in each class. A statistic is, for example, an average value, a mode value, a median value, or the like. In the following description, the mean value of each of the emotional parameter variables is used as an example. In FIG. 4, emotion parameters are represented by multidimensional vectors, and each variable indicates a different emotion. The emotion parameter in the emotion parameter column 1162 indicates the average value of each variable in the corresponding class.

FIG. 5 shows a configuration example of the condition-class conversion model 1170 shown in FIG. 1A. Condition-to-class conversion model 1170 determines class IDs from conditions of utterances specified for the generation of synthesized speech. Emotion parameters corresponding to the output target ID are determined from the target speech statistics database 1160 . In the configuration example shown in FIG. 5, condition-class transformation model 1170 is a decision tree. The configuration of the condition-class conversion model is arbitrary.

FIG. 6 is a block diagram schematically showing a hardware configuration example of the speech synthesis system 100. As shown in FIG. The speech synthesis system 100 includes an arithmetic device 101 having arithmetic performance, and a main storage device 102 providing a storage area for storing programs to be executed by the arithmetic device 101 and data to be processed. The arithmetic device 101 is, for example, a CPU including one or more cores, and the main storage device 102 is, for example, a RAM including a volatile storage area.

The speech synthesis system 100 further includes a communication interface 106 that performs data communication with other computer devices and external storage devices, an auxiliary storage device 103 that provides a non-volatile storage area using a HDD (Hard Disk Drive), flash memory, etc. including. The speech synthesis system 100 also includes an input device 104 that receives operations from a user, and an output device 105 that presents the output results of each process to the user.

Input device 104 includes, for example, a keyboard and mouse, and output device 105 includes, for example, monitor and printer. These components of speech synthesis system 100 can communicate via internal bus 107 .

Programs to be executed by the arithmetic device 101 and data to be processed are loaded from the auxiliary storage device 103 to the main storage device 102, for example. The functional units shown in FIG. 1A or 1B can be implemented by the arithmetic device 101 executing corresponding programs. Various data such as the corpus and database shown in FIG. 1A or 1B can be stored in the auxiliary storage device 103, for example.

The speech synthesis system 100 may be a physical computer system (one or more physical computers), or a system built on a group of computational resources (a plurality of computational resources) such as a cloud platform. The speech synthesis system 100 may be a mobile device such as a smart phone or a tablet. A computer system or group of computing resources includes one or more interface devices, one or more storage devices (including, for example, main storage devices and auxiliary storage devices), and one or more computing devices.

When a function is realized by executing a program by an arithmetic device, the function is at least a part of the arithmetic device because the specified processing is performed using a storage device and/or an interface device as appropriate. may The processing described with function as the subject may be processing performed by a system having an arithmetic device or its processor.

Programs may be installed from program sources. The program source may be, for example, a program distribution computer or a computer-readable storage medium (eg, a computer-readable non-transitory storage medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

Returning to FIG. 1A or 1B, speech synthesis model training unit 1070 trains speech synthesis model 1080 . For this training, the speech synthesis model learning unit 1070 inputs the text of the record acquired from the speech synthesis corpus 1010 to the language feature amount conversion unit 1050 to acquire language feature amounts. The speech synthesis model learning unit 1070 inputs the speech sample of the record to the acoustic feature quantity conversion unit 1040 to acquire the acoustic feature quantity. Further, speech synthesis model learning section 1070 inputs the speech sample to emotion recognizer 1060 to acquire emotion parameters.

The speech synthesis model learning unit 1070 inputs language feature quantities and emotion parameters to the speech synthesis model 1080 . The speech synthesis model 1080 outputs acoustic features corresponding to the input. The configuration parameters of the speech synthesis model 1080 are updated based on the error between the acoustic feature quantity output by the speech synthesis model 1080 and the acoustic feature quantity from the acoustic feature quantity conversion unit 1040 .

The speech synthesizer 1230 generates synthesized speech 1240 from the acoustic features output from the speech synthesis model 1080 . Therefore, the speech synthesis model 1080 learns so that the speech waveform of the synthesized speech 1240 is finally generated. As described above, in this embodiment, multidimensional vectors are assumed as the format of emotion parameters. Each dimension represents the intensity of a different aspect of emotion. Note that the number of dimensions of the vector is any number greater than or equal to 1, and one dimension can represent multiple types of emotions.

Details of the learning procedure of the speech synthesis model 1080 will be described. FIG. 7 is a flowchart of an example training procedure for the speech synthesis model 1080 . In step S101, the speech synthesis model learning unit 1070 inputs each speech sample in the speech synthesis corpus 1010 to the emotion recognizer 1060 to obtain emotion parameters.

In step S102, the speech synthesis model learning unit 1070 inputs each speech sample in the speech synthesis corpus 1010 to the acoustic feature amount conversion unit 1040 to obtain acoustic feature amounts. In step S103, the speech synthesis model learning unit 1070 inputs each text in the speech synthesis corpus 1010 to the language feature conversion unit 1050 to obtain language features.

In step S<b>104 , the speech synthesis model learning unit 1070 learns (trains) the speech synthesis model 1080 . Specifically, the speech synthesis model learning unit 1070 inputs the emotion parameters and language feature values obtained in steps S101 and S103 to the speech synthesis model 1080, and obtains the output acoustic feature values. The speech synthesis model learning unit 1070 trains the speech synthesis model 1080 so that the error between the output acoustic feature quantity and the acoustic feature quantity acquired in step S102 is reduced. In other words, learning is performed so as to maximize the probability of generating correct acoustic features when language features and emotion parameters are input to the speech synthesis model 1080 .

Returning to FIG. 1A or 1B, speech samples in target speech database 1090 are input to emotion recognizer 1130 . Emotion recognizer 1130 is implemented by a different program or the same program as emotion recognizer 1060 used for speech synthesis corpus 1010 . In one example, emotion recognizers 1130 and 1060 are identical.

In the configuration example shown in FIG. 1A , the emotion parameters output by the emotion recognizer 1130 are input to the cluster generator 1140 . The cluster generator 1140 clusters the records in the target speech database 1090 and generates classes to which each record belongs. The details of the class generation method will be described later.

In the configuration example of FIG. 1A, the class generation result is input to the target speech statistic calculator 1150 . The target speech statistics calculator 1150 calculates statistics of records belonging to each class and stores them in the target speech statistics database 1160 . As described with reference to FIG. 4, the statistic is the average value of emotion parameters of records in each class. Target speech statistics calculator 1150 also generates condition-class conversion model 1170 . The generated condition-class conversion model 1170 is stored in the auxiliary storage device 103 .

In the configuration example of FIG. 1B , the emotion parameters output by the emotion recognizer 1130 are input to the normalization section 1145 . The normalization unit 1145 normalizes the emotion parameter. In the configuration example of FIG. 1A, normalization processing is performed by the cluster generator 1140 . Details of the normalization process will be described later. Note that the normalization process is executed when the normalization mode is designated.

In the configuration example of FIG. 1B, the target speech statistic calculation unit 1150 calculates the emotion parameter normalized by the normalization unit 1145 or the emotion parameter without normalization, and the condition indicated by the condition column 1125 of the target speech database 1090. , training (learning) of the condition-emotion parameter conversion model 1165 is performed.

Details of the above processing for the target speech database 1090 will be described. FIG. 8 shows a flow chart of an example of processing for generating a model that associates conditions and emotional parameters from the target speech database 1090 .

In step S201, the emotion recognizer 1130 determines emotion parameters from each speech sample in the target speech database 1090. FIG. In step S202, the cluster generation unit 1140 or the normalization unit 1145 determines whether or not to apply the normalization mode according to presetting. The normalization mode is a mode that enables processing to reduce speaker-related effects in the target speaker's speech.

Steps S203 and S204 are processes when the normalization mode is valid. In step S<b>203 , the cluster generator 1140 or the normalizer 1145 finds the average value of the emotional parameters of each speaker in the target speech database 1090 . Specifically, the cluster generation unit 1140 extracts records having the same speaker ID, and obtains the average value of each variable of the emotion parameter of all the extracted records.

In step S204, the cluster generation unit 1140 or the normalization unit 1145 subtracts the average value of the speaker's emotion parameter of each target voice sample from the emotion parameter of each target voice sample obtained by the emotion recognizer 1130. FIG. As a result, it is possible to obtain emotion parameters indicating changes in emotion from the reference level of each speaker.

In step S205, the target speech statistic calculation unit 1150 models the relationship between the condition indicated by the condition column 1125 of the target speech database 1090 and the statistic of the emotion parameter of the target speech sample.

There are several examples of methods for computing affective parameter statistics from input conditions, ie models that associate input conditions with affective parameter statistics. Clustering-based modeling, mean-value-based modeling, and machine-learning-based modeling are described below. Clustering-based modeling corresponds to the example configuration of FIG. 1A, and machine learning-based modeling corresponds to the example configuration of FIG. 1B. Mean value-based modeling corresponds to a configuration example in which the cluster generator 1140 is omitted from the configuration example in FIG. 1A.

First, clustering-based modeling is described. FIG. 9A is a flowchart illustrating an example of clustering-based modeling. In step S<b>311 , the cluster generator 1140 performs unsupervised clustering on the emotion parameters of the target speech samples obtained from the emotion recognizer 1130 . This will generate several classes (clusters).

In step S312, the target speech statistic calculation unit 1150 calculates the average value (average value of each variable) of the emotion parameters of the target speech samples belonging to each class. The emotion parameter before the average value is calculated is the vector from which the speaker's emotion parameter average value is subtracted in the normalized mode, and the vector output from the emotion recognizer 1130 in the non-normalized mode.

In step S313, the target speech statistic calculation unit 1150 sets the condition shown in the condition column 1125 of the target speech database 1090 as an explanatory variable, and creates a decision tree that outputs the class ID of each target speech sample determined in step S311. train. In step S 314 , target speech statistic calculator 1150 saves the trained decision tree as condition-class transformation model 1170 .

In step S<b>315 , the target speech statistics calculation unit 1150 stores the average value of the emotion parameters of each class calculated in step S<b>312 in the emotion parameter column 1162 of the target speech statistics database 1160 . The corresponding class ID is stored in class ID column 1161 .

A corresponding class ID can be identified by inputting a condition 1180 representing a situation when synthetic speech is generated for the decision tree generated in step S313. By inputting the emotional parameters belonging to the identified class and the linguistic features of the uttered text to the speech synthesis model 1080, it is possible to generate emotional speech suitable for the situation. Note that a modeling method different from the decision tree may be used as long as it is a model that outputs a class ID with conditions as input.

Next, mean value-based modeling will be described. FIG. 9B is a flowchart illustrating an example of mean-based modeling. First, in step S321, a table for converting conditions into classes is prepared. This table can be assumed to be created by the developer. In the simplest case, one class can adopt the corresponding relationship for each combination of conditions.

In step S322, the target speech statistic calculation unit 1150 calculates the average value of the emotion parameters of the target speech samples belonging to each class. The class of target speech samples is determined by the above table. In step S323, the target speech statistic calculation unit 1150 saves the table obtained in step S321 as the condition-class conversion model 1170. FIG.

In step S 324 , the target speech statistic calculation unit 1150 stores the average value of the emotion parameter of each class obtained in step S 322 in the emotion parameter column 1162 of the target speech statistic database 1160 . The corresponding class ID is stored in class ID column 1161 .

The average value-based method is simple, but this method is particularly effective when the breakdown of the situations for which speech is to be generated is clear and the speech of the target speaker in each situation can be collected.

Next, machine learning-based modeling is described. FIG. 9C is a flowchart illustrating an example of machine learning-based modeling. In step S331, the target speech statistic calculation unit 1150 creates a data set with conditions as input variables and emotion parameters as outputs (that is, prediction target variables). In step S332, the target speech statistic calculator 1150 generates a machine learning model that transforms inputs into outputs based on the data set. In step S333, target speech statistic calculation unit 1150 stores the generated machine learning model as condition-emotion parameter conversion model 1165 shown in FIG. 1B.

In this method, the machine learning model directly outputs the emotion parameters when the conditions representing the situation are specified. Therefore, the target speech statistics database can be omitted. The trained machine learning model outputs affective parameters based on the affective parameters of speech samples in the target speech database 1090 . The output of the machine learning model represents the sentiment parameter statistics of the speech samples.

Returning to FIG. 1A or 1B, speech synthesis based on the trained speech synthesis model and the statistics of the emotion parameters of the target speech sample will be described. Condition 1180 is a variable that represents the situation in which speech is emitted by the dialogue system. Condition 1180 is represented by variables defined in condition column 1125 of target speech database 1090 . A condition 1180 is input from an external system (not shown) of the speech synthesis system 100 . Condition 1180 is input to emotion parameter acquisition section 1190 .

In the configuration example of FIG. 1A, emotion parameter acquisition section 1190 uses condition-class conversion model 1170 to determine the class corresponding to specified condition 1180 . Furthermore, the emotion parameter acquisition unit 1190 reads the emotion parameter corresponding to the determined class from the emotion parameter column 1162 of the target speech statistic database 1160 . In the configuration example of FIG. 1B, emotion parameters are generated from input conditions 1180 by condition-emotion parameter conversion model 1165 .

The emotion parameter conversion unit 1200 does not perform any particular processing in this embodiment. Therefore, the emotion parameter obtained by emotion parameter acquisition section 1190 is input to speech synthesizer 1230 without conversion.

A text 1210 is a character string defining a sentence to be generated by speech synthesis. The linguistic feature conversion unit 1220 converts the input text 1210 into a linguistic feature to be input to the speech synthesizer 1230 . This process is the same as the process of the language feature quantity conversion unit 1050 .

The speech synthesizer 1230 inputs the input linguistic feature amount and emotion parameter to the speech synthesis model 1080 to obtain an acoustic feature amount. A speech synthesizer 1230 generates and outputs a synthesized speech 1240 from the acoustic features.

The details of the speech synthesis procedure will be explained. FIG. 10 is a flowchart illustrating an example method for speech synthesis. In step S401, the process branches depending on whether the method of calculating the statistics for the emotion parameter of the target voice sample is machine learning based.

If a method different from the machine learning base is used (S401: NO), the flow proceeds to step S402. Emotion parameter acquisition section 1190 determines a class ID from input condition 1180 using condition-class conversion model 1170 . Furthermore, in step S403, the emotion parameter acquisition unit 1190 reads the emotion parameter corresponding to the class ID from the emotion parameter column 1162 of the target speech statistics database 1160. FIG.

If a machine learning based technique is used (S401: YES), flow proceeds to step S410. When using a machine learning base, the condition-class conversion model 1170 is a machine learning model that outputs emotion parameters when conditions are input. Emotion parameter acquisition section 1190 inputs condition 1180 to condition-class conversion model 1170 and acquires an emotion parameter.

In step S404, emotion parameter acquisition section 1190 determines whether or not the specified mode is normalization mode. If the normalization mode is valid (S404: YES), the flow proceeds to step S405. In the normalization mode, the emotion parameter obtained in step S403 or step S410 is not an absolute value but a variation (relative value) from an average emotion parameter included in the speech of one speaker.

Therefore, in step S405, emotion parameter acquisition section 1190 adds the average emotion parameter of the specific speaker to this emotion parameter. It is arbitrary from which speaker the emotion parameter to be added here is obtained. For example, a method of selecting the speaker who speaks with the most preferable voice among the target speakers, a method of selecting a speaker who uttered the voice of the speech synthesis corpus 1010, and averaging the emotional parameters obtained from various voices of the speaker values can be added.

If the normalization mode is invalid (S404: YES) or after step S405, the emotion parameter conversion unit 1200 further converts the obtained emotion parameter in step S406. However, this conversion is not performed in this embodiment. Therefore, there is no conversion.

In step S407, the linguistic feature quantity conversion unit 1220 generates and outputs a linguistic feature quantity from the input text 1210. FIG. In step S<b>408 , speech synthesizer 1230 receives the linguistic feature amount from linguistic feature amount conversion section 1220 and the emotion parameter from emotion parameter conversion section 1200 .

In step S409, speech synthesizer 1230 generates synthesized speech 1240 and outputs it. The speech synthesizer 1230 inputs the acquired linguistic feature quantity and emotion parameter to the speech synthesis model 1080 and acquires the outputted acoustic feature quantity. A speech synthesizer 1230 generates synthesized speech 1240 from the acoustic features.

As described above, in the first embodiment, the speech synthesis model is trained using the emotion parameters obtained through the emotion recognizer for the speech synthesis corpus, and the statistics of the emotion parameters obtained by using the emotion recognizer from the target speech are obtained. to perform speech synthesis. This makes it possible to easily realize speech synthesis that imitates the emotion of the target speech.

<Second embodiment>
In this embodiment, the emotion parameters of the target speech are transformed so that the emotion parameter distribution of the target speech database becomes closer to the emotion parameter distribution of the speech synthesis corpus. The converted emotion parameters are input to the speech synthesizer.

This enables speech synthesis by simulating intra-speaker variations in the emotions of the speakers of the speech synthesis corpus used for training. As a result, it is possible to synthesize a voice that is closer to the emotion desired by the developer. Since this embodiment is obtained by making additional changes to the first embodiment, differences from the first embodiment will be mainly described.

FIG. 11 schematically shows a logical configuration example of a speech synthesis system 100 according to the second embodiment. In addition to the configuration example shown in FIG. 1A, the speech synthesis system 100 includes a speech synthesis corpus statistics calculator 1250 and a speech synthesis corpus statistics database 1260 . Furthermore, the speech synthesis system 100 includes a transformation parameter database 1280 and a transformation parameter calculator 1270 . Note that the configuration of this embodiment is similarly applied to the configuration example shown in FIG. 1B.

Referring to FIG. 11 , speech synthesis corpus statistic calculation unit 1250 calculates statistics in emotion parameters of speech samples in speech synthesis corpus 1010 , that is, parameters that define distribution. The speech synthesis corpus statistics database 1260 stores the statistics obtained by the speech synthesis corpus statistics calculator 1250 .

Although any form of distribution can be used, a normal distribution is assumed here to find the mean and standard deviation. Since it is assumed that the emotion parameter output by the emotion recognizer is a multidimensional vector, the mean and standard deviation are also vectors.

Transformation parameter calculation unit 1270 refers to both speech synthesis corpus 1010 and target speech database 1090 to obtain parameters to be used in a transformation algorithm to approximate the distribution of emotion parameters of target speech to speech synthesis corpus 1010 .

When the emotion parameter of the target speech is obtained by the machine learning model (condition-emotion parameter conversion model) 1165, the conversion parameter calculation unit 1270 creates a list of conditions for using the dialogue system, that is, a list of conditions that can be used as the condition 1180. are used to feed machine learning models 1165 with their respective conditions. The conversion parameter calculator 1270 obtains statistics from the emotion parameters obtained from the machine learning model 1165 and uses them as substitutes for the emotion parameters stored in the target speech statistics database 1160 .

This procedure will be described with reference to FIG. In step S501, transformation parameter calculator 1270 inputs each speech sample in speech synthesis corpus 1010 to emotion recognizer 1060 to obtain the emotion parameter of each speech sample.

In step S502, transformation parameter calculation section 1270 obtains a statistic from the emotion parameter. Here, the average and standard deviation are obtained as statistics. Assuming that each audio sample is w and the entire set of audio samples is W, the average value a and the standard deviation σa of the emotion parameter are calculated by the following equations.

|W| is the number of speech samples. SER(w) is a function that obtains the emotional parameter of voice sample w. Both the square and the square root in the standard deviation formula apply to the values in each dimension of the vector.

In step S503, conversion parameter calculator 1270 creates a condition set including many conditions. It is desirable that this set of conditions universally include conditions representing situations in which the dialogue system is actually used. Since there are one or more condition variables that characterize one situation, this set of condition variables is called a condition tuple. Therefore, the condition set contains multiple condition tuples. Furthermore, the ratio of the number of condition tuples to be included may be adjusted according to the frequency of use of each condition in actual use.

Denoting the condition set as X and each stored condition tuple as x, the following equation can be defined. |X| is the number of condition tuples.

The i-th condition tuple xi can be expressed as follows. qi1, qi2, . . . are respective conditions that make up the condition tuple xi, and |Q|

Step S504 is a branch, and branches depending on whether machine learning is used for modeling the emotional parameters of the target speech.

If machine learning is not used (S504: NO), the flow proceeds to step S505. In step S505, the conversion parameter calculator 1270 obtains class IDs from the respective condition tuples. Assuming that the i-th class ID is ci, ci is represented by the following formula. CLS is a function that converts a condition tuple into a class ID.

In step S506, transformation parameter calculation section 1270 obtains emotion parameters corresponding to each class. Assuming that the i-th emotion parameter is ei, ei is expressed by the following equation. EP is a function that converts class IDs into emotion parameters.

If machine learning is used (S504: YES), the flow proceeds to step S509. Transformation parameter calculation section 1270 obtains the emotion parameter directly from the condition tuple as shown in the following formula. EPM is a function that converts condition tuples into emotion parameters, and is implemented by a machine learning model.

In step S507, transformation parameter calculation section 1270 obtains the statistic of the emotion parameter (ei). Here, the average value and standard deviation are obtained. e and σe are the mean value and standard deviation of the emotional parameter obtained from the target speech, respectively.

In step S<b>508 , conversion parameter calculation section 1270 determines a, σa, e, and σe obtained above as parameters for conversion used in emotion parameter conversion section 1200 . These are saved in the conversion parameter database 1280 .

Next, a method of using the above conversion parameters when synthesizing speech will be described. This process is executed by the emotion parameter conversion unit 1200 in step S406 of the flowchart in FIG. Suppose that a certain condition x is specified at the time of speech synthesis, and the corresponding emotion parameter read from the emotion parameter column 1162 is m.

First, the emotion parameter conversion unit 1200 calculates the standard score z in the emotion distribution of the target speech from the emotion parameter m.

Next, the emotion parameter conversion unit 1200 calculates the emotion parameter b in the emotion distribution of the speech synthesis corpus 1010 corresponding to the standard score z.

The emotion parameter conversion unit 1200 inputs this emotion parameter b to the speech synthesizer 1230 to synthesize speech. In other words, the emotion parameter is linearly transformed here.

In this embodiment, since the emotion parameter of the target speech is converted based on the information on the speaker's emotion distribution in the speech synthesis corpus, it is possible to synthesize speech that is closer to the emotion desired by the developer.

<Third Embodiment>
This embodiment updates (corrects) the conversion parameter of the emotion parameter based on the emotion parameter distribution of the speech generated by the speech synthesizer. As a result, it is possible to synthesize a voice that is closer to the emotion desired by the developer. The initial value of the conversion parameter of the emotion parameter may be, for example, the conversion parameter determined in the second embodiment, or may be a preset conversion parameter. The initial value does not have to be set.

FIG. 13 shows a logical configuration example of the speech synthesis system 100 of this embodiment. In addition to the configuration example shown in FIG. 11, an emotion recognizer 1290, an output speech statistic calculator 1300, and an output speech statistic database 1310 are included.

Components related to this embodiment will be described with reference to FIG. An emotion recognizer 1290 obtains emotion parameters from the synthesized speech 1240 output by the speech synthesizer 1230 . The output speech statistic calculation unit 1300 acquires the emotion parameter from the emotion recognizer 1290 and calculates the statistic of the emotion parameter. The output speech statistics calculator 1300 stores the emotion parameter statistics in the output speech statistics database 1310 .

FIG. 14 shows a configuration example of the output speech statistic database 1310 . The output speech statistics database 1310 stores statistical information of synthesized speech generated by the speech synthesizer 1230 . Specifically, output speech statistics database 1310 includes class ID column 1320 and emotion parameter column 1330 . The class ID column 1320 indicates the ID of the class to which each synthesized speech generated by the speech synthesizer 1230 belongs. The emotion parameter column 1330 shows the statistics of the emotion parameter of the target speech in each class. Together with the target speech statistics database 1160, the mean values for each of the emotional parameter variables are stored.

Correction processing of conversion parameters according to the present embodiment will be described. FIG. 15 is a flow chart showing an example of conversion parameter correction processing for emotion parameters. In step S601, conversion parameter calculator 1270 creates a condition set including many conditions. This process is the same as step S503 in the flowchart of FIG.

Step S602 is a branch, depending on whether machine learning is used to model the emotional parameters of the target speech. If machine learning is not used (S602: NO), the flow proceeds to step S604. In step S604, the conversion parameter calculator 1270 obtains class IDs from the respective condition tuples. This process is the same as step S505 in the flowchart of FIG. In step S605, conversion parameter calculation section 1270 obtains emotion parameters corresponding to each class.

If machine learning is used (S602: YES), the flow proceeds to step S603. In step S603, conversion parameter calculation section 1270 obtains an emotion parameter directly from the condition tuple. This process is the same as step S509 in the flowchart of FIG.

In step S606, conversion parameter calculation section 1270 passes the emotion parameter through emotion parameter conversion section 1200 to obtain the emotion parameter after conversion. This processing assumes that the present embodiment is performed after the second embodiment is performed. However, the first embodiment and this embodiment can also be combined. In that case, the emotion parameter is not converted in step S606. If the mathematical expression of the second embodiment is followed, the number of emotion parameters obtained at this point is |X|.

In step S607, conversion parameter calculation section 1270 uses the emotion parameter obtained in step S606 to perform speech synthesis. Any text can be used for speech synthesis. For example, it is possible to use sentences that are often uttered in situations where the dialogue system is used. Also, for each condition tuple obtained with each emotion parameter, a frequently uttered sentence may be individually designed, and the sentence may be input.

In step S608, conversion parameter calculation section 1270 inputs the synthesized speech obtained in step S607 to emotion recognizer 1290 to obtain emotion parameters of the synthesized speech. In step S609, conversion parameter calculation section 1270 obtains a statistic for the emotion parameter obtained in step S608. That is, the distribution of emotion parameters in synthesized speech is obtained. Here, it is assumed that the average value and standard deviation are to be obtained. This process can take the same method as the 6050.

Step S610 is a branch, determining an emotional parameter distribution to be compared with the synthetic speech emotional parameter distribution. It is assumed that the user has designated in advance which emotion parameter distribution to select. Note that the specified emotion parameter distribution may be different from the example below. Also, only one emotion parameter distribution may be specified.

In step S611, transformation parameter calculation section 1270 determines the emotion parameter distribution of speech synthesis corpus 1010 as the comparison target distribution. In this case, the conversion parameter calculation unit 1270 reads the average value and standard deviation stored in the speech synthesis corpus statistics database 1260 .

In step S612, conversion parameter calculation section 1270 determines the comparison target distribution as the emotion parameter distribution before conversion of the target speech. That is, the conversion parameter calculator 1270 reads the average value e and the standard deviation σe obtained in the second embodiment from the conversion parameter database 1280 .

In step S613, conversion parameter calculation section 1270 determines the comparison target distribution as the emotion parameter distribution after conversion of the target speech. That is, the conversion parameter calculation unit 1270 obtains the converted emotion parameter b obtained in the second embodiment for each condition tuple, calculates the average value and standard deviation of the converted emotion parameter between the condition tuples, load.

In step S614, conversion parameter calculation section 1270 obtains the difference between the emotion parameter distribution of synthesized speech and the comparison target distribution. Various methods can be used to calculate this difference. For example, Kullback-Leibler divergence can be used, or a method of simply calculating an average difference can be used.

In step S615, the conversion parameter calculator 1270 determines whether the difference between the distributions obtained in step S614 is a predetermined threshold. If the difference is equal to or less than the threshold (S615: YES), the process ends. If the difference exceeds the threshold (S615: NO), the flow proceeds to step S616.

In step S616, transformation parameter calculation section 1270 updates the transformation parameters so that the distribution difference is reduced. That is, in the second embodiment, the post-conversion emotion parameter b could be uniquely determined. '.

The transformation parameter calculation unit 1270 adjusts σa′ and a′ and updates them so that the distribution difference becomes smaller.

In step S<b>617 , the conversion parameter calculator 1270 saves the converted parameters in the conversion parameter database 1280 . Furthermore, returning to step S602, the transformation parameter calculation unit 1270 repeats until the distribution difference becomes equal to or less than a predetermined threshold. In speech synthesis, the adjusted conversion parameters are used to determine emotion parameters to be used in speech synthesis.

As described above, this embodiment is characterized in that the behavior of the emotion parameter converter is determined based on the emotion recognition result of the speech actually generated by the speech synthesizer. The emotion included in the speech generated by the speech synthesizer does not necessarily match the emotion parameter input to the speech synthesizer. Therefore, as described in the present embodiment, by using the emotion estimation result obtained by the emotion recognizer for the actual synthesized speech, it is possible to generate speech that is closer to the desired emotion.

<Fourth Embodiment>
The fourth embodiment provides a function for the developer to adjust the emotional parameters for speech synthesis obtained in the first to third embodiments. This embodiment will be described with reference to Figures 16A to 16C.

Each graph of FIGS. 16A to 16C shows the distribution of emotions for voice groups, plotting the magnitude of emotion on the horizontal axis and the probability density on the vertical axis. The magnitude of emotion on the horizontal axis is the value of each dimension of the emotion parameter expressed in multiple dimensions, such as activity (Arousal), emotional valence (Pleasantness), and dominance (Dominance).

FIG. 16A shows the emotion distribution between conditions obtained from the emotion parameters of the target voice. Emotion distribution between class IDs may be used instead of conditions. FIG. 16B shows the emotion distribution of speech samples included in the speech synthesis corpus.

The target speech distribution shown in FIG. 16A and the speech synthesis corpus distribution 8020 are often different. Therefore, an embodiment has been described in which emotion parameters are converted so that the distribution of the target speech is close to the distribution of the speech synthesis corpus. Therefore, the speech generated by the dialogue system has emotions generated along the distribution 8020 of the speech synthesis corpus.

FIG. 16C shows the emotional distribution of speech actually used in the application. In the above case, the shape follows the distribution of the speech synthesis corpus 1010 , so the distribution has a shape like line 8035 . For example, if the target voice has an emotion level of x, it is synthesized with an emotion level of x' at the time of synthesis. Both have the same standard score on their respective distributions.

However, depending on the use of the dialogue system, there are cases where it is desirable to manually adjust the emotions from this distribution. For example, there are cases where it is desired to add more emotional ups and downs, or where it is desired to make the person speak calmly overall.

In order to deal with such cases, this embodiment provides a function for the developer to change the conversion parameters. For example, an operation is assumed in which the standard deviation is increased to make the voice more emotional, and the average value is decreased to make the voice calmer overall. Therefore, a function is provided that allows the developer to manipulate the parameters (σa' and a') in the emotion conversion algorithm.

In FIG. 16C, line 8040 shows an example distribution after adjustment. In this case, the emotion level x in the target voice is synthesized with the emotion level x'' at the time of synthesis. Both receive the same standard score in their respective emotional distributions.

The speech synthesis system 100 displays a GUI screen for parameter manipulation on the output device 105 . The speech synthesis system 100 presents, for example, the information of the emotional parameter distribution to be compared to the developer on the output device 105, and according to the user input from the input device 104, the parameters (σa' and a') in the conversion algorithm are changed. change.

For example, the GUI accepts designation from the user of corresponding points between two distributions to be compared, and the speech synthesis system 100 calculates transformation parameters from the positions of those corresponding points. parameters may be determined. The speech synthesis system 100 may display the distribution map before change and the distribution map after change on the output device 105, for example. The transformation parameter may be adjustable for each emotion parameter variable.

A method of specifying a corresponding point between two distributions to be compared by the user and determining a conversion parameter from the corresponding store will be described. The user uses input device 104 to designate a single target speech sample as a reference speech sample and listens to it via output device 105 . Next, speech synthesis system 100 inputs text and emotion parameters to speech synthesizer 1230 to synthesize speech. The user listens to it on the output device 105 .

The user listens to and compares the target speech and the synthesized speech, and changes the emotion parameter input to the speech synthesizer 1230 via the input device 104 so that the emotion of the synthesized speech approaches the emotion of the target speech. The speech synthesis system 100 synthesizes again with the emotion parameter changed by the user, and the user listens to the synthesized speech. This process is repeated to determine the emotional parameters that are input to the speech synthesizer to produce speech that more closely resembles the target speech.

Here, let xi be the emotion parameter obtained by inputting the target voice sample to be compared to the emotion recognizer 1290 and yi be the emotion parameter input to the speech synthesizer 1230 . i represents the order of the target speech used for comparison. The user performs the above operations for multiple target voices. When this operation is performed for N target voices, the emotion parameters of the obtained target voices are x1, x2, . . . , xN. y2, . . . , yN.

The emotion parameter conversion unit 1200 uses the emotion parameters obtained here to convert the distribution. Assume here that the transformation of xi and yi is performed by a linear function. However, transformations in other functional forms may be used. When converting from xi to yi according to the conversion formula of the numerical formula of the third embodiment, the following formula can be used.

e and σe are the mean value and standard deviation of the emotional parameter obtained from the target speech, and are known values. Therefore, to convert from xi to yi, a' and σa' should be obtained.

Assuming a linear relationship between xi and yi, a least squares method can be used to determine a' and σa'. That is, the conversion error D defined by the following formula is calculated.

a' and .sigma.a' that minimize this conversion error D are obtained by calculation. This calculation can be performed by known methods.

In this embodiment, by providing means for changing a small number of conversion parameters, the developer can easily change the emotion of the voice.

<Fifth Embodiment>
The present embodiment provides a function that allows the developer to easily adjust emotions while listening to audio information. FIG. 17 is a flow chart showing an example of an adjustment method. In step S704, the developer designates a single target speech class through the input device 104, which the speech synthesis system 100 receives. As an alternative to classes, conditions may be specified.

In step S<b>705 , the speech synthesis system 100 uses the emotional parameters of the specified class to perform speech synthesis and reproduce it on the output device 105 . In step S707, the input device 104 changes the emotion parameter at the time of voice synthesis based on the result of listening to the voice by the developer. The speech synthesis system 100 receives the changed emotion parameters. Then, in step S708, the speech synthesis system 100 generates synthesized speech using the changed emotion parameters, reproduces the speech by the output device 105, and listens to it by the developer. This process is repeated until the developer finishes editing (loop of NO at step S706).

When the developer has finished editing, the speech synthesis system 100 asks the developer whether to specify the class again. If the developer designates the end of editing (S706: YES), the flow advances to step S702 via step S701. In step S702, the speech synthesis system 100 calculates conversion parameters to convert pre-edited emotion parameters into post-edited emotion parameters. In step S<b>703 , speech synthesis system 100 stores the calculated transformation parameters in transformation parameter database 1280 .

In this embodiment, a developer designs a desired synthesized voice through trial and error. In this embodiment, the entire distribution of emotion parameters is converted based on the relationship between emotion parameters before and after editing specified by the user. That is, even if the user edits the emotions based on, for example, at most two classes, the result is that the overall emotion of the dialogue system's voice is changed to be consistent with the developer's edits. Therefore, it is possible to greatly reduce the time and effort of trial and error of the developer, and to realize efficient design of emotional voices.

<Sixth embodiment>
This embodiment implements the transformation of the emotion parameter distribution in the second embodiment from a different point of view. In the second embodiment, when the emotion parameter distribution is transformed, the variation in emotion parameters between conditions is adjusted to the standard deviation of the speech synthesis corpus. Instead, the present embodiment accommodates variations between emotional conditions in the target speaker's individual speech.

In this embodiment, the following processing is further performed in the calculation of the emotion parameter statistics (S507) in the second embodiment. Here, let S denote the entire set of target speakers. s1, s2, . . . represent respective target speakers. The number of target speakers is |S|.

Furthermore, let us and c denote the speech uttered by the target speaker s in the condition tuple belonging to class c. The conversion parameter calculator 1270 uses the speech us,c to calculate the variance σu,s2 of the interclass emotion parameter appearing in the speech of one target speaker. Multiplication, addition, square calculation, square root calculation, and division, which will be described below, are all calculated for each vector element.

where C is a list of classes.

Furthermore, transformation parameter calculator 1270 averages the variance of each target speaker over all target speakers. Further take the square root and convert to standard deviation σu.

The conversion parameter calculator 1270 saves σu calculated here in the conversion parameter database 1280 .

Next, a method of synthesizing speech using conversion parameters will be described. It is assumed that a certain condition x is designated at the time of speech synthesis, and the emotion parameter read from the emotion parameter column 1162 for that condition is m. Using this value, the emotion parameter b to be input to the speech synthesizer 1230 is obtained by the following equation.

As the average value a in this equation, the average value obtained from the emotion parameter of the speech synthesis corpus or the average value obtained from the emotion parameter of the target speech can be used. The coefficient λ in this formula is obtained by dividing the standard deviation value for each dimension of the emotion parameter. However, when the number of classes is small, the value of each dimension of the standard deviation may be close to 0, and the division may result in an extremely large value. As a countermeasure, λ used in the above equation may be calculated as a scalar using the following equation.

Alternatively, the following formula can also be used.

Here, E is the number of dimensions of the emotion parameter. [i] indicates to refer to the i-th element of the vector.

That is, in the second embodiment, the variation in emotion is made to match the speech of the speech synthesis corpus. On the other hand, this embodiment synthesizes speech so as to simulate the variation (distribution) that appears in each target speaker. As a result, compared to the second embodiment, it is possible to synthesize speech that more closely mimics the manipulation of the target speaker's emotions. In practice, a developer may compare the sound effects of the second embodiment and the sound of this embodiment and select an embodiment suitable for the application.

<Other activities>
In the above embodiments, conversion using a linear function has been described in conversion of emotion parameters, but conversion using other non-linear functions may be employed.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the described configurations. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace part of the configuration of each embodiment with another configuration.

Further, each of the configurations, functions, processing units, etc. described above may be realized by hardware, for example, by designing a part or all of them using an integrated circuit. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (Solid State Drives), or recording media such as IC cards and SD cards.

In addition, the control lines and information lines indicate those considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In fact, it may be considered that almost all configurations are interconnected.

100 Speech Synthesis System 101 Arithmetic Device 102 Main Storage Device 103 Auxiliary Storage Device 104 Input Device 105 Output Device 106 Communication Interface 1010 Speech Synthesis Corpus 1040 Acoustic Feature Amount Converter 1050 Language Feature Amount Converter 1060, 1130, 1290 Emotion Recognizer 1070 Voice Synthesis model learning unit 1080 Speech synthesis model 1090 Target speech database 1140 Cluster generation unit 1145 Normalization unit 1150 Target speech statistics calculation unit 1160 Target speech statistics database 1165 Condition-emotion parameter conversion model 1170 Condition-class conversion model 1190 Emotion parameter acquisition Unit 1200 Emotion parameter conversion unit 1220 Language feature conversion unit 1230 Speech synthesizer 1250 Speech synthesis corpus statistics calculation unit 1260 Speech synthesis corpus statistics database 1270 Conversion parameter calculation unit 1280 Conversion parameter database 1300 Output speech statistics calculation unit 1310 Output speech statistics database

Claims (15)

A speech synthesis system,
one or more computing devices;
one or more storage devices,
The one or more storage devices are
storing a speech synthesis model that generates acoustic features of the text based on the emotional parameters and the linguistic features of the text; and a transformation model that associates speech conditions with statistics of the emotional parameters of a target speech sample;
the target voice sample is a voice sample of a target speaker whose emotion is to be mimicked;
The one or more computing devices,
Get the input current utterance condition,
generating current emotion parameters from the current utterance condition using the conversion model;
generating acoustic features of the target utterance using the speech synthesis model based on the current emotion parameters and the linguistic features of the target utterance;
A speech synthesis system that generates synthesized speech based on the acoustic features. A speech synthesis system according to claim 1,
A speech synthesis system, wherein the speech synthesis model is trained using speech samples of a speaker different from the target speaker. A speech synthesis system according to claim 1,
The transformation model is
determining a class corresponding to the current utterance condition;
determining a statistic of an emotion parameter corresponding to the class;
A speech synthesis system that outputs statistics of the emotion parameter as the current emotion parameter. A speech synthesis system according to claim 3,
The one or more computing devices,
performing clustering of the emotion parameter of the target audio samples to determine the relationship between the classes to which the statistics of the emotion parameter belong;
A speech synthesis system, wherein information of a class to which the emotion parameter statistics belong is included in the conversion model. A speech synthesis system according to claim 4,
The speech synthesis system, wherein the emotion parameter on which the clustering is performed is an emotion parameter excluding an average value of emotion parameters obtained from target speech samples of the same speaker. A speech synthesis system according to claim 1,
The speech synthesis system, wherein the transformation model is a machine learning model that outputs the current emotion parameter from the input current utterance condition. A speech synthesis system according to claim 6,
The speech synthesis system, wherein the emotion parameter of the learning data of the machine learning model is an emotion parameter excluding an average of emotion parameters obtained from target speech samples of the same speaker. A speech synthesis system according to claim 1,
the one or more storage devices store training data for the speech synthesis model;
The one or more arithmetic devices convert the current emotion parameter so that the distribution of the conversion result of the conversion model approaches the distribution of the emotion parameter of the training data of the speech synthesis model before generating the acoustic feature quantity. , speech synthesis system. A speech synthesis system according to claim 1,
The one or more computing devices convert the current emotion parameter according to the conversion parameter before generating the acoustic feature quantity,
A speech synthesis system that updates the transformation parameters so that the distribution of emotion parameters of synthesized speech approaches a distribution of predesignated emotion parameters. A speech synthesis system according to claim 9,
The distribution of the pre-specified emotion parameter is transformed by the distribution of the emotion parameter of training data of the speech synthesis model, the distribution of the emotion parameter of the target voice sample, or the transformation parameter of the emotion parameter of the target voice sample. A speech synthesis system that is either a distribution of emotional parameters of A speech synthesis system according to claim 9,
The speech synthesis system, wherein the one or more arithmetic units repeatedly update the conversion parameter until a difference between the distribution of the emotion parameter of the synthesized speech and the distribution of the pre-designated emotion parameter becomes equal to or less than a threshold. A speech synthesis system according to claim 9,
The speech synthesis system, wherein the one or more arithmetic units present information on the distribution of emotion parameters of the synthesized speech and information on the distribution of the pre-specified emotion parameters, and receive editing of the transformation parameters by a user. A speech synthesis system according to claim 1,
The one or more computing devices convert the current emotion parameter so that the distribution of the transformation result of the transformation model approaches the speaker's emotion parameter distribution of the target speech sample before generating the acoustic feature. speech synthesis system. A speech synthesis system according to claim 1,
The one or more computing devices,
transforming the current emotion parameter according to a transformation parameter before generating the acoustic feature;
A speech synthesis system that accepts a user designation of a correspondence point between a reference speech sample and a current distribution of emotion parameters of synthesized speech according to the transformation parameters, and updates the transformation parameters based on the correspondence points. A method for a system to synthesize speech, comprising:
The system stores a speech synthesis model that generates acoustic features of the text based on the emotional parameters and the linguistic features of the text, and a transformation model that associates utterance conditions with statistics of the emotional parameters of a target speech sample. ,
the target voice sample is a voice sample of a target speaker whose emotion is to be mimicked;
The method comprises: the system comprising:
Get the input current utterance condition,
generating current emotion parameters from the current utterance condition using the conversion model;
generating acoustic features of the target utterance using the speech synthesis model based on the current emotion parameters and the linguistic features of the target utterance;
generating synthetic speech based on the acoustic features.

Images