JPH0744727A - Method and device for generating picture - Google Patents

Abstract

PURPOSE:To easily obtain an animation in which the mouth shape changes in synchronism with an input sound. CONSTITUTION:A transformation function transforming an acoustic parameter extracted from sound into a weighting parameter for changing the mouth shape of a picture is previously obtained by multiple regression analysis. The acoustic parameter is extracted from inputted sound in an acoustic analysis part 1, and the acoustic parameter is transformed into the weighting parameter by the previously obtained transformation function. The animation in which the mouth shape changes is generated by a face picture synthesizer 3 based on the weighting parameter.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image producing method and apparatus suitable for use in producing, for example, an animation in which a mouth shape is changed in synchronization with input voice.

[0002]

2. Description of the Related Art Conventionally, an animation in which the mouth and chin move in response to a voice is composed of a plurality of cells in which the shape of the mouth and chin gradually changes so that the mouth and chin appear to move. It is created using drawings.

[0003]

Therefore, in order to create an animation in which the mouth, chin, etc. move in accordance with the voice, many cel images have to be drawn, which requires a great deal of labor.

Therefore, a method of extracting an acoustic parameter from a voice and changing the animation parameter so that the mouth and chin of the animation move in accordance with the acoustic parameter can be considered.

However, in the conventional art, when the animation parameters are calculated from the acoustic parameters, in order to obtain an animation in which the motion does not feel unnatural, it is necessary to perform a non-linear arithmetic processing on the acoustic parameters, and therefore the load on the parameters is increased. If an attempt is made to realize a device that can withstand, there is a problem that the device becomes large in size and high in cost.

The present invention has been made in view of such a situation, and makes it possible to easily obtain an image in which a facial expression changes abundantly in synchronization with a voice.

[0007]

The image creating method of the present invention is an image creating method for creating an image in synchronization with an input voice, in which acoustic parameters extracted from the voice are used to change the mouth shape of the image. A conversion function for converting into a weighted parameter is obtained in advance by multiple regression analysis, the acoustic parameter is converted into a weighted parameter by the conversion function, and the preset basic mouth shape is changed based on the weighted parameter to change the image. It is characterized by creating.

In this image forming method, the basic mouth shape is
The mouth shape can be made to sound "a", "i", and "u", and the mouth shape can be closed.

The image creating apparatus of the present invention is an image creating apparatus for creating an image in synchronization with an input voice, and acoustic analysis unit 1 as an analysis means for acoustically analyzing voice and calculating acoustic parameters, and multiple regression. By the conversion function that converts the acoustic parameters obtained in advance into the weighted parameters for changing the mouth shape of the image,
A weighting parameter converting unit 2 as a converting means for converting an acoustic parameter into a weighting parameter, and changing a preset basic mouth shape based on the weighting parameter,
A face image synthesizer 3 as a creating means for creating an image is provided.

In this image forming apparatus, the basic mouth shape is
The mouth shape can be made to sound "a", "i", and "u", and the mouth shape can be closed.

[0011]

In the image creating method and the image creating apparatus of the present invention, the sound is analyzed acoustically, the sound parameter is calculated, and the sound parameter previously obtained by the multiple regression analysis is changed to change the mouth shape of the image. It is converted into a weighted parameter by a conversion function for converting into a weighted parameter. Then, based on the weighting parameter, the preset basic mouth shape is changed to create an image. Therefore, an image whose mouth shape changes in synchronization with the sound,
It can be obtained with a small amount of calculation.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a block diagram showing the configuration of an embodiment of a voice / face image synchronizing apparatus to which the present invention is applied. The uttered voice is input to the voice analysis unit 1. Speech analysis unit 1
Extracts, for example, an LPC cepstrum coefficient as an acoustic parameter of the voice from the input voice and outputs it to the weighted parameter conversion unit 2. Weighted parameter converter 2
Converts the LPC cepstrum from the voice analysis unit 1 into weighted parameters for generating an animation in which the mouth shape changes, and outputs the weighted parameter to the face image synthesizer 3. The face image synthesizer 3 generates an animation in which the mouth shape changes based on the weighted parameter from the weighted parameter conversion unit 2, and displays it on a display (not shown), for example.

In the present specification, the mouth shape means
Unless otherwise specified, it means all parts of the face, such as the lower jaw, that move to produce a voice, in addition to the mouth.

Next, the operation will be described. The acoustic analysis unit 1 samples the input speech, for example, as shown in FIG. 2, performs acoustic analysis processing such as linear prediction analysis processing, and calculates a linear prediction coefficient representing the spectrum envelope characteristic of the speech. In this embodiment, the input voice is sampled at 8 kHz,
A Hamming window having a width of 0 sample (= 30 ms) is applied while shifting by 120 samples (= 15 ms), and linear prediction coefficients up to the 10th order are calculated by the so-called autocorrelation method.

Then, the acoustic analysis unit 1 uses the linear prediction coefficient calculated as described above, and a widely known recursive equation (for example, "Digital Speech Processing", by Furui, Tokai University Press, page 47). According to the described recursive formula), the LPC cepstrum coefficient up to the same degree as the linear prediction coefficient is calculated and output to the weight parameter conversion unit 2. Therefore, in this embodiment, LPs up to the 10th order are generated every 15 ms.
The C cepstrum coefficient will be output to the weighting parameter conversion unit 2.

Here, it is known that the LPC cepstrum accurately represents the spectral envelope characteristic of speech with a small order.

The weighting parameter conversion unit 2 outputs L in time series from the acoustic analysis unit 1 by a conversion function (multiple regression equation) described later, which is obtained in advance by multiple regression analysis.
The PC cepstrum coefficient is, for example, as shown in FIG. 3, for example, each of four basic patterns (hereinafter, referred to as mouth shape objects) for expressing a mouth shape of an animation is weighted parameters W _{1 to} W ₄ as weights.
To the face image synthesizing unit 3 and sequentially output to

The face image synthesizer 3 applies, for example, a multiple interpolation method between the weighting parameters W _{1 to} W ₄ output from the weighting parameter converter 2 and the preset basic mouth shape. An animation in which the shape of the mouth (mouth shape) changes is generated in synchronization with the sound input to the acoustic analysis unit 1.

That is, the face image synthesizer 3 has the structure shown in FIG.
Four mouth shape objects are stored as a basic pattern of the mouth shape of the animation, and each mouth shape object is mixed (multiple interpolation) based on the weighting parameters W _{1 to} W ₄ from the weighting parameter conversion unit 2. , Generates a mouth-shaped animation corresponding to the voice input to the acoustic analysis unit 1.

Here, the mouth-shaped object of FIG. 3 (A) shows a mouth shape for uttering the sound of "A" with the lower jaw lowered and the mouth opened, and the mouth-shaped object of FIG. 3 (B) is a tooth. The figure shows the mouth shape of uttering the voice of "i" with the lips opened to the side. In addition, the mouth shape object of FIG. 3C shows a mouth shape for uttering a “u” voice with closed lips, and the mouth shape object of FIG. 3D is a mouth shape when the mouth is naturally closed. The shape is shown.

For example, the mouth shape for producing the voice "E" is
A mouth shape (Fig. 3 (A)) that produces the sound of "A",
It can be expressed by a mouth shape intermediate to the mouth shape (Fig. 3 (B)) for uttering the "I" voice, and the mouth shape for uttering the "O" voice utters the "A" voice. Mouth shape (Fig. 3
(A)) and a mouth shape that produces a voice of "U" (Fig. 3).
It can be expressed by a mouth shape in the middle of (C)).

Furthermore, the above-mentioned "A", "I", "U", "E", and "O" such as mouth shapes for uttering consonants are used.
Mouth shapes other than vowels can also be expressed by mixing the mouth shape objects shown in FIG. 3 at a predetermined ratio (weight).

Therefore, in the face image synthesizer 3, FIG.
As shown in, the weight parameter converting unit 2 outputs a mouth-shaped mouth with a closed mouth based on each of the weighted parameters W _{1 to} W ₄ output in time series. By changing the ratio at which the shape objects N _{1 to} N ₄ are mixed (multi-interpolation) (in the figure, addition operation is performed (indicated by (+)), the mouth shape corresponding to the voice input to the acoustic analysis unit 1 is obtained. A changing animation of A is generated.

Here, for the voice of the waveform shown in FIG.
FIG. 4 shows changes in the weighting parameters W _{1 to} W ₃ of the mouth shape objects N _{1 to} N ₃ of “A”, “I”, and “U”, respectively. The weight parameter takes a value in the range of 0 to 1, and when it is 1.0, the mouth shape object corresponding to the weight parameter is mixed with another mouth shape object as it is. Then, as the weighting parameter becomes a value smaller than 1.0, the mouth shape object for that weighting parameter is made into a shape that is substantially similar to the original shape, and is mixed with other mouth shape objects.

When the weight parameter is 0, the mouth shape object corresponding to the weight parameter is not mixed with other mouth shape objects. Therefore, in this case, the mouth shape object whose weighting parameter is 0 is not used, and the mouth shape of the animation is generated.

Next, a method for obtaining a conversion function (multiple regression equation) for converting the LPC cepstrum coefficients output in time series from the acoustic analysis unit 1 into weighted parameters by multiple regression analysis will be described. The multiple regression analysis will be briefly described.

The details of the multiple regression analysis are described in, for example, “A Story of Multivariate Analysis”, Arima, Ishimura, Tokyo Book Publishing, “Multivariate Analysis Method”, Okuno et al., Nikkan Giren, etc. Has been done.

The multiple regression analysis method is a kind of multivariate analysis method.
The predicted value Y of the target variable y with the smallest error is calculated using the result called the target variable y and the cause that influences it, called the explanatory variable x _q (q = 1, 2, ..., Q). The given formula (multiple regression formula) Y = a ₀ + a ₁ x ₁ + a ₂ x ₂ + ... + a _Q x _Q (1) is a method for predicting the target variable y.

In the multiple regression equation (1), a ₀ is called a constant term, and a _{1 to} a _Q are called regression coefficients (partial regression coefficients).

Now, consider a case where a result as an objective variate is obtained from the cause of the following set I as an explanatory variate.

Objective variate y ₁ ← Explanatory variables x ₁₁ , x ₂₁ , ..., x _Q1 Objective variate y ₂ ← Explanatory variables x ₁₂ , x ₂₂ , ..., x _Q2 ··· Objective variability y _i ← Explain Variables x _1i , x _2i , ..., x _Qi ··· Objective variable y _I ← Explanatory variables x _1I , x _2I , ..., x _QI

The predicted value Y of the target variable y with the smallest error
To obtain the multiple regression equation (1) that gives
The regression coefficients a _{1 to} a _Q and the constant term a ₀ are calculated so that −Y | becomes small.

The predicted value Y _i for the explanatory variable x _qi is given by the formula Y _i = a ₀ + a ₁ x _1i + a ₂ x _2i + ... + a _Q x _Qi from the equation (1), and therefore the target variable y _i and predicted value Y _i
The error E _i between and is given by the equation E _i = | y _i − (a ₀ + a ₁ x _1i + a ₂ x _2i + ... + a _Q x _Qi ) | (2).

In order to find a _{0 to} a _Q that minimizes the equation (2), the square error ε shown by the following equation is used based on the least square method.
It suffices to find a _{0 to} a _Q that minimizes. ε = ΣE _i ² However, Σ means the summation regarding _i (= 1, 2, ..., I).

In this case, the squared error ε is partially differentiated by a _{0 to} a _Q , and Q + 1 equations obtained as a result are set to 0, and Q + 1 simultaneous equations are solved to reduce the squared error ε to the minimum. A _{0 to} a _Q can be obtained.
It is known that this is equivalent to solving the simultaneous equations obtained as follows.

That is, the covariance of the explanatory variables x _qi and x _pi is s
_qp (where p = 1, 2, ... Q), the covariance of the explanatory variable x _qi and the objective variable y _i is s _qy , and the average value of the explanatory variable x _qi is x ′.
_{If q} and the average value of the target variables y _i are y ′, simultaneous equations s ₁ ² a ₁ + s ₁₂ a ₂ + ... + s _1Q a _Q = s _1y s ₂₁ a ₁ + s ₂ ² a ₂ + ... + S _2Q a _Q = s _2y ··· s _Q1 a ₁ + s _Q2 a ₂ + ・ ・ ・ + s _Q ² a _Q = s _Qy a ₀ = y '-(x' ₁ a ₁ + x ' ₂ a ₂ + ... · + x _'Q a _Q) by solving (3), can be obtained a ₀ to a _Q.

S _q ² represents the covariance s _{qq of} x _qi and x _qi , that is, the variance of x _qi . Also, since s _qp = s _pq , in obtaining simultaneous equations (3), s _qp and s are not _calculated for all p and q.
_You only need to find one of _pq .

In obtaining the multiple regression equation as the conversion function used in the weighting parameter conversion section 2, first, the acoustic analysis section 1 processes a speech signal having a phonological label as a learning sequence as described above. Acoustic analysis is performed to find the LPC cepstrum coefficient x _q up to the Qth order.

Further, a face image having a mouth shape corresponding to the audio signal for which the LPC cepstrum coefficient x _q is obtained is recorded on, for example, a video tape, and the mouth shape of the reproduced face image can be obtained. Weighting parameters w _{1 to} w ₄ of the _four mouth-shaped objects shown in 3 are obtained.

This makes it possible to associate the LPC cepstrum coefficient x _q with the weighting parameters w _{1 to} w _{4 at} time t when a certain phoneme is uttered.

Here, for the voice signal as the learning sequence, when the device is applied to Japanese, a word set or a sentence set including as many combinations of Japanese phonemes as possible is used for a plurality of speakers.

The LPC cepstrum coefficient x _q and the weighting parameters w _{1 to} w ₄ which are associated as described above are used as explanatory variables and objective variables, respectively, and the equations for the weighting parameters w _{1 to} w ₄ as the objective variables are expressed respectively. Create simultaneous equations corresponding to (3).

By solving the simultaneous equations, the LPC cepstrum coefficient x _q as an explanatory variable is
Predicted value W of the weighted parameters w _{1 to} w ₄ as the target variable
Conversion of the multiple regression equation for converting each of ₁ to W ₄ function _{W 1 = a 0W1 + a 1W1} x 1 + a 2W1 x 2 + ··· + a QW1 x Q W 2 = a 0W2 + a 1W2 x 1 + a 2W2 x 2 + _{··· + a QW2 x Q W 3} = a 0W3 + a 1W3 x 1 + a 2W3 x 2 + ··· + a QW3 x Q W 4 = a 0W4 + a 1W4 x 1 + a 2W4 x 2 + ··· + a QW4 x Q ( 4) is obtained.

In the weighted parameter conversion unit 2, the LPC cepstrum coefficient x _q output from the acoustic analysis unit 1 in time series.
Is a weighting parameter W by the conversion function shown in equation (4).
It is converted into each of _{1 to} W ₄ and sequentially output to the face image synthesizer 3.

Then, in the face image synthesizer 3, as described above, based on each of the weighting parameters W _{1 to} W ₄ output from the weighting parameter converter 2 in time series,
“A”, “I”, “U”, and mouth-shaped objects N _{1 to} N ₄ representing closed mouth shapes are mixed (multi-interpolated) by changing the ratio, which is input to the acoustic analysis unit 1. An animation in which the mouth shape changes is generated in synchronization with the voice.

As described above, since the conversion function as the multiple regression equation obtained by the multiple regression analysis is used,
It is possible to convert the LPC cepstrum coefficient as a sound acoustic parameter into a weighted parameter by a linear operation. Moreover, the amount of calculation is small, which allows
The processing speed can be increased, the device can be downsized, and the cost can be reduced.

Therefore, even when the present invention is applied to, for example, a workstation equipped with a voice input / output interface, it is possible to perform real-time processing of animation generation with a change in mouth shape only by software processing. .

Furthermore, when the present invention is applied to, for example, a voice mail reading process or a computer with low computing ability, parameters for synthesizing face images other than mouth are calculated in advance. By doing so, that is, by completing the face image of the part other than the mouth, it is possible to perform the real-time processing of generating the face image in which the mouth moves in synchronization with the input voice.

Further, in the present invention, since the weighted parameter obtained by converting the LPC cepstrum coefficient obtained from the natural voice (the voice actually uttered) is used,
A change in mouth shape can give a natural animation.

Further, in the present invention, an animation in which the mouth shape changes in synchronization with the voice uttered by the voice actor can be easily produced without drawing a plurality of cel images in which the mouth shape changes stepwise. .

In this embodiment, the case where the present invention is applied to animation production has been described.
The present invention can also be applied to other applications such as computer graphics (for example, the above-mentioned voice mail reading processing).

Further, in this embodiment, the LPC cepstrum coefficient is used as the acoustic parameter of the voice converted into the weighted parameter, but any other acoustic parameter can be used.

That is, for example, an analysis result obtained by spectrum analysis of voice by FFT (Fast Fourier Transform) and voice by BP
Filtered by F (bandpass filter), the output of the filter is full-wave rectified and smoothed, and sampled (bandpass filter bank analysis result) (each BPF
It is possible to use the average value of the power of the voice in the pass band of the above).

Further, as the acoustic parameter, for example, a formant may be used, but the spectral envelope of the voice such as the above-mentioned LPC cepstrum coefficient, spectrum analysis result by FFT, bandpass filter bank analysis result, etc. By using the acoustic parameter indicating the characteristic, the extraction accuracy is higher, and high-speed and stable acoustic analysis can be performed, so that a more accurate mouth shape animation can be realized.

That is, for example, it is difficult to extract an accurate formant of a consonant, but an LPC extracted from a consonant is difficult.
Acoustic parameters that represent spectral envelope characteristics, such as cepstrum coefficients, represent the characteristics of their consonants with some accuracy. Therefore, using the acoustic parameter representing the spectral envelope characteristic of the voice reflects a more accurate change in the mouth shape, as compared with the case of using the formant or the like.

[0056]

As described above, according to the present invention, a voice is acoustically analyzed, an acoustic parameter is calculated, and the acoustic parameter previously obtained by multiple regression analysis is used to change the mouth shape of an image. It is converted into a weighted parameter by a conversion function for converting into a weighted parameter. And
An image is created by changing a preset basic mouth shape based on the weighting parameter. Therefore, an image in which the mouth shape changes in synchronization with the voice can be obtained with a small amount of calculation.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of a voice / face image synchronization apparatus to which the present invention is applied.

FIG. 2 is a waveform diagram showing an example of voice input to the acoustic analysis unit 1.

FIG. 3 is a diagram showing a mouth shape object.

FIG. 4 is a diagram showing weighting parameters output in time series from the weighting parameter conversion unit 2 with respect to the voice waveform of FIG.

5 is a diagram for explaining the operation of the face image synthesizer 3. FIG.

[Explanation of symbols]

 1 Acoustic analysis unit 2 Weighted parameter conversion unit 3 Face image synthesizer

Claims (4)

[Claims] 1. An image creating method for creating an image synchronized with an input voice, wherein a conversion function for converting an acoustic parameter extracted from the voice into a weighting parameter for changing a mouth shape of the image Obtained in advance by regression analysis, by the conversion function, the acoustic parameter is converted to the weighted parameter, based on the weighted parameter, a preset basic mouth shape is changed, and the image is created. How to create an image. 2. The basic mouth shape is a mouth shape that pronounces “a”, “i”, and “u”, and a mouth shape with a closed mouth. Image creation method. 3. An image creating apparatus for creating an image in synchronization with an input voice, wherein an analyzing means for acoustically analyzing the voice and calculating an acoustic parameter, and the acoustic parameter previously obtained by multiple regression analysis A conversion means for converting the acoustic parameter into the weighting parameter by a conversion function for converting into a weighting parameter for changing the mouth shape of the image; and changing a preset basic mouth shape based on the weighting parameter. And an image creating device for creating the image. 4. The basic mouth shape is a mouth shape that pronounces “a”, “i”, and “u”, and a mouth shape with a closed mouth. Image creation device.