CN115116468A - Video generation method and device, storage medium and electronic equipment - Google Patents

Abstract

The embodiment of the invention discloses a video generation method, which comprises the following steps: acquiring a frame sequence to be selected; determining a target frame from a frame sequence to be selected according to the frame selection dimension; performing voice driving on a target frame based on a current voice signal to obtain a target video; the frame selection dimension comprises at least one of a first frame selection dimension and a second frame selection dimension. Through the scheme of the embodiment, the target frame meeting the voice driving requirement is obtained, and the subsequent voice driving effect is improved.

Description

A video generation method, device, storage medium and electronic device

technical field

This article relates to video generation technology, especially to a video generation method, device, storage medium and electronic device.

Background technique

The method of generating video by voice drive has been widely used in various fields. In the prior art, an unfiltered single static frame is usually used as an input frame, and a video is generated by voice driving. However, the voice driver has many requirements on the input frame. For example, the input frame needs to be clear in image quality, centered on the face, and neutral in expression. It is difficult to meet the requirements of the voice driver only based on unfiltered single-frame static frames.

SUMMARY OF THE INVENTION

Compared with the related art, the technical solution described in the present application obtains a target frame that meets the requirements of voice driving, which improves the effect of subsequent voice driving; at the same time, it solves the problem of missing facial details that may be caused by changes in expression coefficients in the process of voice driving. , to make the generated video more vivid and natural.

In order to achieve the purpose of the embodiment of the present invention, the embodiment of the present invention provides a video generation method, and the method may include:

Get the sequence of frames to be selected;

Determine a target frame from the to-be-selected frame sequence according to the frame selection dimension;

Based on the current voice signal, the target frame is voice-driven to obtain the target video;

Wherein, the frame selection dimension includes at least one of a first frame selection dimension and a second frame selection dimension.

In an exemplary embodiment of the present invention, determining the target frame from the to-be-selected frame sequence according to the frame selection dimension includes:

According to the frame selection dimension, obtain a preselected frame that satisfies the frame selection condition from the to-be-selected frame sequence, wherein the preselected frame is one or more frames;

When the preselected frame is one frame, the preselected frame is the target frame;

When the preselected frame is a multi-frame, the multi-frame preselected frame is fused to obtain the target frame;

Wherein, the frame selection condition includes at least one of a first frame selection condition and a second frame selection condition.

In an exemplary embodiment of the present invention, the fusion includes at least one of a first fusion or a second fusion.

In an exemplary embodiment of the present invention, the obtaining, according to the frame selection dimension, a preselected frame that satisfies the frame selection condition from the to-be-selected frame sequence includes:

Calculate the first dimension value of each frame in the to-be-selected frame sequence according to the first frame selection dimension; obtain the first dimension value that satisfies the first frame selection condition from the to-be-selected frame sequence the first preselected frame;

Wherein, the first preselected frame is one or more frames.

In an exemplary embodiment of the present invention, the first frame selection condition is that the first dimension value is within a first frame selection range.

In an exemplary embodiment of the present invention, when the first preselected frame is a frame, the first preselected frame is the target frame;

When the first pre-selected frame is a multi-frame, the first pre-selected frame of the multi-frame is first fused to obtain the target frame.

In an exemplary embodiment of the present invention, the obtaining, according to the frame selection dimension, a preselected frame that satisfies the frame selection condition from the to-be-selected frame sequence includes:

According to the second frame selection dimension, calculate the second dimension value of each frame in the to-be-selected frame sequence;

Obtain a second pre-selected frame whose second dimension value satisfies the second frame selection condition from the to-be-selected frame sequence;

Wherein, the second pre-selected frame is one or more frames.

In an exemplary embodiment of the present invention, when the second preselected frame is one frame, the second preselected frame is the target frame;

When the second pre-selected frame is a multi-frame, the target frame is obtained by performing a second fusion on the multi-frame second pre-selected frame.

In an exemplary embodiment of the present invention, the second frame selection condition is that the second dimension value or the second dimension comprehensive value is the lowest or the highest.

In an exemplary embodiment of the present invention, the obtaining, according to the frame selection dimension, a preselected frame that satisfies the frame selection condition from the to-be-selected frame sequence includes:

According to the second frame selection dimension, calculate the second dimension value of each frame in the first preselected frame;

Acquiring, from the first preselected frame, a second preselected frame whose second dimension value satisfies the second frame selection condition;

Wherein, the second pre-selected frame is one or more frames.

In an exemplary embodiment of the present invention, when the second preselected frame is one frame, the second preselected frame is the target frame;

When the second pre-selected frame is a multi-frame, the target frame is obtained by performing a second fusion on the multi-frame second pre-selected frame.

In an exemplary embodiment of the present invention, when the first preselected frame is a multi-frame, the second frame selection condition is that the second dimension value or the second dimension comprehensive value is the lowest or the highest.

In an exemplary embodiment of the present invention, when the first preselected frame is one frame, the second frame selection condition is that the second dimension value is within the second frame selection range.

In an exemplary embodiment of the present invention, performing voice driving on the target frame based on the current voice signal to obtain the target video includes:

According to the current voice signal, the corresponding driving expression coefficient is generated through the trained voice driving model;

Matching the target frame and the driving expression coefficient to generate a key frame;

Based on the to-be-selected frame sequence and the target frame, performing expression matching on the key frame to obtain a drive frame;

Successive said drive frames constitute said target video.

The embodiment of the present invention also provides a video generation device, which may include:

an acquisition unit, configured to acquire a sequence of frames to be selected;

a frame selection unit, configured to determine a target frame from the to-be-selected frame sequence according to the frame selection dimension;

A drive unit, configured to perform voice driving on the target frame based on the current voice signal, and obtain the target video

An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, implements the steps of any one of the above video generation methods .

The embodiment of the present invention also provides an electronic device, which may include:

processor; and

a memory for storing executable instructions for the processor;

Wherein, the processor is configured to execute the video generation method described in any one of the above by executing the executable instructions.

Through the above-mentioned solution of the embodiment of the present invention, the target frame that meets the voice driving requirements is obtained, which improves the effect of subsequent voice driving; at the same time, the problem of missing facial details that may be caused by the change of the expression coefficient during the voice driving process is solved, so that the generated video more vivid and natural.

Other features and advantages of the present application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present application. Other advantages of the present application may be realized and attained by the approaches described in the specification and drawings.

Description of drawings

The accompanying drawings are used to provide an understanding of the technical solutions of the present application, and constitute a part of the specification. They are used to explain the technical solutions of the present application together with the embodiments of the present application, and do not constitute a limitation on the technical solutions of the present application.

1 is a flowchart of a video generation method according to an embodiment of the present application;

2 is a flowchart of determining a target frame from a sequence of frames to be selected according to an embodiment of the present application;

3 is a flowchart of determining a target frame from a sequence of frames to be selected according to another embodiment of the present application;

4a is a schematic diagram of an eye feature point according to an embodiment of the present application;

4b is a schematic diagram of a mouth feature point according to an embodiment of the present application;

5 is a flowchart of determining a target frame from a sequence of frames to be selected according to yet another embodiment of the present application;

FIG. 6 is a flowchart of performing voice driving on a target frame to obtain a target video according to an embodiment of the present application;

7 is a flowchart of a video generation method in a video call according to an embodiment of the present application;

FIG. 8 is a block diagram of a video generating apparatus according to an embodiment of the present application.

Detailed ways

This application describes a number of embodiments, but the description is exemplary rather than restrictive, and it will be apparent to those of ordinary skill in the art that within the scope of the embodiments described in this application can be There are many more examples and implementations. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are possible. Unless expressly limited, any feature or element of any embodiment may be used in combination with, or may be substituted for, any other feature or element of any other embodiment.

This application includes and contemplates combinations with features and elements known to those of ordinary skill in the art. The embodiments, features and elements that have been disclosed in this application can also be combined with any conventional features or elements to form unique inventive solutions as defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other inventive arrangements to form another unique inventive arrangement defined by the claims. Accordingly, it should be understood that any of the features shown and/or discussed in this application may be implemented alone or in any suitable combination. Accordingly, the embodiments are not to be limited except in accordance with the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.

Furthermore, in describing representative embodiments, the specification may have presented methods and/or processes as a particular sequence of steps. However, to the extent that the method or process does not depend on the specific order of steps described herein, the method or process should not be limited to the specific order of steps described. Other sequences of steps are possible, as will be understood by those of ordinary skill in the art. Therefore, the specific order of steps set forth in the specification should not be construed as limitations on the claims. Furthermore, the claims directed to the method and/or process should not be limited to performing their steps in the order written, as those skilled in the art will readily appreciate that these orders may be varied and still remain within the spirit and scope of the embodiments of the present application Inside.

An embodiment of the present application provides a video generation method, as shown in FIG. 1 , the method includes:

S101: Obtain the frame sequence to be selected;

The sequence of frames to be selected may include: at least one of real-time video cache frames and pre-stored frames pre-shot by the user, and the sequence of frames to be selected includes no less than two frames to be selected, wherein the pre-stored frames pre-shot by the user can be selected by the user. The system prompts the captured frames based on different frame selection dimensions;

The frame to be selected needs to contain face information;

S102: Acquire a target frame in the frame sequence to be selected according to the frame selection dimension;

Based on the requirements of the voice-driven model for the target frame, the frame selection dimension may include at least one of a first frame selection dimension and a second frame selection dimension, wherein the first frame selection dimension may be a picture dimension, including face position, face At least one of orientation, body posture, and light. The second frame selection dimension can be at least one of image quality and facial features. , at least one of the dimensions of the mouth;

The frame selection dimension can be preset or automatically generated according to the needs of the voice-driven model;

The requirements of the voice-driven model for the target frame may include at least one of picture requirements, picture quality requirements, and facial features requirements, wherein the picture requirements include at least one of face position centering, face facing forward, neutral human posture, and moderate lighting. , the image quality requirements can include: clear images, facial features can include at least one of eyes open and mouth closed;

The target frame is one or more frames that satisfy all frame selection conditions in the frame sequence to be selected;

Specifically, step S102 may include:

S1021: According to the frame selection dimension, obtain a preselected frame that satisfies the frame selection condition from the to-be-selected frame sequence, wherein the preselected frame is one or more frames;

The frame selection condition is the condition that needs to be satisfied to meet the requirements of the voice-driven model for the target frame;

The frame selection condition may include at least one of the first frame selection condition and the second frame selection condition;

S1022: Determine whether the preselected frame is a frame;

S1023: when the preselected frame is one frame, the preselected frame of the one frame is the target frame;

S1024: when the preselected frame is multiple frames, fuse the multiple preselected frames to obtain the target frame;

The above-mentioned fusion includes at least one of the first fusion or the second fusion;

S103: Based on the current voice signal, voice-driven the target frame to generate a target video.

Through the method in this embodiment, a target frame that meets the voice driving requirements can be obtained, and the effect of subsequent voice driving can be improved.

An embodiment of the present application provides a method for obtaining a preselected frame that satisfies a frame selection condition from a sequence of to-be-selected frames. As shown in FIG. 2 , the method includes:

S201: Calculate the first dimension value of each frame in the frame sequence to be selected according to the first frame selection dimension;

In this embodiment, the first frame selection dimension is the picture dimension, which may include at least one of the position of the face, the orientation of the face, the posture of the human body, and the light;

Correspondingly, the first dimension value may include at least one of a face position value, a face orientation value, a human body posture value, and a light value;

In an exemplary embodiment, the method of calculating a face position value includes:

Based on the face feature points, obtain the center point bbox_center corresponding to the face bounding box in the frame to be selected, and calculate the horizontal and vertical coordinate ratio bbox_center_u/v of the center point bbox_center in the frame to be selected. The horizontal and vertical coordinate ratio bbox_center_u/v is is the face position value;

In an exemplary embodiment, the method for calculating a face orientation value includes:

Based on the face feature points, the face orientation angle (roll, yaw, pitch) of the frame to be selected is obtained, and the face orientation angle (roll, yaw, pitch) is the face orientation value;

In an exemplary embodiment, the method of calculating a human body pose value includes:

By comparing the joint points of the human body with the positive posture and the joint points of the human body to be selected, the relative relationship value T _val of the joint points of the human body is obtained, and the relative relationship value T _val of the joint points of the human body is the human body posture value;

In an exemplary embodiment, the method of calculating a ray value includes:

Calculate the proportion of pixels less than the underexposure brightness threshold and greater than the overexposure brightness threshold to obtain the underexposure ratio and overexposure ratio, and the underexposure ratio and overexposure ratio are the light value;

Among them, the under-exposure brightness threshold and the over-exposure brightness threshold can be preset according to requirements, or can be automatically generated by the system;

Obtain the ratio of the dark part of the frame to be selected by counting the brightness distribution of the pixels of the frame to be selected, and this ratio is the light value;

S202: Obtain a first pre-selected frame whose first dimension value satisfies the first frame selection condition from the frame sequence to be selected;

Wherein, the first frame selection condition is that the first dimension value is within the range of the first frame selection;

Corresponding to the first dimension, the first frame selection range may include at least one of a face position range, a face orientation range, a body posture range, and a light range;

In an exemplary embodiment, the face position range is,

TMin _{u/v <} bbox_center_u/v<TMax _u/v ;

Among them, _{bbox_center_u} /v is the face position value, TMin _u/v is the minimum threshold of the abscissa ratio, TMax _u/v is the maximum threshold of the abscissa ratio, TMin _u/v and TMax _u/v can be preset according to requirements It can also be automatically generated by the system;

The frame whose face position value is within the range of face position satisfies the requirement of face position centering;

In an exemplary embodiment, the face orientation range is,

roll<T _roll ,yaw<T _yaw ,pitch<T _pitch ;

Among them, (roll, yaw, pitch) is the face orientation value, (T _roll , T _yaw , T _pitch ) is the face orientation threshold value, (T _roll , T _yaw , T _pitch ) can be preset according to requirements, or can be set by the system Automatic generated;

Frames whose face orientation value is within the face orientation range satisfies the requirement of face orientation forward;

In an exemplary embodiment, the human body pose range is,

T _val <∈;

Among them, T _val is the human body posture value, ∈ is the human body posture threshold value, the joint points of the human body in the positive posture and the human body posture threshold value can be preset according to the requirements, or can be automatically generated by the system;

The frame whose human body pose value is within the range of human body pose meets the requirement of neutrality of human body pose;

In an exemplary embodiment, the ray range is,

Overexposure ratio≤overexposure threshold, underexposure ratio≤underexposure threshold;

Among them, the underexposure ratio and the overexposure ratio are light values, and the overexposure threshold and the underexposure threshold can be preset according to requirements, or can be automatically generated by the system;

Frames with light values within the light range meet the needs of moderate light;

The first preselected frame can be one or more frames;

If there is no frame that satisfies the first frame selection condition in the to-be-selected frame sequence, the user is prompted to shoot or upload an image to the to-be-selected frame sequence according to the first frame-selection condition, until there is a frame that meets the first frame selection condition in the to-be-selected frame sequence , the frame is the target frame;

S203: Determine whether the first preselected frame is a frame;

S204: when the first preselected frame is a frame, the first preselected frame of the one frame is the target frame;

S205: when the first pre-selected frame is a multi-frame, perform a first fusion on the first pre-selected frame of the multi-frame to obtain a target frame;

Specifically, the first fusion includes:

Taking any one of the first preselected frames of multiple frames as a reference frame, and other frames as matching frames;

Obtain the Harris corner point of the reference frame, which is recorded as the reference point;

Calculate the feature descriptor of the reference point;

Obtain the matching range of the matching frame, where the matching range can be the range of a circle obtained by taking the point corresponding to the reference point of the reference frame in the matching frame as the center of the circle and the matching distance as the radius, optionally, the matching distance is 5-15 pixels ;

Calculate the feature descriptors of the points within the matching range, and select the points closest to the feature descriptors of the reference points in the reference frame as the matching points;

Based on the reference point of the reference frame and the matching point of the matching frame, the homography matrix is obtained through projective transformation. Optionally, the homography matrix has 8 degrees of freedom. At this time, the homography can be obtained from at least 4 pairs of reference points and matching points. response matrix;

Based on the homography matrix, the pixel correspondence between the reference frame and the matching frame is obtained through matrix transformation and pixel interpolation;

Subtract the pixels of the reference frame and the matching frame correspondingly to obtain the absolute value of the pixel difference;

Compare the absolute value of the pixel difference with the pixel noise threshold to obtain the pixel weight;

According to the pixel weight, weighted average is performed on the corresponding pixels in the reference frame and the matching frame to obtain the target frame;

Through the first fusion, not only can the first preselected frames of multiple frames be fused into one target frame, but also the target frame after the first fusion has higher spatial resolution, more obvious information performance and lower noise;

Through the method in this embodiment, a target frame that meets the voice driving requirement can be obtained, which improves the effect of subsequent voice driving.

An embodiment of the present application provides a method for obtaining a preselected frame that satisfies a frame selection condition from a sequence of frames to be selected. As shown in FIG. 3 , the method includes:

S301: Calculate the second dimension value of each frame in the frame sequence to be selected according to the second frame selection dimension;

In this embodiment, the second frame selection dimension is at least one of an image quality dimension or a facial feature dimension, the image quality dimension may include blurriness, and the facial feature dimension may include at least one of an eye dimension and a mouth dimension;

Correspondingly, the second dimension value may include at least one of ambiguity value and facial feature dimension value (eye dimension value, mouth dimension value);

In an exemplary embodiment, the method of calculating the ambiguity value may include:

Gaussian blur is performed on each frame in the sequence to be selected to obtain its Gaussian blurred image;

Perform horizontal gradient calculation and vertical gradient calculation on each frame in the frame sequence to be selected and its Gaussian blurred image to obtain their horizontal gradient value and vertical gradient value;

Based on the above-mentioned horizontal gradient value and vertical gradient value, calculate the horizontal gradient difference and vertical gradient difference of each frame in the frame sequence to be selected and its Gaussian blurred image;

Summing the above horizontal gradient difference and vertical gradient difference to obtain the ambiguity value;

The relationship between the blurriness value calculated by the above method and the clarity of the frame is that the higher the blurriness value, the blurrier the frame, and the lower the blurriness value, the clearer the frame;

The present application does not limit the method of calculating the ambiguity value, and other methods can also be selected to calculate the ambiguity value. In other methods, the lower the ambiguity value may be, the more blurred the frame, and the higher the ambiguity value, the clearer the frame;

In an exemplary embodiment, as shown in FIG. 4a, the method for calculating the eye dimension value may be:

eye_val=1-len(pt ₄₂ -pt ₄₈ )/len(pt ₃₉ -pt ₄₅ )

Among them, pt ₄₂ , pt ₄₈ , pt ₃₉ , pt ₄₅ are the eye feature points obtained based on the face feature points, len(pt ₄₂ -pt ₄₈ ) is the distance between pt ₄₂ and pt ₄₈ , len(pt ₃₉ - pt ₄₅ ) is the distance between pt ₃₉ and pt ₄₅ ;

The relationship between the eye dimension value and the eye calculated by the above method is that the lower the eye_val, the higher the degree of eye opening;

This application does not limit the method of calculating the eye dimension value, and other methods can also be selected to calculate the eye dimension value. In other methods, the lower the eye dimension value may be, the lower the degree of eye opening, and the higher the eye dimension value. , the higher the degree of eye opening;

In an exemplary embodiment, as shown in FIG. 4b, the method for calculating the mouth dimension value may include:

mouth_val=len(pt ₈₉ -pt ₉₃ )/len(pt ₈₇ -pt ₉₁ )

Among them, pt ₈₉ , pt ₉₃ , pt ₈₇ , pt ₉₁ are the mouth feature points obtained based on the face feature points, len(pt ₈₉ -pt ₉₃ ) is the distance between pt ₈₉ and pt ₉₃ , len(pt ₈₇ - pt ₉₁ ) is the distance between pt ₈₇ and pt ₉₁ ;

The relationship between the mouth dimension value calculated by the above method and the mouth is that the lower the mouth_val, the higher the degree of mouth closure;

This application does not limit the method of calculating the mouth dimension value, and other methods can also be selected to calculate the mouth dimension value. The higher the degree of mouth closure;

Corresponding to the second frame selection dimension, the frame selection conditions include at least one of clear image quality, open eyes, and closed mouth;

When the first preselected frame is multiple frames, the method includes:

S302: Obtain a second pre-selected frame whose second dimension value satisfies the second frame selection condition from the frame sequence to be selected;

When the first preselected frame is multiple frames, the second frame selection condition is the lowest or highest second dimension value or the second dimension comprehensive value;

When the second frame selection condition is the lowest or highest value of the second dimension;

Specifically, the method includes:

S3021: Obtain the frame with the lowest or highest second dimension value in the frame sequence to be selected as the second pre-selected frame;

In an exemplary embodiment, when the second frame selection dimension includes the ambiguity, acquiring the frame with the lowest ambiguity value in the frame sequence to be selected is the second pre-selected frame;

In an exemplary embodiment, when the second frame selection dimension includes the eye dimension, acquiring the frame with the lowest eye dimension value in the frame sequence to be selected is the second pre-selected frame;

In an exemplary embodiment, when the second frame selection dimension includes the mouth dimension, the frame with the lowest mouth dimension value in the frame sequence to be selected is obtained as the second pre-selected frame;

In an exemplary embodiment, when the second frame selection dimension includes ambiguity and eye dimension, the frame with the lowest ambiguity value and the frame with the lowest eye dimension value in the frame sequence to be selected are obtained as the second pre-selected frame, and the fuzzy The frame with the lowest degree value and the frame with the lowest eye dimension value can be the same frame or different frames;

In an exemplary embodiment, when the second frame selection dimension includes ambiguity and mouth dimension, the frame with the lowest ambiguity value and the frame with the lowest mouth dimension value in the frame sequence to be selected are obtained as the second pre-selected frame, and the fuzzy The frame with the lowest degree value and the frame with the lowest mouth dimension value can be the same frame or different frames;

In an exemplary embodiment, when the second frame selection dimension includes the eye dimension and the mouth dimension, the frame with the lowest eye dimension value and the frame with the lowest mouth dimension value in the frame sequence to be selected are obtained as the second pre-selected frame. , the frame with the lowest eye dimension value and the frame with the lowest mouth dimension value can be the same frame or different frames;

In an exemplary embodiment, when the second frame selection dimension includes the ambiguity, the eye dimension, and the mouth dimension, obtain the frame with the lowest ambiguity value, the frame with the lowest eye dimension value, and the mouth in the frame sequence to be selected. The frame with the lowest dimension value is the second pre-selected frame, and the frame with the lowest ambiguity value, the frame with the lowest eye dimension value, and the frame with the lowest mouth dimension value can be the same frame or different frames;

In some embodiments, it is possible that the higher the blurriness value, the clearer the frame, the higher the eye dimension value, the higher the eye opening degree, the higher the mouth dimension value, the higher the mouth closing degree. In this case, obtain At least one of the frame with the highest ambiguity value, the frame with the highest eye dimension value, and the frame with the highest mouth dimension value in the frame sequence to be selected is the second preselected frame;

Since the second frame selection dimension contains a variety of frame selection conditions, and different evaluation indicators often have different dimensions and dimensional units, such a situation will affect the analysis results. In order to eliminate the dimensional influence between the indicators, this paper The application introduces the second dimension comprehensive value;

When the second frame selection condition is the lowest or highest comprehensive value of the second dimension;

Specifically, the method includes:

S3022: Calculate the second dimension comprehensive value of each frame in the frame sequence to be selected;

The second dimension comprehensive value may be a weighted value of the second dimension value;

S3023: Obtain the frame with the lowest or highest comprehensive value of the second dimension in the frame sequence to be selected as the second pre-selected frame;

In an exemplary embodiment, when the second frame selection dimension includes the ambiguity, the eye dimension and the mouth dimension, the weighted value of the second dimension of each frame in the frame sequence to be selected is calculated to obtain the integrated value of the second dimension. , obtaining the frame with the lowest or highest comprehensive value of the second dimension in the frame sequence to be selected as the second pre-selected frame;

The second preselected frame can be one or more frames;

S303: Determine whether the second preselected frame is a frame;

S304: when the second preselected frame is a frame, the second preselected frame of the one frame is the target frame;

S305: When the second preselected frame is a multi-frame, perform a second fusion on the second preselected frame of the multi-frame to obtain the target frame;

Specifically, the second fusion includes:

Based on the face feature points, obtain the face deviation value of the second pre-selected frame of multiple frames;

Compare the face deviation value with the fusion threshold;

When the face deviation value is less than the fusion threshold, the optimal fusion boundary is obtained based on the face feature points;

According to the optimal fusion boundary, perform facial features fusion on the second pre-selected frames of multiple frames to obtain the target frame;

When the face deviation value is not less than the fusion threshold, obtain the facial features correspondence of the second pre-selected frames of multiple frames through affine transformation;

Perform facial feature fusion on the second pre-selected frame of multiple frames based on the above-mentioned facial feature correspondence to obtain the target frame;

Among them, the fusion threshold can be preset according to requirements, or can be automatically generated by the system;

Through the method in this embodiment, a target frame that meets the voice driving requirement can be obtained, which improves the effect of subsequent voice driving.

An embodiment of the present application provides a method for obtaining a pre-selected frame that satisfies a frame selection condition from a to-be-selected frame sequence according to the frame selection dimension. As shown in FIG. 5 , the method includes:

S401: Calculate the second dimension value of each frame in the first pre-selected frame according to the second frame selection dimension;

The first preselected frame can be one or more frames;

S402: Obtain a second preselected frame whose second dimension value satisfies the second frame selection condition from the first preselected frame;

Specifically, S402 includes:

S4021: Determine whether the first preselected frame is a multi-frame;

When the first preselected frame is multiple frames, the second frame selection condition is the lowest or highest second dimension value or the second dimension comprehensive value;

S4022: Obtaining the second dimension value or the lowest or highest frame of the second dimension comprehensive value from the first preselected frame of multiple frames is the second preselected frame;

When the first preselected frame is one frame, the second frame selection condition is that the second dimension value is within the second frame selection range;

S4023: Determine whether the second dimension value of the first preselected frame is within the range of the second selected frame;

The second frame selection range may include at least one of the blurriness range and the facial feature range, and the facial feature range may include at least one of the eye range and the mouth range;

In an exemplary embodiment, the ambiguity range is: ambiguity value < ambiguity threshold;

In an exemplary embodiment, the ambiguity range is: ambiguity value > ambiguity threshold;

In an exemplary embodiment, the range of facial features is: dimension value of facial features < threshold of facial features;

In an exemplary embodiment, the range of facial features is: facial feature dimension value>facial feature threshold;

The facial features threshold may include at least one of an eye threshold and a mouth threshold;

The ambiguity threshold and facial feature threshold can be preset according to requirements, or can be automatically generated by the system;

S4024: when the second dimension value of the first pre-selected frame is within the range of the second pre-selected frame, the first pre-selected frame is the second pre-selected frame;

In an exemplary embodiment, the frame selection range includes the blurriness range and the facial features range. If the blurriness value of the first preselected frame is within the blurriness range, and the facial feature dimension value is within the facial feature range, then the first preselected frame satisfies the first preselected frame. Two frame selection conditions, the first pre-selection frame is the second pre-selection frame;

S4025: when the second dimension value of the first preselected frame is not within the range of the second selected frame, obtain a third preselected frame from the sequence of frames to be selected;

The third preselected frame is the frame with the lowest or highest second dimension value or the second dimension comprehensive value in the frame sequence to be selected;

In an exemplary embodiment, the frame selection range includes a blurriness range and a facial feature range. If the blurriness value of the first pre-selected frame is not within the blurriness range, but the facial feature dimension value is within the facial feature range, the first pre-selected frame If the two-dimensional value is not within the range of the second frame selection, obtain the frame with the lowest or highest ambiguity value in the frame sequence to be selected, and this frame is the third pre-selected frame;

In an exemplary embodiment, the frame selection range includes the blurriness range and the facial features range. If the blurriness value of the first preselected frame is not within the blurriness range, and the facial feature dimension value is not within the facial features range, then the first preselected frame If the two-dimensional value is not within the range of the second frame selection, obtain the frame with the lowest or highest ambiguity value and the frame with the lowest or highest facial feature dimension value in the frame sequence to be selected, or obtain the ambiguity value and facial feature dimension value in the frame sequence to be selected. The frame with the lowest or highest comprehensive value, this frame is the third pre-selected frame;

The third preselected frame can be one or more frames;

S4026: Fusion of the first pre-selected frame and the third pre-selected frame to obtain a pre-selected fusion frame;

S4027: Determine whether the second dimension value of the preselected fusion frame is within the range of the second selected frame;

If the value of the second dimension of the preselected fusion frame is within the range of the second selected frame, the preselected fusion frame is the second preselected frame;

If the second dimension value of the preselected fusion frame is not within the range of the second selected frame, the user will be prompted to shoot or upload an image according to the second selected frame condition, and the image will be used as the third preselected frame to be fused with the first preselected frame until the preselected frame is obtained. The second dimension value of the fusion frame is within the range of the second frame selection to obtain the second pre-selection frame;

The fusion includes at least one of the first fusion and the second fusion;

S403: Determine whether the second preselected frame is a frame;

S404: when the second preselected frame is one frame, the second preselected frame of the one frame is the target frame;

S405: when the second pre-selected frame is a multi-frame, perform a second fusion on the second pre-selected frame of the multi-frame to obtain the target frame;

Through the method in this embodiment, a target frame that meets the requirements of the voice driving model can be obtained, which improves the effect of subsequent voice driving.

The embodiment of the present application provides a method for obtaining a target video by voice-driving a target frame based on a current voice signal. As shown in FIG. 6 , the method includes:

S501: Train a voice-driven model;

Specifically, step S501 includes:

S5011: Obtain training materials;

The training material needs to include voice information and its corresponding expression coefficient information;

The training material can be video material, which needs to include voice information and image information, wherein the image information needs to include facial expression information;

The video material can be a pre-recorded video or a video crawled from the Internet;

S5012: Collect speech signal samples in the training material and their corresponding expression coefficient samples;

The speech signal sample is a time-series signal, which can be a speech signal or a spectral feature of the speech signal, for example, a Melp feature;

When the training material is a video material, specifically, step S5012 may include:

According to the frame rate of the training material, extract the speech signal samples and their corresponding expression information in the training material;

Based on the expression information, obtain the expression coefficient corresponding to the speech signal sample;

Filter and synthesize the expression coefficients to obtain expression coefficient samples;

S5013: Train a voice-driven model based on voice signal samples and expression coefficient samples;

Specifically, 1D convolutional network training can be performed on speech signal samples and expression coefficient samples; the speech signal samples can also be converted into 2D images, and 2D convolutional network training can be performed on speech signal samples and expression coefficient samples; LSTM ( Long short-term memory, Long short-term memory) network for auxiliary training; Transform network can also be used for training;

The loss function Loss can be calculated directly using the expression coefficient samples, or the expression coefficient samples can be restored to grids for Loss training;

S502: According to the current voice signal, generate a corresponding driving expression coefficient through the trained voice driving model;

S503: Match the target frame with the continuous driving expression coefficient to generate a key frame;

Specifically, step S503 may include:

S5031: Preprocess the target frame;

Preprocessing includes: foreground character segmentation, character depth estimation and 3D face reconstruction, foreground character segmentation to obtain foreground mask map, character depth estimation to obtain character depth map, 3D face reconstruction to obtain 3D face model;

S5032: Obtain a face driving model according to the driving expression coefficient;

S5033: Obtain key frames based on the target frame and the face driving model;

Specifically, step S5033 includes:

According to the foreground mask map, extract the contour of the foreground area in the target frame;

According to the character depth map, sample the corresponding depth of the character in the target frame;

Taking the outline of the foreground area as the boundary, the foreground area of the target frame is subjected to Delaunay triangulation to obtain the character 3D mesh B _s in the projection space;

Remove the face area on the character 3D mesh B _s to obtain mesh B′ _S ;

Based on the 3D reconstruction of the face, the projection matrix P is obtained, and the face deformation source grid is transformed into the projection space to obtain the 3D face model F _s ;

Merge the 3D face model F _s with the mesh B′ _S , and connect the seam part of the boundary between the two through triangulation to obtain the deformation source mesh M _s ;

Transform the face-driven model into the projection space through the projection matrix P, and obtain the face-driven model F _t in the projection space;

Applying all the vertex positions in the face driving model F _t to the corresponding vertices of the 3D face model F _s in the deformation source mesh M _s to obtain the face mesh M _t ;

the non-face region U _t =M _t /F _t in the face mesh M _t , which corresponds to U _s =M _s /F _{s on the deformed source mesh M s ;}

则U_s、U_t的内边界分别为

外边界分别为

其中

Take the boundaries of F _s and F _t respectively

Then the inner boundaries of U _s and U _t are respectively

The outer boundaries are

in

对应顶点的位置相同，将其作为固定锚点，而

对应顶点的位置不同，将其作为移动锚点；Adjust the vertex positions in U _t by optimizing the grid weighted Laplace energy, so that F _t has a smooth and continuous transition in the face area. Here,

The positions of the corresponding vertices are the same, and they are used as fixed anchors, while

The positions of the corresponding vertices are different, and they are used as movement anchors;

的测地距离d，以1/d²为系数估计该点权重，迭代优化得到平滑非人脸区域网格U′_t，并有平滑变形目标网格M′_t＝U′_t∩F_t；Compute the vertices of U _s to

The geodesic distance d is estimated, the weight of the point is estimated with 1/d ² as the coefficient, and the smooth non-face area grid U' _t is obtained by iterative optimization, and there is a smooth deformation target grid M' _t =U' _t ∩F _t ;

For the target pixel obtained by rendering M' _t into the image space, the barycentric coordinates of the pixel corresponding to the grid M' _t during rasterization can be obtained, and by applying the coordinates to M _s , a point p on the surface of M _s can be obtained '_s;

Project the point p _'s onto the preprocessed target frame to get the corresponding source pixel;

The coordinate key frame is obtained by inversely interpolating the offset of the target pixel coordinates and the source pixel coordinates in the image space;

Based on the coordinate key frame, the key frame is obtained through the image warp algorithm of the least squares algorithm;

S504: Based on the to-be-selected frame sequence and the target frame, perform expression matching on the key frames to obtain a driving frame;

When the expression matching includes mouth matching, step S504 includes:

S5041: Obtain the facial expression coefficient of each frame in the frame sequence to be selected;

S5042: Obtain a face model corresponding to each frame in the frame sequence to be selected based on the facial expression coefficient;

S5043: Calculate the mouth deviation between the face model corresponding to each frame in the frame sequence to be selected and the face driving model;

S5044: Obtain the frame corresponding to the face model with the smallest mouth deviation as a rendering frame;

S5045: Use the rendering frame to render the key frame to obtain the driving frame;

由真实风格信息

和结构信息z_geo得到具有真实嘴部纹理质感和牙齿结构的驱动帧；In an exemplary embodiment, using the rendered frame to render the key frame includes: extracting structural information z _geo and style information z _style of the mouth in the key frame, and at the same time, extracting the real style information of the mouth in the rendered frame

by real style info

and structural information z _geo to get driving frames with real mouth texture and tooth structure;

When the expression matching includes eye matching, step S504 includes:

S5046: Obtain the eye opening range based on the driving expression coefficient;

S5047: Input the eye opening range and the target frame into the cGAN network, and output the eye image corresponding to the eye opening range;

S5048: Match the eye image with the key frame to obtain the driving frame;

S505: Continuous driving frames constitute a target video;

With the method in this embodiment, the problem of missing details of the mouth (eg, inside the cavity, teeth) that may be caused by the change of the expression coefficient is solved, so that the generated video is more vivid and natural.

An embodiment of the present application provides a video generation method in a video call, as shown in FIG. 7 , the method includes:

S601: monitor the real-time network bandwidth of the video call;

S602: Determine whether the real-time network bandwidth is less than a network threshold;

The network threshold can be preset according to requirements, or can be automatically generated by the system;

When the real-time network bandwidth is less than the network threshold, the video call freezes, and the video generation method includes:

S603: Obtain the frame sequence to be selected;

The frame sequence to be selected may include: at least one of the video buffer frame before the freeze and the pre-stored frame pre-shot by the user, and the frame sequence to be selected includes no less than two frames to be selected;

S604: Determine the target frame from the frame sequence to be selected according to the frame selection dimension;

S605: Based on the current voice signal, voice-driven the target frame to obtain the target video;

The current voice signal is the voice signal of the user after the video freezes;

S606: Switch the screen of the video call to the target video;

S607: When the real-time network bandwidth is not less than the network threshold, switch back to the video call;

With the method in this embodiment, when the network bandwidth is insufficient for the user, the picture of the video call is still natural and smooth.

The embodiment of the present application provides a video generation apparatus 10, as shown in FIG. 8, the apparatus includes:

The acquisition unit 100 is configured to acquire the frame sequence to be selected;

The frame selection unit 200 is configured to determine the target frame from the frame sequence to be selected according to the frame selection dimension;

The driving unit 300 is configured to perform voice driving on the target frame based on the current voice signal to obtain the target video.

Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the video generation method described in any of the preceding embodiments are implemented.

Embodiments of the present application further provide an electronic device, including a processor and a memory, where the memory is used to store executable instructions of the processor; wherein the processor is configured to execute the execution as described in any of the previous embodiments by executing the executable instructions. The video generation method described above.

The above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are only illustrative, for example, the division of the units may be a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims (17)

1. a video generation method, is characterized in that, comprises: Get the sequence of frames to be selected; Determine a target frame from the to-be-selected frame sequence according to the frame selection dimension; Based on the current voice signal, the target frame is voice-driven to obtain the target video; Wherein, the frame selection dimension includes at least one of a first frame selection dimension and a second frame selection dimension. 2. The video generation method according to claim 1, wherein, according to the frame selection dimension, determining a target frame from the frame sequence to be selected, comprising: According to the frame selection dimension, obtain a preselected frame that satisfies the frame selection condition from the to-be-selected frame sequence, wherein the preselected frame is one or more frames; When the preselected frame is one frame, the preselected frame is the target frame; When the preselected frame is a multi-frame, the multi-frame preselected frame is fused to obtain the target frame; Wherein, the frame selection condition includes at least one of a first frame selection condition and a second frame selection condition. 3. The video generation method according to claim 2, wherein the fusion comprises at least one of a first fusion or a second fusion. 4. The video generation method according to claim 3, wherein, according to the frame selection dimension, obtaining a preselected frame that satisfies the frame selection condition from the to-be-selected frame sequence, comprising: Calculate the first dimension value of each frame in the to-be-selected frame sequence according to the first frame selection dimension; Obtain a first pre-selected frame whose first dimension value satisfies the first frame selection condition from the to-be-selected frame sequence; Wherein, the first preselected frame is one or more frames. 5. video generation method according to claim 2, is characterized in that, The first frame selection condition is that the first dimension value is within the first frame selection range. 6. video generation method according to claim 4, is characterized in that, When the first preselected frame is one frame, the first preselected frame is the target frame; When the first pre-selected frame is a multi-frame, the first pre-selected frame of the multi-frame is first fused to obtain the target frame. 7. The video generation method according to claim 3, wherein, according to the frame selection dimension, obtaining a preselected frame that satisfies the frame selection condition from the frame sequence to be selected, comprising: According to the second frame selection dimension, calculate the second dimension value of each frame in the to-be-selected frame sequence; Obtain a second pre-selected frame whose second dimension value satisfies the second frame selection condition from the to-be-selected frame sequence; Wherein, the second pre-selected frame is one or more frames. 8. video generation method according to claim 7, is characterized in that: When the second preselected frame is one frame, the second preselected frame is the target frame; When the second pre-selected frame is a multi-frame, the target frame is obtained by performing a second fusion on the multi-frame second pre-selected frame. 9. video generation method according to claim 2, is characterized in that, The second frame selection condition is that the second dimension value or the second dimension comprehensive value is the lowest or the highest. 10. The video generation method according to claim 4, wherein, according to the frame selection dimension, obtaining a preselected frame that satisfies a frame selection condition from the to-be-selected frame sequence, comprising: According to the second frame selection dimension, calculate the second dimension value of each frame in the first preselected frame; Acquiring, from the first preselected frame, a second preselected frame whose second dimension value satisfies the second frame selection condition; Wherein, the second pre-selected frame is one or more frames. 11. The video generation method according to claim 10, wherein, When the second preselected frame is one frame, the second preselected frame is the target frame; When the second pre-selected frame is a multi-frame, the target frame is obtained by performing a second fusion on the multi-frame second pre-selected frame. 12. The video generation method according to claim 10, wherein, When the first preselected frame is multiple frames, the second frame selection condition is that the second dimension value or the second dimension comprehensive value is the lowest or the highest. 13. The video generation method according to claim 10, wherein, When the first preselected frame is one frame, the second frame selection condition is that the second dimension value is within the second frame selection range. 14. The video generation method according to claim 1 , wherein, based on the current voice signal, the target frame is voice-driven to obtain the target video, comprising: According to the current voice signal, the corresponding driving expression coefficient is generated through the trained voice driving model; Matching the target frame and the driving expression coefficient to generate a key frame; Based on the to-be-selected frame sequence and the target frame, performing expression matching on the key frame to obtain a drive frame; Successive said drive frames constitute said target video. 15. A video generation device, comprising: an acquisition unit, configured to acquire a sequence of frames to be selected; a frame selection unit, configured to determine a target frame from the to-be-selected frame sequence according to the frame selection dimension; The driving unit is configured to perform voice driving on the target frame based on the current voice signal to obtain the target video. 16. A computer-readable storage medium on which a computer program is stored, the computer program implementing the steps of the method according to any one of claims 1 to 14 when executed by a processor. 17. An electronic device, characterized in that, comprising: processor; and a memory for storing executable instructions for the processor; Wherein, the processor is configured to perform the video generation method of any one of claims 1 to 14 by executing the executable instructions.

Images