KR20240139282A - Method and apparatus for generating face harmonization image based on feature fusion for visitor experiential exhibition - Google Patents

Abstract

An embodiment of the present invention discloses a method and device for generating a style fusion image for an experiential exhibition for visitors, the method including the steps of extracting facial identity features based on facial key features of an input image, obtaining facial attribute features of a fusion target image, obtaining fusion features by fusion of the facial attribute features of the fusion target face and the facial identity features of the input image, generating a fusion image based on the fusion features, and generating a style fusion image by converting the fusion image into a facial coordinate system of the fusion target image.

Description

{METHOD AND APPARATUS FOR GENERATING FACE HARMONIZATION IMAGE BASED ON FEATURE FUSION FOR VISITOR EXPERIENTIAL EXHIBITION}

The present invention relates to a method for extracting features from a face of an input image and a target person's face of an exhibited target image for a visitor experience exhibition, and then generating a fused face from features obtained by fusing the two features.

Often, applying the texture, color, and other painting styles of the target image (painting) to the entire input image (real-life image) is called Style Transfer. In contrast, synthesizing a portion of the region of interest (Region of Interest) in the input real-life image to the target image and adjusting the texture, color, etc. of the target image so that there is no sense of incongruity is called Fusion Generation. More specifically, when the target image is a real-life image, it is called Image Harmonization, and when the target image is a painting, it is called Style Harmonization.

In the conventional technology, in the face style fusion generation that fuses a face in an input real-life image with a person in a target conversation image, the existing technologies fuse faces by combining face replacement and style transfer based on facial landmarks. However, in the face replacement process, artifacts occur at the face boundary, and especially when the difference in facial pose between the two faces is large, this artifact problem is aggravated.

In addition, when there is an error in detecting the main facial features, the quality of the fusion generation result is degraded, and when the size of the face in the target conversation image is small, there is still a problem that the fusion result image does not reflect the style, such as texture and color, well, resulting in a deterioration in the fusion quality.

The purpose of the present invention is to provide a method for generating face style fusion based on feature fusion that is robust to facial area landmark detection errors and changes in face size within an image and to solve the problem of facial area boundary artifacts for exhibition experiential display by visitors in art exhibition halls such as art galleries.

An embodiment of the present invention discloses a method and device for generating a style fusion image for an experiential exhibition for visitors, the method including: (a) extracting facial identity features based on facial key features of an input image, (b) obtaining facial attribute features of a fusion target image, (c) obtaining fusion features by fusion of the facial attribute features of the fusion target face and the facial identity features of the input image, (d) generating a fusion face image based on the fusion features, and (e) generating a style fusion image by converting the fusion face image into a facial coordinate system of the fusion target image.

In addition, the step (b) may include a step of extracting facial key features of the fusion target image, a step of performing face alignment based on the extracted facial key features, and a step of obtaining facial attribute features based on the aligned face.

In addition, the step of performing the face alignment may include a step of performing the face alignment based on predefined facial normal coordinates based on facial key feature points of the fusion target image.

In addition, the step (a) may further include a step of performing face alignment of facial key feature points of the input image based on predefined facial normal coordinates.

Additionally, the facial identification features may refer to facial features of the input image.

Additionally, the facial identity features can be adjusted based on the channel-by-channel statistical characteristics of the facial identity features of the fusion target image.

Additionally, the facial attribute features may include at least one of facial pose, expression, lighting, background, and other object information of the fusion target face.

In addition, the step (c) may include a step of matching the spatial resolution of the facial attribute features and the facial identity features using image interpolation, and a step of obtaining a fused feature by assigning to each pixel a value obtained by summing or multiplying each pixel existing in the spatial resolution of the facial attribute features of the fused target image and the facial identity features of the input image.

In addition, the step (d) may include a step of generating a fused face image when the fused feature is input to a face generation model, and the face generation model may include a model trained to generate the fused face image from the fused feature using a generator during training, and trained to distinguish the face of the fused target image from the generated fused face image using a discriminator.

In addition, the step (e) may further include a step of synthesizing the fused face image and the face in the fused target image using a Gaussian mask to alleviate an artifact existing at the boundary.

According to an embodiment of the present invention, instead of performing style fusion after replacing face regions as in the prior art, by fusion of features extracted from two face regions and then generating a new face region from the fused features, a style fusion technology that is robust to differences in facial poses, errors in major facial feature points, and the size of the face region in the fusion target image can be provided.

Additionally, the sharpness of the facial area in the resulting image can improve the fusion quality.

According to an embodiment of the present invention, an experiential exhibition in which the face of a viewer is fused with a character in a work of art is possible in an art exhibition hall such as an art museum, thereby improving the quality of content provided, enhancing viewer satisfaction, and providing differentiated content.

Figure 1 illustrates a computing device (100) according to one embodiment of the present invention.
Figure 2 illustrates a preprocessing process of a fusion target image for generating a fusion image according to an embodiment of the present invention.
FIG. 3 is a block diagram showing a process for generating a facial style fusion image for a viewer experience exhibition according to an embodiment of the present invention.
FIG. 4 illustrates an example of extracting key facial features of a fusion target image according to an embodiment of the present invention.
Figure 5 illustrates an example of face alignment according to an embodiment of the present invention.
Figure 6 illustrates a method for obtaining fusion features according to an embodiment of the present invention.
Figure 7 illustrates a process of learning a face generation model according to an embodiment of the present invention.
FIG. 8 is an example showing a process for improving the sharpness of a fused facial image according to an embodiment of the present invention.
Figure 9 is an example showing a fusion image generation process according to an embodiment of the present invention.
Figure 10 illustrates an artifact mitigation process according to an embodiment of the present invention.

The technology described below can have various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to specific embodiments, and it should be understood that all modifications, equivalents, or substitutes included in the spirit and scope of the technology described below are included.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element in between.

Although the terms first, second, A, B, etc. may be used to describe various components, these components are not limited by these terms, and are only used to distinguish one component from another. For example, without departing from the scope of the technology described below, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and/or includes any combination of a plurality of related described items or any item among a plurality of related described items.

As used herein, the singular expressions should be construed to include the plural expressions unless the context clearly dictates otherwise, and the term "comprises" and the like should be understood to mean the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Before going into a detailed description of the drawings, it should be made clear that the division of components in this specification is only a division based on the main function of each component. In other words, two or more components to be described below may be combined into one component, or one component may be divided into two or more components with more detailed functions. In addition to its own main function, each component to be described below may additionally perform some or all of the functions of other components, and of course, some of the main functions of each component may be exclusively performed by other components.

In addition, in performing a method or method of operation, each process constituting the method may occur in a different order from the stated order unless the context clearly states a specific order. That is, each process may occur in the same order as the stated order, may be performed substantially simultaneously, or may be performed in the opposite order.

Hereinafter, it is described that a computing device (100) performs an operation according to an embodiment of the present invention. The computing device (100) is a device that processes input data consistently and performs necessary operations according to a specific model or algorithm. For example, the computing device may be implemented in the form of a PC, a server on a network, a smart device, a chipset with a design program embedded therein, etc.

Figure 1 illustrates a computing device (100) according to one embodiment of the present invention.

FIG. 1 illustrates a block diagram of a computing device related to one embodiment of the present disclosure. The components of the computing device (100) illustrated in FIG. 1 are exemplary. Only some of the components illustrated in FIG. 1 may constitute the computing device (100), and additional component(s) other than the components illustrated in FIG. 1 may be included in the computing device (100).

As illustrated in FIG. 1, the computing device (100) may include a processor (110), a memory (120), a communication unit (130), an input unit (140), and a display (150).

According to an embodiment, the computing device (100) may include an input unit (140) equipped with a camera for obtaining an input image and may also include a display (150) for outputting a fused image.

The communication unit (130) can transmit and receive data with external devices such as other electronic devices or servers using wired or wireless communication technology. For example, the communication unit (130) can transmit and receive sensor information, user input, learning models, control signals, etc. with external devices.

The memory (120) can store data that supports various functions of the computing device (100).

The processor (110) can determine at least one executable operation of the computing device (100). Then, the processor (110) can control components of the computing device (100) to perform the determined operation.

To this end, the processor (110) may request, retrieve, receive or utilize data from the memory (120) and control components of the computing device (100) to execute at least one of the executable operations, either a predicted operation or an operation determined to be desirable.

At this time, if linkage of an external device is required to perform a determined operation, the processor (110) can generate a control signal for controlling the external device and transmit the generated control signal to the external device.

The processor (110) can control at least some or a combination of components of the computing device (100) to run an application program stored in the memory (120).

A computing device (100) according to one embodiment of the present invention can transmit and receive data through wireless and/or wired interconnection. The computing device of the present disclosure can include any type of computing device capable of computing data in electronic form.

For example, it can be implemented as a fixed device or a movable device, such as a TV, a projector, a mobile phone, a smart phone, a desktop computer, a laptop, a digital broadcasting terminal, a PDA (personal digital assistant), a PMP (portable multimedia player), a navigation device, a tablet PC, a wearable device, a set-top box (STB), a DMB receiver, a radio, a washing machine, a refrigerator, a desktop computer, digital signage, a robot, a vehicle, etc.

It should be understood that the following embodiments of the present invention are performed by a computing device (100) that provides a face style fusion generation function that fuses a face in an input image with a person in a target image, and may be named a computing device for an exhibition for viewers to experience, or a face area style fusion generation device.

The following are definitions of terms used in the present invention.

The input image may refer to an image of a person to be fused, and may refer to a real-life image. For example, in an embodiment for a visitor experience exhibition, it may include an image of a visitor. The face area of the input image may refer to the face area of the visitor in the visitor image.

The fusion target image refers to a target image for fusion of the face of a person in an input image, and in this embodiment, it can refer to a target conversation image including a person, which is mainly image data of a conversation work exhibited at an exhibition, etc.

The fusion feature is a feature tensor generated by fusing the facial attribute features of the fusion target face and the facial identity features of the input image, and can be converted into a fused facial image through a face generation model.

A fused image may refer to an image resulting from style fusion of the input image and the fused target image.

The preprocessing process of the fusion target image may refer to a data preprocessing process for creating a style fusion image of the input image and the fusion target image.

Below, according to an embodiment of the present invention, a preprocessing process of a fusion target image and a fusion image generation process are described.

FIG. 2 illustrates a preprocessing process of a fusion target image for generating a fusion image according to an embodiment of the present invention.

According to an embodiment of the present invention, a processor of a computing device can extract facial key features for a fusion target image (S210), perform face alignment based on the extracted facial key features (S220), and extract facial attribute features based on the aligned face (S230).

Specifically, facial key feature point extraction (S210) for the fusion target image can automatically extract facial key feature points using a facial landmark detector.

Meanwhile, as a facial key feature extractor used in an embodiment of the present invention, conventional technologies such as One Millisecond Face Alignment with an Ensemble of Regression Trees (Vahid Kazemi and Josephine Sullivan, CVPR 2014) and SAN (Style Aggregated Network for Facial Landmark Detection, CVPR 2018) can be used.

FIG. 4 illustrates an example of extracting key facial features of a fusion target image according to an embodiment of the present invention.

For example, Fig. 4 (a) shows a fusion target image, and Fig. 4 (b) shows an example of the result of extracting facial key features. Referring to Fig. 4 (b), it can be seen that facial key features are output as multiple points.

Each of the above multiple points can be mapped to predefined face normal coordinates during later face alignment.

Let's explain Figure 3 again.

According to an embodiment of the present invention, if the result of facial key feature extraction for a fusion target image is inaccurate, the result of feature point extraction can be corrected based on user input. For example, if the evaluation index of facial key feature extraction for a fusion target image is smaller than a predetermined value, the engineer can correct the facial key feature through user input for the user fusion target image.

According to an embodiment of the present invention, a processor may perform face alignment based on predefined facial normal coordinates based on facial key feature points of an extracted fusion target image (S220). The predefined facial normal coordinates may refer to facial key feature point coordinates when a person looks straight ahead in a proper posture, and may be acquired from a large amount of facial data. For example, among a large amount of facial data, the average feature point coordinates of faces in proper posture may be acquired as facial normal coordinates.

Figure 5 illustrates an example of face alignment according to an embodiment of the present invention.

Referring to Fig. 5, a face region (510) of a fusion target image is shown, and the face region may include major facial features of the fusion target image. The face region (510) of the fusion target image may correspond to a pose tilted to the left by a certain angle from the observer's viewpoint.

A processor according to an embodiment of the present invention can perform image transformation based on predefined facial normal coordinates (520) based on facial key feature points of a fusion target image extracted from S210.

Specifically, coordinate mapping can be performed by extracting facial normal coordinates (520) corresponding to the facial key feature points of the fusion target image and transforming the facial key feature points into the facial normal coordinates. As a result of the image transformation, a facial region (530) of the aligned fusion target image can be obtained.

Again, Figure 2 is described. A processor according to an embodiment of the present invention can extract facial attribute features of a fusion target image (S230).

At this time, the facial attribute may mean the pose, expression, lighting, background, and other objects (various objects such as hair, hands, accessories, hats, and musical instruments other than the face existing within the facial area), and the facial attribute feature may mean a feature in which the facial attribute of the fusion target image is embedded.

In order to extract the facial attribute features from the facial region of the fusion target image, in one embodiment of the present invention, various models used as decoders in artificial neural network-based generative models such as U-Net (Ronneberger, O., Fischer, P. and Brox, T., U-Net: Convolutional Networks for Biomedical Image Segmentation. International Conference on Medical image computing and computer-assisted intervention, 2015) can be used.

Additionally, the above facial attribute features may be composed of a combination of features extracted at various stages of the artificial neural network.

Meanwhile, since the preprocessing process (S200) of the fusion target image can be executed in advance without having to be executed in real time during the style fusion generation process, it is possible to manually perform the extraction of major facial feature points. In addition, the preprocessing process can be executed simultaneously with the fusion image generation process of Fig. 3, which will be described later, but can be performed in advance before the image fusion generation process of Fig. 2 in order to reduce the computation time and prevent duplicate calculations.

FIG. 3 is a block diagram illustrating a process for generating a facial style fusion image for a viewer experience exhibition according to an embodiment of the present invention.

Referring to FIG. 3, a processor according to an embodiment of the present invention can perform style fusion by performing, when an input image is input, facial key feature point detection (S310), face alignment (S320), facial identity feature extraction (S330), feature fusion (S340), face creation based on fused features (S350), facial sharpness improvement (S360), and fused face-painting synthesis (S370) of an input image.

Specifically, according to one embodiment of the present invention, the processor may obtain an image of a visitor as an input image by using a device equipped with a camera, such as a photo kiosk connected to a computing device, and may detect major facial landmarks from the input image by using a facial landmark detector (S310).

The facial key feature point detection (S310) of the present invention can be performed through the same process as the S210 embodiment of FIG. 2, and the number of detected facial key feature points can be the same as the number of facial key feature points extracted from the fusion target image through the S210 process.

Face alignment (S320) according to an embodiment of the present invention may mean a process of aligning the face of a viewer corresponding to an input image based on predefined facial normal coordinates based on facial key feature points of an extracted input image, similar to the embodiment of process S220 of FIG. 2.

At this time, the face alignment result of the input image can be aligned to the same size and position as the face of the fusion target image.

Specifically, the input image can correspond feature points extracted based on a facial key feature point detection process to predefined facial normal coordinates, and the processor can transform each of a plurality of facial key feature points into each corresponding facial normal coordinate.

That is, the number of points appearing in the facial normal coordinates and the number of feature points extracted from S210 and S310 may be the same.

A processor according to an embodiment of the present invention can extract facial identity features from a face of an aligned input image (S330).

At this time, identity can mean the distinguishing characteristics of a person, such as facial features, using the audience's facial image data.

In order to extract such identities as features, various face recognition models such as ArcFace (Deng, J., Guo, J., Xue, N. and Zafeiriou, S., Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019) and CosFace (Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., Li, Z. and Liu, W., Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018) can be used in one embodiment of the present invention.

According to another embodiment of the present invention, in order to better reflect the style of the fusion target image in the fusion image generated as a result of style fusion, facial identification features of the fusion target image are additionally extracted from the face alignment image of the fusion target image, and the statistical characteristics (e.g., mean, standard deviation, etc.) of the facial identification features of the input image (viewer image) can be adjusted for each channel to match the statistical characteristics of the facial identification features of the fusion target image (S331).

Through this, the artistic style of the fusion target face can be reflected in the features of the input face.

Meanwhile, a method such as AdaIN (Huang, X. and Belongie, S., Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, 2017) can be used as an algorithm to reflect the statistical characteristics of the above facial identity features.

A processor according to an embodiment of the present invention can obtain facial attribute features of a fusion target image and facial identity features of an input image, and fuse the features to generate fusion features (S340).

At this time, the fusion feature can be obtained by matching the spatial resolution of the two features through image interpolation, and then adding or multiplying pixels existing in the spatial resolution of the facial attribute feature of the fusion target image and the facial identity feature of the input image for each pixel (element).

Figure 6 illustrates a method for obtaining fusion features according to an embodiment of the present invention.

Referring to FIG. 6, there are multiple matrices in which the spatial resolution of the facial attribute features (620) of the fusion target image and the facial identity features (610) of the input image match, and a fusion feature (630) can be generated by adding or multiplying pixels existing in the spatial resolution by each pixel (element).

According to another embodiment of the present invention, an algorithm such as SPADE (Park, T., Liu, M.Y., Wang, T.C. and Zhu, J.Y., Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019) can be used to fuse the attribute features and identity features.

A processor according to an embodiment of the present invention can generate a fused facial image from fused features using a facial generation model (S350).

The above fused facial image may mean an image in which the facial region of the input image is synthesized based on the facial region of the fused target image.

Specifically, in the fusion feature-based face generation (S350) step, a face generation model can be trained using a Generative Adversarial Network (GAN) structure.

Figure 7 illustrates a process of learning a face generation model (700) according to an embodiment of the present invention.

For example, as illustrated in FIG. 7, a face generation model can be trained by training a generator (720) to generate a fused face image (730) from fused features (710), and providing feedback to a predicted label (760) to distinguish between an actual image (750, a fused target image (exhibition image) in the present invention) and the generated image (730) using a discriminator (740).

According to an embodiment of the present invention, after training, the face generation model may generate a fused face image from fused features using a generator (Generator, 720) in the actual face generation process.

A processor according to an embodiment of the present invention can improve the sharpness of a generated fused facial image (S360).

FIG. 8 is an example showing a process for improving the sharpness of a fused facial image according to an embodiment of the present invention.

Referring to Figure 8, compared to the fused facial image before sharpness improvement, the fused facial image with improved sharpness can be seen to have clearer edge and boundary features.

In one embodiment of the present invention, a Blind Face Restoration technology such as GPEN (Yang, T., Ren, P., Xie, X. and Zhang, L., Gan prior embedded network for blind face restoration in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021) and GFP-GAN (Wang, X., Li, Y., Zhang, H. and Shan, Y., Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021) may be used.

A processor according to an embodiment of the present invention can perform fusion face-conversation synthesis based on a fusion face image with improved sharpness (S370).

Specifically, the processor can obtain a fusion target image and facial key features of the fusion target image, transform a fusion facial image with improved sharpness based on the S360 step into a facial coordinate system of the fusion target image, and synthesize the fusion facial image with the fusion target image in the transformed coordinate system to generate a style fusion image.

Figure 9 is an example showing a style fusion image generation process according to an embodiment of the present invention.

Referring to FIG. 9, the processor can obtain a final style fusion image (910) in which a face of an input image is reflected and a face region of a fusion target image. In addition, the processor can obtain a face region (930) of the input image.

According to an embodiment of the present invention, a fused facial image with improved sharpness based on steps S310 to S360 described above is mapped to predetermined facial normal coordinates based on face alignment (S220, S320).

Accordingly, by converting this back into the facial coordinate system of the fusion target image, a fused facial image (940) converted into the pose of the person in the fusion target image can be generated. At this time, the inverse matrix of the image transformation used for face alignment is multiplied by the fused facial image, so that the fused facial image can be mapped into the facial coordinate system of the fusion target image.

Thereafter, a processor according to an embodiment of the present invention can generate a final style fusion image (910) by synthesizing a fusion facial image (940) converted into a pose of a person in the fusion target image with the fusion target image.

Figure 10 illustrates an artifact mitigation process according to an embodiment of the present invention.

Referring to FIG. 10, the processor may use a Gaussian mask to synthesize a fused face image and a face in a fused target image during an image synthesis process, thereby alleviating artifacts present at the boundary.

Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (for convenience, referred to herein as software), or a combination of both.

The above-described present invention can be implemented as a computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all kinds of recording devices that store data that can be read by a computer system. Examples of the computer-readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

Claims (20)

In a method for creating a fusion video for an exhibition for visitors to experience,
(a) A step of extracting facial identity features based on key facial feature points of an input image;
(b) A step of acquiring facial attribute features of a fusion target image;
(c) a step of obtaining fusion features by fusion of facial attribute features of the fusion target face and facial identity features of the input image;
(d) a step of generating a fused facial image based on the above fused features; and
(e) a step of generating a style fusion image by converting the fused facial image into the facial coordinate system of the fusion target image;
How to create a style fusion video. In paragraph 1,
Step (b) above,
A step of extracting key facial features of the above fusion target image;
A step of performing face alignment based on the extracted facial key features, and
Comprising a step of obtaining facial attribute features based on the aligned face.
How to create a style fusion video. In the second paragraph,
The steps for performing the above face alignment are:
A step of performing face alignment based on predefined facial normal coordinates based on the facial key feature points of the fusion target image,
How to create a style fusion video. In paragraph 1,
Step (a) above,
Further comprising a step of performing face alignment based on predefined facial normal coordinates of facial key feature points of the input image.
How to create a style fusion video. In paragraph 1,
The above facial identification features refer to the facial features of the input image.
How to create a style fusion video. In paragraph 5,
The above facial identity features are adjusted based on the channel-by-channel statistical characteristics of the facial identity features of the fusion target image.
How to create a style fusion video. In paragraph 1,
The above facial attribute features include at least one of facial pose, expression, lighting, background, and other object information of the fusion target face.
How to create a style fusion video. In paragraph 1,
Step (c) above,
A step of matching the spatial resolution of the facial attribute features and the facial identity features using image interpolation; and
A step of obtaining a fusion feature by assigning to each pixel the sum or product of each pixel existing in the spatial resolution of the facial attribute feature of the fusion target image and the facial identity feature of the input image,
How to create a style fusion video. In paragraph 1,
Step (d) above,
When the above fusion feature is input into the face generation model, a step of generating a fusion face image is included.
The above face generation model includes a model trained to generate the fused face image from the fused features using a generator during training, and trained to distinguish the face of the fused target image from the generated fused face image using a discriminator.
How to create a style fusion video. In the first paragraph, step (e) is,
A step of further synthesizing the fused face image and the face in the fused target image using a Gaussian mask to alleviate artifacts existing at the boundary,
How to create a style fusion video. In a fusion video generation device for an exhibition for visitors to experience,
Containing at least one processor,
The above processor (a) extracts facial identity features based on facial key features of an input image, (b) obtains facial attribute features of a fusion target image, (c) obtains fusion features by fusion of the facial attribute features of the fusion target face and the facial identity features of the input image, (d) generates a fusion face image based on the fusion features, and (e) generates a style fusion image by converting the fusion face image into a facial coordinate system of the fusion target image.
Style fusion video generator. In Article 11,
When the above processor performs the above operation (b),
Extracting the facial key features of the above fusion target image, performing face alignment based on the extracted facial key features, and obtaining facial attribute features based on the aligned face.
Style fusion video generator. In Article 12,
When the above processor performs the face alignment, the face alignment is performed based on predefined face normal coordinates based on the facial key feature points of the fusion target image.
Style fusion video generator. In Article 11,
When the above processor performs the above operation (a), face alignment is performed based on the facial key feature points of the input image to predefined facial normal coordinates.
Style fusion video generator. In Article 11,
The above facial identification features refer to the facial features of the input image.
Style fusion video generator. In Article 15,
The above facial identity features are adjusted based on the channel-by-channel statistical characteristics of the facial identity features of the fusion target image.
Style fusion video generator. In Article 11,
The above facial attribute features include at least one of facial pose, expression, lighting, background, and other object information of the fusion target face.
Style fusion video generator. In Article 11,
The above processor, when performing the operation (c), matches the spatial resolution of the facial attribute feature and the facial identity feature by using image interpolation, and obtains the fusion feature by assigning to each pixel the sum or product of each pixel existing in the spatial resolution of the facial attribute feature of the fusion target image and the facial identity feature of the input image.
Style fusion video generator. In Article 11,
When the above processor performs the above operation (d),
When the above fusion feature is input to a face generation model, a fusion face image is generated, and the face generation model is trained to generate the fusion face image from the fusion feature using a generator during training, and includes a model trained to distinguish the face of the fusion target image from the generated fusion face image using a discriminator.
Style fusion video generator. In Article 11,
When the above processor performs the above operation (e),
By synthesizing the fused face image and the face in the fused target image using a Gaussian mask, artifacts existing at the boundary are alleviated.
Style fusion video generator.

Images