KR101444816B1 - Image Processing Apparatus and Method for changing facial impression - Google Patents

Abstract

An image processing method and apparatus for facial raising conversion according to an embodiment of the present invention aims at changing only an impression of a face while maintaining the identity of a face. An image processing method and apparatus according to an exemplary embodiment of the present invention includes a plurality of groups of feature points in a face image, each group including a plurality of face images and using a database including a plurality of groups classified on the basis of a face type, And a distance vector, generating an impression score estimation function, and modifying the plurality of feature points and the distance vector to convert the face image.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an image processing method and a facial image processing method,

Embodiments of the present invention relate to an image processing method and an image processing apparatus for face-up conversion for automatically converting an impression of a face while maintaining the identity of a face represented by the image.

In general, facial images can best represent individual characteristics and have been used as mediators to make communication easy and natural. Applications of these facial images include access control / security systems, criminal search / montage creation systems, computer interfaces, photoshop, animation and games.

Especially, montage writing system plays an important role in the public security and social order maintenance by helping arrest criminals in a short period of time. However, since witnesses often witness a very short period of time due to the urgency of the situation, it is difficult to create an accurate montage if the witness makes abstract testimony. For example, a witness may testify about the impression of a face such as "drowsy", "during", or "beautiful".

When changing the impression of the montage image based on such testimony, the montage image may be changed by the subjective idea of the creator of the montage. In addition, since the conventional face conversion method has been made by the user selecting and combining the eyes, nose, mouth, or face type from a predetermined database, the face becomes unnatural or many parts such as face type are changed, It may not be possible to keep it. There is also a conventional method of predicting changes in facial expressions and the like by linear combination of models by preliminarily defining a face change base model, but this method is suitable when it is desired to reproduce a specific facial expression change, Not suitable for change.

Therefore, it has been pointed out that changing the impression of the face in the conventional way is time consuming, difficult, and likely to escape the identity of the original face.

Accordingly, in order to solve the above problems, an embodiment of the present invention provides a system and method for automatically changing a face increase while maintaining the identity of the original face when changing the face impression displayed on the face image .

According to an aspect of the present invention, there is provided an image processing method for converting a face of a face into a plurality of groups including a plurality of face images, The image processing method for use in the face enhancement transformation is largely divided into a process of obtaining an impression score from the evaluators at the database building step and an impression change step of the face image inputted by the user. The landmark points for the image, the distance vector between the points, and the impression score evaluated by the evaluator should be stored initially. Extracting image information by obtaining a plurality of feature points in a face image input from a user and determining a group matching a face type of the inputted face image in the database in the impression change step; Generating an impression score estimation function based on the type of the impression and information of the determined group if the type of impression to be changed is selected for the face image; And modifying the position of the plurality of feature points through the impression change score and the impression score estimation function to convert the face image to a desired impression while maintaining the identity of the input face image; .

Preferably, the identity of the face image includes a face type, a face size, a face length, and a facial expression.

The step of extracting the image information may further include obtaining the plurality of feature points and then obtaining a distance vector that connects any two neighboring feature points among the plurality of feature points.

The step of extracting the image information may include calculating a distance vector of the input face image and then calculating a distance vector between the center vector that is an average value of the distance vectors of the face images in each group in the database, Calculating an Euclidean distance between the reference points; And determining the group having the smallest center value as the matching group.

In addition, the database may include distance vector information for each stored face image and information about an impression score set for each impression type.

In addition, the impression score input by the user and the impression score set for the face image of the database are 0 to 1.

The step of converting the face image may further include: obtaining a correction vector obtained by modifying the distance vector through the impression score estimation function and the impression change score; Modifying a position of the plurality of feature points through the correction vector; And transforming the face image based on the modified plurality of feature points and the correction vector. And a control unit.

The step of acquiring the correction vector may further include: inputting a distance vector of the input face image to the impression score estimation function to obtain an impression score of the input face image; Designating as a correction vector a vector having an impression score equal to the obtained impression score plus the impression change score; And a control unit.

The modifying the positions of the plurality of feature points may modify the plurality of feature points to a position where an error value between the distance vector of the input face image and the correction vector is minimized.

A database in which a plurality of groups including a plurality of face images are classified based on a face type; A user input unit for inputting a face image from a user and a type of an impression for changing the face image and an impression change score; An image information extraction unit for obtaining a plurality of feature points in the input face image and determining a group matching the face type of the input face image in the database; A function generation unit for generating an impression score estimation function based on the type of impression of the face image and the determined group information; And an image conversion unit for modifying the position of the plurality of feature points through the impression change score and the impression score estimation function to convert the face image into an impression desired by the user while maintaining the identity of the input face image, ; And

In addition, the identity of the face image includes a face type, a face size, a face length, and a facial expression.

The image information extracting unit may include a feature point extracting unit that obtains the plurality of feature points and obtains a distance vector connecting two adjacent feature points of the plurality of feature points.

The Euclidean distance between the center vector, which is an average value of the distance vectors of the face images in each group in the database, and the distance vector of the input face image, And a matching group determining unit for determining the group having the smallest center value as the matching group by calculating the Euclidean distance.

In addition, the database may include distance vector information for each stored face image and information about an impression score set for each impression type.

In addition, the impression score input by the user and the impression score set for the face image of the database are 0 to 1.

The image transformation unit may include an image information modification unit for inputting the distance vector to the impression score estimation function to obtain a correction vector as an output and correcting the positions of the plurality of feature points through the correction vector. do.

The image modifying unit may further include a distance vector calculating unit that calculates a distance between a value obtained by adding the impression score to the obtained impression score and a value obtained by adding the impression score to the inputted impression score, And a correction vector obtaining unit for obtaining a vector having the same impression score and designating it as a correction vector.

The image information modifying unit may include a minutia point position modifying unit for modifying the plurality of minutiae to a position where an error value between the distance vector of the input facial image and the correction vector is minimized.

The image processing apparatus according to an embodiment of the present invention can automatically change the impression of the desired image, while the conventional technique manually changes the image by the user's operation. In contrast, in the prior art, image transformation manually operated by a user is inaccurate and error-prone, whereas an embodiment of the present invention transforms an image based on a database scored for impression of a plurality of images, The impression can be converted while maintaining the identity of the original face image. In addition, one embodiment of the present invention can change for various kinds of impression, and the impression change degree can be scored with respect to one kind of impression, thereby improving the convenience and efficiency of the user with impression change .

1 is a structural view of an image processing apparatus for face-up conversion according to an embodiment of the present invention.
FIG. 2 is a schematic diagram illustrating a relationship between impression types and impressions that can be input to an image processing apparatus for face-up conversion according to an exemplary embodiment of the present invention.
3 is a conceptual diagram for conceptually explaining the functions of the minutiae and distance vector extraction unit according to an embodiment of the present invention.
FIG. 4 is a graph showing an example of an impression score estimation function according to an embodiment of the present invention.
5 is a flowchart illustrating an image processing method for face-up conversion according to an exemplary embodiment of the present invention.
6 is a detailed flowchart of the image information extracting step of FIG.
7 is a detailed flowchart of the image information correction step of FIG.
FIG. 8A is a general face image before the face-up transformation according to an embodiment of the present invention is performed.
FIG. 8B is a face image after face-up transformation according to an embodiment of the present invention.

Hereinafter, an image processing method and an image processing apparatus for face up conversion according to an embodiment of the present invention will be described in detail with reference to the drawings.

In the present specification, the same or similar reference numerals are given to different embodiments in the same or similar configurations. In addition, it should be considered that the components of the drawings attached hereto can be enlarged or reduced for convenience of explanation.

As used herein, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Furthermore, terms including ordinals such as first, second, etc. used in this specification may be used to describe various components, but since the terms are used only for the purpose of distinguishing one component from another, The elements are not limited by these terms.

The image processing apparatus 100 according to an embodiment of the present invention includes a user input unit 110 and an image processing unit 120.

The user input unit 110 includes an image input unit 111 and an image selection unit 112 and may be configured as a keypad, a keyboard, a mouse, a touch screen, etc., .

The image input unit 111 is a configuration in which a user inputs a face image to change an impression. The impression selection unit 112 is configured to input the type of impression to be changed, the impression change score, and the like in accordance with a user's input.

Here, the relationship between the type of impression that the user can input and each impression can be schematized as shown in FIG. Fig. 2 is a diagram showing an example in which when a value for "at least (x)", "mature (y)", "sharp (w)" and "smooth (z) Means that the values for the eight impressions, such as "cuteness, time (a)", "goodness (b)", "warmth (c)" ... " Specifically, the range of scores for each impression is normalized from 0 to 1, where x and y are complementary relationships, and w and z are also complementary relationships. For example, when x = 0.2, y = 1 - 0.2 = 0.8. Similarly, when w = 0.4, z = 0.6. In this case, a = x, b = x * 1/2 + z * 1/2, c = z, d = y * 1/2 + z * 1/2, e = y, f = a = 0.2, b = 0.4, c = 0.6, d = 0.7, e = 0.8, and f = = 0.6, g = 0.4, and h = 0.3. (0? A? H? 1) In other words, in one embodiment of the present invention, the impression change score and the type of impression can be determined by inputting scores for two impressions (for example, Can be automatically calculated as a score for the impression. The correlation between each impression as shown in FIG. 2 is defined by analyzing the correlation between the vocabulary and the structure of the face regarding the impression. On the other hand, the correlation between the impressions is not limited to the example of FIG. 2 and can be defined in various ways.

The image processing unit 120 processes the face image input from the user input unit 110. Specifically, the image processing unit 120 changes the impression of the input face image to the degree of impression selected by the user, and maintains the identity of the input face image . Here, identity is a concept including face type, face size, face length, facial expression, and the like. The image processing unit 120 includes an image information extraction unit 121, a database 122, a function generation unit 123, an image information correction unit 124, an image conversion unit 125, and a display unit 126.

The information of the face image inputted by the user is extracted by the image information extracting unit 121. The information of the face image includes information about the feature point and the distance vector of the face image and the group to which the face image inputted in the database 122 belongs . Accordingly, the image information extraction unit 121 includes a feature point and distance vector extraction unit 121a and a matching group determination unit 121b.

The feature point and distance vector extraction unit 121a extracts feature points and distance vectors of a face image input from a user using a feature extraction algorithm (for example, an Active Appearance Model (AAM), a Constrained Local Model . A feature point is a characteristic part of a face, such as a facial feature, which is also called a landmark point. Since the feature points are extracted based on the outline of the features, the boundaries of the colors, the bending points, and the difference in contrast, the feature points are generally composed of a plurality of features. In one embodiment of the present invention, it is preferable that 75 to 85 feature points are extracted. The distance vector is a vector that connects two adjacent feature points and provides intuitive information about the face structure. In Figure 3, the distance vectors for a particular face image are shown. As shown in FIG. 3, when the distance vector is represented on the face image, it is understood that the image that represents the 2D mesh is represented. Further, since the feature points are constituted by a plurality of feature points, the distance vectors are also composed of a plurality of feature vectors. Therefore, the distance vector is defined by a plurality of dimensions (Dimension). For example, when n vectors from the V1 vector to the Vn vector are defined, the distance vector becomes an n-dimensional vector.

The matching group determination unit 121b determines a group matching with the face image input in the group for the face image stored in the database 122. [ A plurality of face images are stored in the database 122, and all of the images are classified based on face type. The matching group determination unit 121b searches for and determines a group of images of the face type matching with the face type of the face image input from the user.

The database 122 stores a plurality of face images as described above. The stored face image may be an expressionless frontal face image for a different person's face. In addition, it is preferable that at least 100 stored face images are stored. On the other hand, each of the stored face images is given an impression score in advance. For example, a score may be assigned to eight impressions included in the box of FIG. 2, and a score of 0 or more and 1 or less may be given. The database builder will be able to enter impressions for eight impressions for each face image by entering only the score for the two impressions. For example, if one face image is input with "at: 0.6 points" and "soft: 0.4 points", the score will be given as "good: 0.5 points" ... "warm: 0.4 points" .

In addition, the database 122 includes information on feature points and distance vectors of each stored face image, normalizes each face image based on the face size, classifies the face images according to face types, . For example, the plurality of groups 122g may be classified into the form of a slender shape, an egg shape, a round shape, an oval shape, an inverted oval shape, a square shape, and the like. This classification may be made by firstly classifying a plurality of pictures directly by the builder of the database 122, and may be automatically classified through the high-speed information processing apparatus using image information such as minutiae.

The function generation unit 123 generates an impression score estimation function using the information of the group matching the input face image based on the input face image information, the type of the impression inputted by the user, and the impression score stored in the database . The impression score estimation function is set based on the images in the group matching with the inputted face image, and can be defined as Equation (1), indicating the degree of impression to be converted while maintaining the identity of the face image.

In Equation 1, w is a weight value, b is a constant value, the input value x is a distance vector of the input face image, and the output value f (x) is an impression score in the matching group of the input face image . In this case, w and b are conversion factors such as the slope value of the impression score estimation function and the y-intercept value. The impression score estimation function is defined for a plurality of dimensions. Thus, although the impression score estimation function is schematically shown as Equation 1, the impression score estimation function has a one-dimensional output for the n-dimensional input. For example, if the n-dimensional distance vectors X1, X2, ..., Xn are input to the impression score estimation function, the resulting output is Y. [

On the other hand, the impression score estimation function is generated based on the information of the group matching the input face image, the type of the impression inputted by the user, and the impression score of the inputted face image stored in the database. For example, if the input face image matches an elliptical face group and the user gives a score of "soft: 0.2 points", "mature: 0.4 points" and scores 0.3 points If given, the impression score estimation function is created using the distance vector of the images in the elliptical face group and the impression score related to the elegance stored in the database. Specifically, as shown in Fig. 4, when the coordinates of the impression score relating to elegance are defined as a Y-axis value (one-dimensional value) and the distance vector is defined as an X-axis value (n-dimensional value) When a graph is sorted from low to high, a graph having curvature as shown in FIG. 4 can be generated. At this time, when inputting the distance vector Xa of the input face image, an impression score of Ya is output as an output. If Ya is 0.5 (medium value), it can be seen that the input face image has an intermediate level with respect to elegance within the elliptical face group. That is, by estimating the impression score of the inputted face image through the impression score estimation function, it is possible to estimate the degree of the impression that the input face image occupies in the matching group.

The image information correction unit 124 corrects the distance vector and the feature point among the information of the input face image and includes a correction vector acquisition unit 124a and a feature point position correction unit 124b.

The correction vector acquisition unit 124a acquires the correction vector through the impression score estimation function and the impression change score selected by the user. At this time, the correction vector is modified based on the distance vector of the input face image. Referring to FIG. 3, when Xa and Ya are the input values and the output values of the distance vector, the input values and the output values of the correction vector are set to Xb and Yb. At this time, the difference (d) between Yb and Ya corresponds to the impression change point value inputted by the user. For example, when the user designates an impression change score of 0.3 points for elegance, "Yb - Ya = 0.3 ", and Xb is set as a correction vector according to such Yb value. Therefore, the correction vector is near the distance vector and at the same time is higher than the distance vector for the impression score.

The minutia point position correction section 124b corrects the first minutiae point in accordance with the correction vector. The minutia point position correction section 124b corrects the minutiae point so that the difference between the scalar value of the distance vector and the scalar value of the correction vector becomes minimum. The modified minutiae become the starting point of the correction vector and are used for image transformation.

The image converting unit 125 converts the inputted face image based on the correction vector and the modified minutiae. The image converting unit 125 may convert the face image through a warping process. On the other hand, the image conversion unit 125 may be configured to include the image information correction unit 124.

The display unit 126 visually displays the converted face image and provides it to the user. The display unit 126 may be a display device such as a liquid crystal display (LCD).

Next, an image processing method for face-up conversion according to an embodiment of the present invention will be described with reference to the drawings through the image processing apparatus 100 described above.

Referring to FIG. 5, first, a specific face image is input by the user to the image processing apparatus (see FIG. 1, 100) of the present invention (S100).

Then, information of the inputted face image is extracted (S110). The step S110 is described in detail in Fig.

In the input face image, feature points are extracted by the feature point extraction unit (S111). Then, the distance vector of the inputted face image is extracted by the distance vector extracting unit based on the minutiae points (S112). The matching group determination unit 121b designates a center vector of each group in the database 122 (see FIG. 1) to search for a group matching the face type of the input face image (S113). Specifically, each group contains information about the distance vectors of the stored face images. Therefore, each group includes information on a distance vector, and a vector having an average value among them can be designated as a center vector. In step S113, the matching group determination unit 121b may process the result. However, the result of step S113 may be stored in advance in the database 122, and the information about the center vector may be a previously stored value. Next, the matching group determination unit 121b calculates the Euclidean distance between the center vector and the distance vector of the input face image. The Euclidean distance is defined by Equation (2).

Here, p and q denote the distance vector between the center vector and the input face image, and n denotes the dimension of each vector. The matching group determination unit 121b determines a matching group having a center vector having the smallest value calculated through the Euclidean distance as a matching group (S115).

Referring again to FIG. 4, the user inputs the type of impression to be changed and the impression change score for the inputted face image (S120). For example, if a user gives a score of "soft: 0.2 points", "mature: 0.4 points", "elegant: 0.3 points", "mature: 0.4 points" : 0.6 points "and so on, the impression change score is inputted for the eight impressions. The function generating unit 123 generates an impression score estimating function based on the impression type input by the user, the impression change score, and the information of the matching group (S130). For example, the function generation unit 123 obtains an impression score estimation function as shown in FIG. 4 by graphing facial images in the matching group on the basis of elegance.

The image information correction unit 124 corrects the distance vector and the feature point of the input image (S140).

Referring to FIG. 7, a correction vector acquisition unit (see FIG. 1) 124a in the image information correction unit 124 (see FIG. 1) acquires a correction vector through an impression score estimation function (S141). As described above, the impression score of the input face image can be estimated through the impression score estimation function, and the value obtained by adding the impression change score input by the user to the estimated impression score is the impression score of the correction vector. The correction vector acquisition unit 124a acquires the correction vector in this manner.

In addition, the minutia position correction unit 124b corrects the position of the minutiae of the distance vector of the input face image (S142). That is, the minutia position correction unit 124b optimizes the inputted face image so as to reduce the error between the distance vector and the correction vector. The calculation for the error is defined as Equation (3).

는 웨이트(weight)값이며,

와

는 입력된 얼굴 이미지의 특징점을 의미하는 것이므로

는 거리벡터의 스칼라값을 의미하며,

는 수정벡터의 스칼라값을 의미한다. 특징점 위치 수정부(124b)는 수학식 3에 의한 결과값이 가장 작도록 특징점의 위치를 수정한다.here,

Is a weight value,

Wow

Is a feature point of the input face image

Denotes a scalar value of the distance vector,

Means the scalar value of the correction vector. The minutia point position correction section 124b corrects the position of the minutiae point so that the result of Equation (3) is the smallest.

Finally, the image converting unit 125 converts the inputted face image based on the modified vector and the modified feature points (S150). The image converting unit 125 may convert an image through a warping process.

In one embodiment of the present invention, the correction vector, which is the basis of the image transformation, is obtained through the impression score estimation function, wherein the impression score estimation function is a function of calculating the information of the image group having the same face type as the inputted face image Score). Therefore, transforming the image through the impression score estimation function means that the image is transformed while maintaining the same identity as the face type of the first face image. 8A and 8B, if the original image shown in FIG. 8A is transformed so that the impression regarding " fear " is weighted through the impression transformation method according to an embodiment of the present invention, The identity of the user is maintained so that a frightening impression is added.

As described above, one embodiment of the present invention can solve the problems that may occur when manually changing the impression of the image by the user's manipulation as in the related art. In addition, one embodiment of the present invention can change for various kinds of impression, and the impression change degree can be scored with respect to one kind of impression, thereby improving the convenience and efficiency of the user with impression change .

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.

Therefore, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concept of the present invention defined in the following claims are also within the scope of the present invention.

110: user input unit 120:
121: Image information extracting unit 122:
123: function generation unit 124: image information management unit
125: image conversion unit 126: display unit

Claims (18)

An image processing method for face-up conversion using a database in which each group including a plurality of face images is classified based on face type,
Extracting image information by obtaining a distance vector connecting a plurality of feature points and the plurality of feature points in a face image input from a user and determining a group matching a face type of the input face image in the database step;
Generating impression score estimation functions based on the type of impression and information of the determined group, the type of impression and the impression change score to be changed with respect to the face image; And
And modifying the position of the plurality of feature points through the impression change score and the impression score estimation function to convert the face image to a user's desired impression while maintaining the identity of the inputted face image However,
The step of converting the face image may further include obtaining a correction vector obtained by modifying the distance vector through the impression score estimation function and the impression change score and outputting the correction vector to the position where the error value between the distance vector and the correction vector becomes minimum. And modifying the position of the minutiae point. The method according to claim 1,
Wherein the identity of the face image includes a face type, a face size, a face length, and a facial expression. delete The method according to claim 1,
Wherein the extracting of the image information comprises:
After calculating the distance vector of the input face image,
Calculating an Euclidean distance between a center vector which is an average value of distance vectors for a face image in each group and a distance vector of the input face image in the database; And
Determining a group having the smallest center value as the matching group;
And converting the face image into a face image. The method according to claim 1,
Wherein the database includes distance vector information for each stored face image and information about an impression score of each stored face image. 6. The method of claim 5,
Wherein the impression change score inputted by the user and the impression score of each face image stored in the database are 0 or more and 1 or less. The method according to claim 1,
Wherein the converting the face image comprises:
And transforming the face image based on the modified plurality of feature points and the correction vector. 8. The method of claim 7,
The correction vector may be expressed as:
A distance vector of the input face image is input to the impression score estimation function to obtain an impression score of the input face image and a vector having an impression score equal to the impression score obtained by adding the impression score And a correction vector is obtained. delete A database in which each group including a plurality of face images is classified based on face type;
A user input unit for inputting a face image from a user and a type of an impression for changing the face image and an impression change score;
An image information extraction unit for obtaining a distance vector connecting a plurality of feature points and the plurality of feature points in the input face image and determining a group matching the face type of the input face image in the database;
A function generation unit for generating an impression score estimation function based on the type of impression of the face image and the determined group information; And
The position of the plurality of feature points is modified through the impression change score and the impression score estimation function to convert the face image to a desired impression while maintaining the identity of the inputted face image, An image converting unit for obtaining a correction vector obtained by modifying the distance vector through an estimation function and the impression change score and correcting a position of the plurality of feature points to a position where an error value between the distance vector and the correction vector is minimized; And an image processor for converting the image of the face into a face image. 11. The method of claim 10,
Wherein the identity of the face image includes a face type, a face size, a face length, and a facial expression. 11. The method of claim 10,
Wherein the image information extracting unit comprises:
And a distance vector extracting unit for extracting the plurality of feature points and the distance vector. 13. The method of claim 12,
Wherein the image information extracting unit comprises:
Calculating an Euclidean distance between a center vector that is an average value of distance vectors of the face images in each group and a distance vector of the input face image in the database, And a matching group determination unit for determining the group having the smallest center value as the matched group. 13. The method of claim 12,
Wherein the database includes distance vector information for each stored face image and information about an impression score of each stored face image. 15. The method of claim 14,
Wherein the impression change score input by the user and the impression score of each face image stored in the database are 0 or more and 1 or less. 11. The method of claim 10,
Wherein the image conversion unit comprises:
And an image information modification unit for inputting the distance vector to the impression score estimation function to obtain a correction vector as an output and correcting the positions of the plurality of feature points through the modification vector, Processing device. 17. The method of claim 16,
Wherein the image modifying unit comprises:
A distance vector of the input face image is input to the impression score estimation function to obtain an impression score of the input face image and a vector having an impression score equal to the impression score obtained by adding the impression score And a correction vector acquiring unit for acquiring the correction vector from the correction vector obtained by the correction vector acquiring unit. delete

Images