CN104346601B - Object identifying method and equipment - Google Patents

Abstract

The invention discloses object identifying methods and equipment.The equipment includes extraction unit, each object properties pair being configured in the set for pre-defined object properties, the feature corresponding to the object properties pair in the diversity extracting object region based on the object properties pair；And recognition unit, it is configured for the object properties of the feature recognition subject area of extracted subject area.

Description

Object recognition method and device

technical field

The present invention relates to methods and devices for object recognition in images. More specifically, the present invention relates to methods and devices for identifying object attributes of object regions in images.

Background technique

In recent years, object detection/recognition in images has been widely used in the fields of image processing, computer vision, and pattern recognition, and plays an important role in them. The object can be any of human face, hand, body, etc.

A common type of object detection/recognition is to detect and recognize faces in images. In face recognition, recognition of an attribute such as an expression of each face in an image containing at least one face image is generally achieved, and there are various techniques for realizing such face recognition.

The following will take the facial expression recognition of a face contained in an image as an example to explain the current technology for facial attribute recognition in an image in the prior art. The basic principle of the method for facial expression recognition follows the framework shown in Fig. 1.

More specifically, for the input face image, the facial expression recognition method first obtains the face area contained in the image (face detection), and then according to the facial feature points extracted in the face area, it may be in different poses The face alignment corresponding to the face region (face registration). Then, the method extracts features of the aligned face images (feature extraction), and finally determines the expression of the face corresponding to the face region from the extracted features.

For feature extraction, some methods focus on the salient region (salient region) in the face image, as shown in Figure 2 here, the salient region refers to the region in the face image that is usually considered to represent the characteristic part of the face ( such as the eye area, nose area, mouth area, etc.).

In such a case, the features of four salient regions (i.e., the left-eye region feature _flefteye , the right-eye region feature _frighteye , the nose region feature _fnose , and the mouth region feature _fmouth ) are respectively extracted, and passed These four salient region features are concatenated to represent the features of the face ( _ftotal ), thus,

f _total = f _{left eye} + f _{right eye} + f _nose + f _mouth

The feature f is _always used to predict the expression of a face corresponding to a face image.

Usually, such methods based on salient regions in the face region extract the features of the salient regions instead of the features of the entire region of the face image, and then predict the facial expression according to the extracted features, as shown in the prior art The flow chart of facial expression recognition based on salient regions in facial images is shown in the left part of FIG. 3 . The right part of Fig. 3 schematically shows an example of such a facial expression recognition method based on salient regions, wherein, after detecting several facial feature points in the face image, four salient regions (i.e., left eye region, right eye region, nose region and mouth region) are positioned accordingly.

US Patent Application US2012/0169895A1 in the name of Industrial Technology Research Institute (TW) discloses a method for capturing facial expressions based on salient regions in facial images. The method captures salient region features of the face in an image from four salient regions respectively to generate a target feature vector, and then compares the target feature vector with a plurality of previously stored feature vectors to generate parameter values. When the parameter value is above a threshold, the method selects one of the images as the target image. Based on this target image, further facial expression recognition and classification processes can be performed. For example, target images are recognized to obtain facial expression states, and images are classified according to the facial expression states.

Instead of salient regions, representative regions of other types of facial images can be used for facial attribute recognition.

US patent application US2010/0111375A1 in the name of Mitsubishi electric research laboratories, INC discloses a method for identifying facial attributes in an image based on a collection of patches contained in the facial image. More specifically, the method divides the face image into a set of sub-blocks, and compares each sub-block with the prototype sub-block one by one to determine the matching prototype sub-block, and according to the attribute set associated with the matching prototype sub-block Determines a set of attributes for a face. Here, the set of sub-blocks extracted in the method may be equivalent to each part in the salient region.

The US patent application US2012/0076418A1 in the name of Renesas Electronics Corporation discloses a face attribute estimation method and device. This method extracts a specific area from a face area, and sets a small area in the specific area. Then, the method uses a similarity calculation method to calculate the similarity between the small area and each of the stored face components one by one to determine the face attribute. Here, in addition to the number of specific regions, the specific regions used in this method may be equivalent to salient regions.

The above-mentioned methods in the prior art usually extract features from salient regions or their equivalent regions (such as multiple sub-blocks or a small specific region in a face image), and compare the extracted features with those corresponding to multiple known Each of a set of predefined features of facial attributes is compared (ie, one-to-one comparison) for facial attribute recognition.

Furthermore, the located salient regions or equivalent regions in the face image to be recognized do not change during recognition, so there is only one and constant feature vector from the face image for all comparisons during recognition. That is, only one feature vector from the facial image is used for comparison with each of a plurality of previously stored feature vectors corresponding to a plurality of known facial attributes.

However, using one constant feature of the face region to be recognized for all one-to-one comparisons during recognition may not be efficient enough to identify the face region accurately.

It should be noted that some salient regions may not be discriminative for some types of expressions. For example, for sad and neutral expressions, there is no great difference in the nose area, therefore, the nose area is not discriminative for sad and neutral expressions. Another problem is that some parts of the salient region are not discriminative. For example, the eyebrow part in the eye area is not discriminative for sad and neutral expressions. That is, if the located salient region and thus the features extracted from this region are constant for comparison with a predefined set of face attributes, some regions and some parts of the regions may be important for being in some expression pairs The recognition of some types of expressions is redundant.

As noted above, there remains a need for a method that can accurately identify attributes of face regions based on more distinctive features from face regions in images.

Contents of the invention

The present invention was developed for the recognition of objects in images and aims to solve the problems described above.

According to one aspect of the present invention, there is provided a method for identifying object regions in an image, the method comprising an extraction step for, for each object attribute pair in a predefined set of object attributes, based on the object attribute The dissimilarity of the pair extracts features of the object region corresponding to the pair of object attributes; and an identifying step for identifying object attributes of the object region based on the extracted features of the object region.

According to another aspect of the present invention, there is provided an apparatus for identifying an object region in an image, comprising: an extraction unit configured for each object attribute pair in a set of predefined object attributes, based on the The dissimilarity of object attribute pairs extracts features of the object region corresponding to the object attribute pairs; and a recognition unit configured to recognize object attributes of the object region based on the extracted features of the object region.

According to the method and device of the present invention, for each object attribute pair in the set of predefined object attributes, based on the dissimilarity of the object attribute pair, the feature of the object area corresponding to the object attribute pair is extracted, and the feature is used for object recognition. Therefore, recognition efficiency and accuracy can be improved.

Other features of the present invention will become apparent from the following description of exemplary embodiments read with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate the embodiments of the invention and together with the description serve to explain the principles of the invention. In the drawings, like reference numerals indicate like items.

Fig. 1 shows a typical process of facial expression recognition in the prior art.

Figure 2 shows typical salient regions in a face.

FIG. 3 is a flowchart showing a facial expression recognition method in the prior art.

FIG. 4 is a block diagram showing an exemplary hardware configuration of a computer system that can implement an embodiment of the present invention.

FIG. 5 is a flowchart illustrating an object attribute recognition method according to the present invention.

FIG. 6 is a block diagram showing an object attribute recognition device according to the present invention.

FIG. 7 is a diagram for explaining face regions in a face image.

Fig. 8 schematically shows feature points in the face area.

Fig. 9 is a flowchart showing the procedure in the extraction step.

FIG. 10 schematically shows the localization of organ regions in the face region.

Fig. 11 schematically shows an example of facial expression pairs.

FIG. 12 is a flowchart schematically showing determination of templates for facial expression pairs.

Figure 13 shows several exemplary averaged images.

Fig. 14 shows correspondingly segmented images for each expression in a pair of facial expressions.

FIG. 15 shows templates of facial expression pairs obtained from segmented images for each of the facial expression pairs.

Fig. 16 shows the positioning of distinct pixel blocks in the face regions of a facial expression pair depending on the template of the facial expression pair.

Fig. 17 is a flowchart showing the procedure in the feature extraction step.

Figure 18 is a flowchart illustrating the process in one implementation of the identifying step.

Fig. 19 is a flowchart showing the procedure in another implementation of the identifying step.

Detailed ways

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

It should be noted that like reference numerals and letters designate like items in the drawings, and thus once an item is defined in one figure, it need not be discussed for subsequent figures.

First, the meaning of certain terms used in the context of the present disclosure will be explained.

In the context of this disclosure, an image will refer to various types of images, such as color images, grayscale images, and the like. Since the processing of the present invention is mainly performed on grayscale images, an image in this disclosure will refer to a grayscale image including a plurality of pixels unless otherwise stated.

It should be noted that the inventive solution is also applicable to other types of images, such as color images, as long as such images can be converted into grayscale images and the inventive processing can be performed on the converted grayscale images.

An image may typically contain at least one object image, and an object image typically contains an object region, so in the context of this disclosure, object image and object region are equivalent to each other and are used interchangeably. A common in-image object is a face in an image.

A feature of an object region in an image is typically a feature representative of the characteristics of such an object region, and may typically be a color feature, a texture feature, a shape feature, or the like. A commonly used feature is a color feature, which is a global feature representing an image and can usually be obtained through a color histogram based on each color bin. Image features are usually obtained in the form of vectors, each component of which corresponds to a color segment.

Object properties refer to apparent states of objects that may correspond to different conditions, and object properties may belong to different categories. Taking the face as an example, the category of the facial attribute can be selected from the group containing facial expressions, the gender of the person corresponding to the face when the face is a human face, and the age of the person. Not limited thereby, and may be other categories as well. When the facial attribute corresponds to a facial expression, the facial attribute may be an expression (eg, sad, smiling, laughing, etc.).

Of course, the object attributes are not limited thereto. For example, the object may be a human body, and the object attributes may correspond to different body states when the person is running, standing, kneeling, or lying down.

An object attribute pair is a pair of any predefined number of object attributes contained in a predefined set of object attributes in which all object attributes are distinguishable in a certain set of categories and which can be identified by Pre-prepared, the predefined set of object attributes may form at least one object attribute pair, and each object attribute pair has the same number of object attributes.

The object attributes contained in the object attribute pairs may be arbitrarily selected from the predefined set of object attributes, and in such a case, the predefined set of object attributes may contain C _n ^t object attribute pairs, where n is the number of object properties in the set, and t is the number of object properties contained in the object property pair.

Preferably, the number of object attributes contained in an object attribute pair may be two.

Preferably, the object attributes in an object attribute pair may be object attributes that differ greatly and are even opposite. For example, taking a face as an example, the object attribute pair may be specifically composed of a laughing expression and a crying expression, so that the extracted parts of such an object attribute pair are more distinctive.

In the present disclosure, the terms "first", "second", etc. are only used to distinguish elements or steps, and are not intended to indicate chronological order, priority or importance.

FIG. 4 is a block diagram showing a hardware configuration of a computer system 1000 in which an embodiment of the present invention can be implemented.

As shown in FIG. 4 , the computer system includes a computer 1110 . Computer 1110 includes processing unit 1120, system memory 1130, non-removable non-volatile memory interface 1140, removable non-volatile memory interface 1150, user input interface 1160, network interface 1170, video interface 1190, and output peripheral interface 1195, which are connected through the system bus 1121.

The system memory 1130 includes a ROM (Read Only Memory) 1131 and a RAM (Random Access Memory) 1132 . A BIOS (Basic Input Output System) 1133 resides in the ROM 1131 . Operating system 1134 , application programs 1135 , other program modules 1136 , and some program data 1137 reside in RAM 1132 .

A non-removable non-volatile memory 1141 such as a hard disk is connected to the non-removable non-volatile memory interface 1140 . The non-removable non-volatile memory 1141 can store an operating system 1144 , application programs 1145 , other program modules 1146 and some program data 1147 , for example.

Removable nonvolatile memory such as a floppy disk drive 1151 and a CD-ROM drive 1155 is connected to the removable nonvolatile memory interface 1150 . For example, a floppy disk 1152 can be inserted into the floppy disk drive 1151 , and a CD (Compact Disc) 1156 can be inserted into the CD-ROM drive 1155 .

Input devices such as a mouse 1161 and a keyboard 1162 are connected to the user input interface 1160 .

The computer 1110 can be connected to a remote computer 1180 through a network interface 1170 . For example, network interface 1170 may be connected to remote computer 1180 via local area network 1171 . Alternatively, network interface 1170 may be connected to modem (modulator-demodulator) 1172 , and modem 1172 is connected to remote computer 1180 via wide area network 1173 .

Remote computer 1180 may include memory 1181 , such as a hard disk, that stores remote application programs 1185 .

The video interface 1190 is connected to a monitor 1191 .

The output peripheral interface 1195 is connected to a printer 1196 and a speaker 1197 .

The computer system shown in Figure 4 is illustrative only and is in no way intended to limit the invention, its application or use.

The computer system shown in FIG. 4 may be implemented for either embodiment as a stand-alone computer, or as a processing system in a device, where one or more unnecessary components may be removed or one or more additional components may be added.

Hereinafter, an object recognition method according to a basic embodiment of the present invention will be described with reference to FIG. 5 , which shows a process in the method according to a basic embodiment of the present invention.

In step S100 (hereinafter referred to as the extraction step), for each object attribute pair in the set of predefined object attributes, based on the dissimilarity (dissimilarity) of the object attribute pair, extract the object region corresponding to the object attribute pair Characteristics.

As mentioned above, all object attributes of the predefined set of object attributes belong to the same category, and an object attribute pair may consist of any predetermined number (such as two) of object attributes contained in the predefined set of object attributes.

Alternatively, the object attribute pair may be a predetermined number (such as two) of object attributes satisfying a predetermined relationship between them.

In one implementation, the object region may be an object region that has been aligned, and the alignment of the object region may be achieved in various ways, such as based on feature points detected in the object region. It should be noted that whether the object regions are aligned or not is not essential to the implementation of the extraction operation.

In step S200 (hereinafter referred to as an identifying step), object attributes of the object area are identified based on the extracted features of the object area.

In one implementation, the process in the extraction step may comprise a process for locating at least one block in the object region corresponding to a template of the object attribute pair (locating step), the template characterizing the relationship between the object attribute pair dissimilarity; and a process for extracting features of the object region corresponding to the object attribute pair based on the located at least one block (feature extraction step).

Here, the template can be regarded as a dissimilarity template representing the dissimilarity between the object attributes of the object attribute pair, and is generally formed by at least one dissimilar pixel block between images of the object attributes contained in the object attribute pair. In practice, each distinct pixel block may correspond to a corresponding pixel block between images of a predetermined number of object attributes contained in an object attribute pair, the corresponding pixel blocks are located at corresponding positions and have corresponding sizes in each image, wherein each The positions and sizes of distinct pixel blocks in an image may be mapped to each other depending on a predetermined rule, such as a ratio between sizes of images depending on respective object attributes when respective images have different sizes.

Preferably, the image of the object region and the object attributes contained in the object attribute pairs may be pre-processed, such as aligned, so as to have the same size, and in this case, each of the distinct pixel blocks in the template may Corresponding pixel blocks between images corresponding to a predetermined number of object attributes included in the object attribute pairs are located at the same position in the respective images and have the same size.

Therefore, at least one block positioned for the object attribute pair from the object region may be a pixel block positioned according to such positions and sizes of different pixel blocks, as long as the pixel blocks can be mapped to each other according to a predetermined rule, And preferably the pixel blocks have the same position and size.

The size of each pixel block can be set arbitrarily without affecting the implementation of the solution of the present invention.

In one implementation, the template of an object attribute pair can be realized by dividing two average object region images respectively corresponding to the two object attributes included in the object attribute pair into a plurality of blocks corresponding to each other; extracting and a feature of each of the plurality of blocks of each divided average object area image corresponding to each object attribute; determining a similarity between features of corresponding blocks in the two divided average object area images; and Blocks in the two divided average object area images are selected to form a template, the similarity between which blocks is below a pre-defined threshold.

Here, the corresponding division means that each image of the object attribute in the object attribute pair can be divided in a corresponding mode, so that each of the divided blocks in one object attribute image can be mapped to another object according to a predetermined rule Each of the divided blocks in the attribute image. Preferably, the individual images of the object attributes in an object attribute pair are of the same size, and thus the division mode for each image is the same and has the same scale, so that a divided block in one object attribute image is identical to the other object attribute image The corresponding divided blocks in have the same location and size. The division pattern can be any pattern, such as grid or the like.

Templates for object attribute pairs can be prepared and stored in advance, or can be prepared during an extraction operation. The operation of obtaining templates of object attribute pairs may or may not be included in the extraction step.

The average object region image corresponding to an object attribute can be pre-prepared in various ways, and in a general implementation, can be prepared by averaging a plurality of similar object region images of the same size corresponding to the same object attribute.

Preferably, the positioning process can be performed based on an auxiliary region (auxiliary region) included in the object region, so as to further improve operation efficiency. The auxiliary region can be positioned in various ways, such as depending on the location of the identified feature points in the object region. In such a case, the locating process may locate at least one block in the auxiliary region corresponding to the template characterizing the dissimilarity of the object-attribute pair, and the template characterizing the dissimilarity of the object-attribute pair may also be based on the object attribute in the object-attribute pair Such auxiliary regions in the image are determined instead of the entirety of the image based on the object attributes in the object attribute pairs.

In one implementation, the feature extraction process may include extracting features from each of the at least one block located in the object region, and concatenating the extracted features of the respective blocks as features of the object region. Therefore, the final extracted features are usually expressed in the form of vectors, and each component in the vector corresponds to each block.

During the identifying step, the identification of object properties can be realized in various ways.

In one implementation, identification can be achieved in a so-called "one against one" fashion, where object attributes can be voted in C _n ^t rounds for a predefined set of object attributes , where n is the number of object properties contained in the set, and t is the number of object properties contained in object property pairs and is preferably 2. The object attribute with the highest score will be determined as the object attribute.

More specifically, the identification process may include an identification step for, for each object attribute pair in the predefined set of object attributes, identifying the object corresponding to the object region based on the features of the object region corresponding to the object attribute pair. One object attribute among the two object attributes included in the object attribute pair, and the score of the object attribute corresponding to the object area is increased by a predetermined value, wherein all object attributes included in the predefined object attribute set have the same initial score; and an attribute determining step, which is used to determine that the object attribute with the highest score in the set of predefined object attributes is the object attribute of the object area.

In another implementation, the recognition can be implemented in a so-called "one beating one" manner, where the number of predefined object properties contained in the set of predefined object properties is n cases Here, object attributes may be determined in n-1 rounds, where only the winning attribute for one object attribute pair in a round will advance to the next round, and the final winning attribute will be determined as the object attribute.

More specifically, the identification process may include an identification step for, for an object attribute pair in the set of predefined object attributes, identify the object corresponding to the object area based on the feature corresponding to the object attribute pair of the object area One object attribute among the two object attributes contained in the object attribute pair, and the attribute determining step, are used to divide the one object attribute based on the object attribute corresponding to the object area and the set of predefined object attributes Determine the object attributes of the object region for the remaining object attributes other than the pair, wherein, if the number of remaining object attributes is equal to 0, then the object attributes corresponding to the object region are determined as the object attributes of the object region, otherwise the object region The corresponding object attributes and the remaining object attributes in the predefined object attribute set except for the one object attribute pair are regrouped into a new object attribute set, and the identification is performed sequentially for the new object attribute set The steps and properties determine the steps.

It should be noted that the above method can be performed each time for one object region in an image which can contain at least one object region, and can be repeated as many times as the number of object regions, wherein one object region contains only one object to be recognized.

FIG. 6 is a block diagram showing an object recognition device according to the present invention.

The apparatus 600 for identifying an object region in an image may include an extraction unit 601 configured to, for each object attribute pair in a set of predefined object attributes, extract a correspondence of the object region based on the dissimilarity of the object attribute pair the features of the object attribute pair; and the identification unit 602 is configured to identify the object attributes of the object area based on the extracted features of the object area.

Preferably, the extraction unit 601 may include a positioning unit 601-1 configured to locate at least one block corresponding to the template of the object attribute pair in the object region, the template representing the dissimilarity between the object attribute pair; And a feature extraction unit 601-2 configured to extract features of the object region corresponding to the object attribute pair based on the located at least one block.

Preferably, the positioning unit 601-1 may include a unit configured to locate the auxiliary area in the object area depending on the position of the identified feature point in the object area; and configured to locate the auxiliary area corresponding to A unit of at least one block of a template of an object attribute pair characterizing a dissimilarity between the object attribute pair.

Preferably, the feature extraction unit 601-2 may include a unit configured to extract features from each of the at least one block in the object area, and configured to concatenate the extracted features of each block as an object The unit that characterizes the region.

Preferably, the recognition unit 602 may include an identification unit 602-1 configured to, for each object attribute pair in the set of predefined object attributes, based on the feature identification corresponding to the object attribute pair of the object area and the One of the two object attributes contained in the object attribute pair corresponding to the object area, and the score of the object attribute corresponding to the object area is increased by a predetermined value, wherein the predefined object attribute set contains All the object attributes have the same initial score; and the attribute determining unit 602-2 is configured to determine the object attribute with the highest score in the set of predefined object attributes as the object attribute of the object area.

Additionally or as an alternative, the identifying unit 602 may include an identifying unit 602-3 configured to, for an object attribute pair in a set of predefined object attributes, based on the feature of the object region corresponding to the object attribute pair Identify one object attribute among the two object attributes included in the object attribute pair corresponding to the object area, and the attribute determining unit 602-4 is configured to be based on the object attribute corresponding to the object area and the predefined The remaining object attributes in the set of object attributes except the one object attribute pair determine the object attributes of the object region, wherein, if the number of remaining object attributes is equal to 0, the object attribute corresponding to the object region is determined as The object attribute of the object area, otherwise the object attribute corresponding to the object area and the remaining object attributes in the set of predefined object attributes except the one object attribute pair are regrouped into a new object attribute set, And for the new object attribute set, the identifying operation and the attribute determining operation are executed in sequence.

Templates characterizing dissimilarities between pairs of object attributes may be pre-formed and stored separately from device 600 as described above. Additionally or alternatively, the device 600 may comprise a unit configured to form a template of an object attribute pair characterizing the dissimilarity between the object attribute pair in the manner described above.

[Favourable technical effect]

In summary, the present invention provides a new idea for object attribute recognition of object regions in images, wherein the concept of object attribute pairs is introduced to improve feature extraction and recognition of object regions.

More specifically, the dissimilarity between the object attributes contained in the object attribute pair is used to extract the dissimilar pixel blocks in the object region for the object attribute pair, and the features of the extracted object region are used to determine the difference between the object region and Which object property in the object property pair corresponds to. Accordingly, extraction and recognition of features of object regions are performed pair by pair, whereby recognition efficiency and accuracy can be improved.

It should be noted that such distinct pixel blocks of the object region are determined and extracted for each object attribute pair in the set of predefined object attributes used as the basis for comparison in each round, and may reflect the object attribute pairs contained in The dissimilarity between the object properties of . Furthermore, such extracted parts may be adaptively changed during recognition, ie, distinct pixel blocks of an object region may change depending on the contrast in each round of comparison, rather than remain constant.

Therefore, some parts of the object region that may be common rather than distinctive to the object attribute pair may not be extracted, and the extracted part may more accurately reflect the dissimilarity between the object attributes contained in the object attribute pair, and have It helps to accurately determine which of the object attributes contained in the object attribute pair the object region corresponds to, so that the object attributes of the object region can be more accurately determined.

In the following, in order to facilitate a thorough understanding of the implementation of the invention, a face is used as an example of an object to be recognized in order to explain an exemplary implementation of the solution of the invention. It should be noted that the solution of the invention can also be applied to other types of objects.

For a face region in an image to be recognized, its attributes may belong to various categories. For example, the category of the face attribute may be one selected from the group including facial expression, gender and age of the person corresponding to the face when the face is a human face. Of course, the categories of face attributes are not limited thereby, and may be other categories than the above-mentioned ones.

[Example 1]

Hereinafter, a process for recognizing facial attributes (such as facial expressions) of a face region in an image according to the present invention will be described.

In general, for a facial region in an input image whose expression is to be recognized, for each facial expression pair in a set of predefined facial expressions, based on the facial expressions contained in the facial expression pair, The feature of the facial region corresponding to the facial expression pair is extracted based on the dissimilarity between them, and then the facial expression of the facial region is recognized based on the extracted features of the facial region. When there are multiple faces in the input image, this process is repeated as many times as the number of faces.

The details of this process are described below.

Initially, for an input image that may contain at least one face, face regions in the input image are detected, usually one face region corresponds to one face in the image. Fig. 7 shows a rectangular face region detected from an input image.

Preferably, before the detected face regions are used for feature extraction, the face regions are usually aligned respectively, and this alignment can be performed in various ways.

In one implementation, the face regions are aligned based on a predetermined number of feature points extracted from the face image, wherein the number of feature points can be set based on the operator's experience and is not limited to some specific number. The feature point extraction method can be ASM disclosed in such as Xudong Cao, Yichen Wei, Fang Wen, Jian Sun. Face alignment by explicit shape regression CVPR, 2012, and D. Cristinacce and T. F. Cootes. Boosted regression active shape models. BMVC, 2007. It should be noted that the feature point extraction method is not limited thereby, and may be any other method known in the art.

Fig. 8 schematically shows extracting 7 feature points from the face area, and as shown in Fig. 8, these 7 feature points are: two corners of each of the two eyes, the tip of the nose, and two corners of the mouth. corner of the mouth.

Alignment can be performed as follows. It should be noted that the following procedures for alignment known in the art are exemplary only, and alignment may also be performed by other procedures.

At the time of alignment, the average position of the extracted 7 feature points is calculated based on a predetermined number of manually labeled samples. Assuming that there are n labeled samples, the average position of seven points P _i ( _xi , y _i ) (i=1~7) is calculated as follows:

Here, x and y may represent a horizontal axis position and a vertical axis position.

The average position of these seven points P _i (xi _, y _i ) (i=1˜7) is defined as the objective face, and an affine mapping process is used to align the input face with the objective face.

The size of the aligned faces may be 200*200 pixels. It should be noted that the size of the aligned face regions is not limited and may be any other size.

Next, the possibly aligned face regions in the input image are subjected to feature extraction. Fig. 9 is a flowchart showing the process for feature extraction, where step S101 is shown by a dotted line, which means that this step is optional.

In this feature extraction process, at least one distinct pixel block corresponding to a template of a facial expression pair from a predefined set of facial expressions, which template characterizes the face in the facial expression pair, is located in the face region The dissimilarity between facial expressions (S102), and then, based on the located at least one dissimilar pixel block, extract the features of the face region corresponding to the facial expression pair (S103).

The template of the facial expression pair can be formed by at least one block corresponding to each other among the images of the facial expressions in the facial expression pair, and the at least one block can reflect the difference between the facial expression images in the facial expression pair. dissimilarity, and the details of the template will be described in detail later.

For the process of locating different pixel blocks, in one implementation, for each facial expression pair, the template of the facial expression pair can be directly relied on the predetermined corresponding relationship (such as, in the same position and with the same block size) to locate at least one distinct block of pixels in the face image. It should be noted that the correspondence relationship between at least one distinct pixel block and the blocks in the template is not thus limited and other rules may be satisfied.

In another implementation, the process (S101) for locating an auxiliary region (such as an organ region) in a facial image may be performed in advance, so that the positioning of distinct pixel blocks in the facial image may only be performed for the facial image It is performed on auxiliary regions in the face image, such as organ regions in the face image, rather than the entire face image.

The auxiliary area can be of any shape, such as rectangular, square, etc., and can be of any size according to the operator's experimentation.

As shown in Figure 10, four organ regions can be located, including two eye regions, one nose region and one mouth region. During recognition, the sizes of these four regions are fixed for each aligned face. For example, in a face of 200*200 pixels, the size of the eye rectangle is 80*60, the size of the nose rectangle is 140*40, and the size of the mouth rectangle is 140*80.

Preferably, the location of the organ region can be determined by feature points in the face image. Let the origin of the coordinates be the upper left corner of the image. When locating the left-eye area, the center of the rectangular area may coincide with the midpoint of the line AB in FIG. 10 . Similarly, the center of the rectangle of the right eye region may be the same as the midpoint of line CD in FIG. 10 . For the nose area, if the position coordinates of the upper left corner are (n1, n2), the position coordinates of the lower right corner are (n3, n4), and the position coordinates of the nose tip E are (e1, e2), then the coordinates of these three points satisfy The following equation:

e1=α*(n1+n3), e2=n2+β*(n4－n2),

Here, 0.3≤α≤0.7, 0.5≤β≤0.8.

For the mouth region, let H(h1, h2) be the midpoint of the line FG, the upper left corner of the mouth region is (m1, m2), and the lower right corner is (m3, m4). The relationship of the coordinates satisfies the following equation:

h1=γ*(m1+m3), h2=m2+δ*(m4－m2),

Here, 0.3≤γ≤0.7, 0.3≤δ≤0.6.

Thereby, four auxiliary regions in the facial image can be located, and different pixel blocks in the facial image for a facial expression pair can be located only in the auxiliary region with reference to the template of the facial expression pair. In such a case, preferably, the template of the facial expression pair may only be composed of distinct blocks in the auxiliary regions in each facial expression in the facial expression pair, the auxiliary regions in the facial expressions are formed in a predetermined manner Corresponding to (such as being at the same location and having the same size as) the auxiliary region positioned in the face region.

Hereinafter, templates of facial expression pairs will be described in detail.

In the present invention, a template is determined for each pair of facial expressions in a set of predefined facial expressions. A facial expression pair may contain a predetermined number of facial expressions, and preferably two facial expressions, and any two of them may form a facial expression pair.

For example, as shown in FIG. 11 , it is assumed that the predefined facial expression set includes three facial expressions: sad, neutral and angry, so there may be C ₃ ² facial expression pairs. For example, as shown in FIG. 11 , a facial expression pair can be composed of a sad expression and a neutral expression, a facial expression pair can be composed of a neutral expression and an angry expression, and a facial expression pair can be composed of a sad expression and an angry expression.

In another implementation, the facial expressions included in the facial expression pair may be facial expressions that are very different from each other or even opposite to each other. For example, a facial expression pair may be specifically composed of a laughing expression and a crying expression, so that the extracted blocks for such a facial expression pair are more distinctive.

FIG. 12 shows a flowchart of a process for configuring templates for facial expression pairs. Such a process may be pre-executed before performing the process of the present invention, so that templates for all facial expression pairs contained in a predefined set of facial expressions may be preconfigured and stored. Alternatively, such processes may be performed in time with the performance of the processes of the present invention.

First, two average face images respectively corresponding to two facial expressions contained in a facial expression pair are divided into a plurality of blocks corresponding to each other.

An average face image for each expression can generally be constructed by averaging the aligned faces in the same expression. Taking the average face of a big smile as an example, assuming that there are N aligned big smile samples, the average face image I of a big smile is obtained by the following formula:

This equation means that the gray values of the aligned pixels in the face image are added together with a weight of 1/N to obtain the average face image of a big smile.

FIG. 13 exemplarily shows average facial images of laughing, neutral, sad, and smiling, and such average facial images are usually prepared in advance based on a facial expression database and stored.

Fig. 14 schematically shows the corresponding division of two average face images. The two average face images are respectively divided using the same pattern such as grid, and the size of divided blocks is not limited. For example, in an average face image of 200*200 pixels, the block size is 10*10 pixels.

It should be noted that the division patterns of the respective average facial images are not limited thereby, and may correspond to each other in other ways, for example, when the respective average facial images have different sizes, each block in the division pattern may be divided according to the respective average The ratios of the sizes of the face images correspond to each other.

Next, the features of each of the blocks of each divided average face image will be extracted. There are various extraction methods, and each of the various extraction methods can be applied to this process.

As known in the art, feature extraction methods can be such as disclosed in Timo Ojala, Matti Pietikainen, and Topi Maenpaa, Multi-resolution gary-scale and rotation invariant texture classification with local binary patterns, IEEE Transaction on Pattern Analysis and Machine Intelligence, 2002 Local Binary Pattern (LBP), or Local Phase Quantization (LPQ) such as disclosed in VilleOjansivu, and Janne Heikkila, Blur insensitive texture classification using local phase quantization, ICISP2008.

In the case of LBP, the block size is the same as that of the distinct pixel block, and the total number of sections is 59, for example. Therefore, each block LBP feature has 59 dimensions. The feature calculation process is briefly described as follows:

1) For each pixel in the input image, calculate LBP _{8, 1}

a) Get the value of the center pixel as the current pixel

b) Extract pixel values in eight adjacent regions

c) Calculate g _P by bilinear interpolation, (P=0, 1, ..., 7)

Here, _gp is the grayscale value of one of the adjacent pixels, and _gc is the grayscale value of the central pixel.

2) Using the LBP value mapping table disclosed in Ville Ojansivu and Janne Heikkila, Blur insensitive texture classification using local phase quantization.ICISP2008, a 59-dimensional LBP histogram is constructed by adding the LBPs of each pixel in the block together.

Next, the similarity between features of corresponding blocks in the two divided average face images is determined.

For example, the similarity of corresponding blocks (refer to FIG. 14 ) may be determined using Euclidean distance. Assuming two feature vectors f1=<a1, a2,...an>, f2=<b1, b2,...bn>, the similarity of f1 and f2 is as follows:

Therefore, the similarity between the two divided average face images can be determined block by block as described above. It should be noted that the determination of similarity is not thus limited and may be accomplished in other ways known in the art.

Finally, blocks in the two divided average face images whose similarity between them is lower than a predetermined threshold are selected to form a template.

More specifically, the similarities of corresponding blocks in the two divided average face images are sorted in ascending order, a first predetermined number of pairs of blocks is selected, and indexes of the predetermined number of pairs of blocks are saved as as a template for emoticon pairs. This predetermined number (also corresponding to a predefined threshold, which may be the similarity of the last pair of the predetermined number of pairs) may be optimized experimentally. An example of a template for an emoticon pair is shown in FIG. 15 .

Thus, for each expression pair, a set of distinct pixel blocks can be located in the face region according to the template for that expression pair that has been formed as described above. Figure 16 illustrates this process.

First, the input face image is divided into blocks according to the template, and the block division may be the same as the template division (such as having the same pattern and the same block size). Then, since the template of the expression pair adjusts the indices of the different pixel blocks in the face image, the different pixel blocks are positioned in the aligned face image according to the indices in the template. Actually, the size of the distinct pixel block may be 10*10 pixels.

As described above, when the auxiliary area in the face image is pre-located, the above-described process may be performed only for the auxiliary area.

Based on such positioned distinct pixel blocks in the face region to be recognized, features of the face region can be extracted. Figure 17 shows a flowchart for such feature extraction. In particular, features may be extracted for each of at least one block in the face area (S1031), and the extracted features of the respective blocks are concatenated as features of the face area (S1032).

The feature extraction method may be any method known in the art, and for example, may be the same as the method (eg, LBP) in the feature extraction process described above.

Then, the features of all different pixel blocks are concatenated to represent the features of facial expressions. The dimension of the final vector is 59*n, where n is the total number of distinct pixel blocks, and 59 represents the total number of segments for feature extraction and can be any other number. In the case of using only distinct pixel blocks in the auxiliary region, in each organ region, the features of each distinct pixel block are concatenated in a fixed order, and then the features of the four obtained organ regions are concatenated.

Recognition of facial expressions of facial images will be described below.

Fig. 18 is a flow chart showing an implementation of the recognition process, in which the recognition is carried out in a so-called "one-to-one" manner, in which facial expressions are compared to predefined facial expressions The set is voted C _n ^t times, where n is the number of facial expressions contained in the set and t is the number of facial expressions contained in a pair of facial expressions, i.e., one vote is taken for a pair of facial expressions, and The facial expression with the highest score will be determined as the final facial expression.

_Each expression _{is denoted} as B ₁ , . For each facial expression pair, when a facial region is determined to correspond to one of the facial expressions (B _i ), the score of such facial expression is increased by a constant value, e.g. f(B _i )=f (B _i )+1.

Finally, the facial expression corresponding to the maximum score f(B)=max{f(B ₁ )...f(B _n )} is determined as the facial expression of the face region.

Fig. 19 is a flow chart showing another implementation of the recognition process, in which the recognition can be implemented in a so-called "one wins one" approach, in which the set of predefined facial expressions contains In the case where the number of predefined facial expressions is n, the facial expressions of the face region will be determined through n-1 rounds, where only the winning expression of the facial expression pair in one round will advance to the next One round, and the winning expression is determined as the final facial expression.

Let each expression be denoted as B ₁ , . . . , B _n , and a facial expression pair contains two facial expressions. Initially, a facial expression pair (B _i , _Bi ) is arbitrarily selected from a set of predefined facial expressions, and for this facial expression pair, it is determined which facial expression the facial region corresponds to. For example, it is determined that the facial expression of the face area is B _i .

Then, the expression _Bi will be excluded from the initial set of expressions, and the remaining expressions will be reorganized into a new set of expressions. For this new collection, the above process is performed again.

Therefore, such a process will be performed for n-1 rounds, and finally the remaining expressions are recognized as those of the face image.

Overall, the "one wins one" approach is more efficient than the "one-on-one" approach with about the same accuracy.

Expression determination for a facial expression pair may be achieved in any manner known in the art, such as by a classifier. In the case of classifiers, for expression pairs, the feature vectors are classified by a binary classifier. A linear SVM such as that disclosed by Chih-Chung Chang and Chih-Jen Lin LIBSVM: a library for support vectormachines.2011 is adopted as the classifier. The decision function is sgn(w ^T f+b), where w and b are stored in a dictionary, w is the weight trained for the SVM and b is the bias item, and f is the feature vector of the face region .

Experimental results

Table 1 shows the datasets used in the experiments. The images in this dataset are frontal face images with natural expressions downloaded from the Internet.

Table 1

laughing out loud neutral sad Smile training set 2042 1976 1746 2140 test set 717 487 1302 518 total 2759 2463 3048 2658

Table 2 shows the performance (eg, facial expression recognition accuracy) of the solution of the present application compared to such as US patent application US2012/0169895.

Table 2

The confusion matrix of the prior art is shown in Table 3, and the confusion matrix of the present application is shown in Table 4.

table 3

Table 4

[Example 2]

Hereinafter, a process for identifying attributes of a face in an image, such as the age of the face, according to the present invention will be described.

There are collections of pre-defined age groups, including children, teens, adults, seniors, etc. Age stage pairs will then be configured in this predefined set of age stages, and the age of the face will be determined pair by pair. This process can be implemented similarly to that in Example 1.

More specifically, initially, the process detects faces and locates face regions in an input face image.

Next, the process locates the distinct pixel blocks in the face region for each age-stage pair according to the trained template for each age-stage pair.

Next, the process obtains features of the face region based on the distinct pixel blocks used to classify each age-stage pair. The characteristics of an age-stage pair can be represented by concatenating the characteristics of different pixel blocks.

Next, the process determines the age stage of the face for each age stage pair, and integrates the classification results to determine the age stage of the face.

Preferably, the faces may be aligned prior to locating the distinct pixel blocks.

Preferably, before or during locating the different pixel blocks, the auxiliary region can be located in the face area, so that during locating the different pixel blocks and subsequent operations, only the auxiliary region can be processed.

It should be noted that the above examples are only illustrative and not restrictive. The solution method of the present invention is not thus limited, and can be applied to other types of object attribute recognition.

[Industrial applicability]

The present invention can be used in a variety of applications. For example, the invention can be applied to detecting and tracking the state of objects in images, such as cameras, audience response systems, and smile detection in automatic photo annotation.

More specifically, in one implementation, an object can be detected, and then the methods of the present invention can be utilized to identify attributes of the object.

In the case of a camera application, an image is captured by a camera. The system selects face images from the captured images through face detection techniques. Input the facial image into the facial expression recognition module. Some pre-defined expressions (eg, happy, sad, neutral, etc.) are recognized. Then, the recognition result is input into the evaluation module, which evaluates the meeting effect according to the expressions of the audience. The system finally outputs the evaluation result of the meeting effect.

The methods and systems of the present invention can be implemented in a number of ways. For example, the methods and systems of the present invention may be implemented by software, hardware, firmware, or any combination thereof. The sequence of steps of the method described above is illustrative only, and unless specifically stated otherwise, the steps of the method of the present invention are not limited to the sequence specifically described above. Furthermore, in some embodiments, the present invention can also be embodied as a program recorded in a recording medium, including machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also covers a recording medium storing a program for implementing the method according to the present invention.

While the invention has been described with reference to exemplary embodiments, those skilled in the art will appreciate that the foregoing examples are illustrative only and are not intended to limit the scope of the invention. It will be appreciated by those skilled in the art that the above-described embodiments may be modified without departing from the scope and spirit of the invention. The scope of the present invention is defined by the appended claims, which are to be given the broadest interpretation so as to embrace all such modifications and equivalent structures and functions.

Claims (10)

1. a kind of object identification device, including：

It is multiple to determine to be configured as the multiple subclass in the set for pre-defined object properties included for determination unit Object properties pair, wherein each object properties of each object properties centering are respectively from different subclass, and the subclass Middle attribute is the different expressions of face；

Extraction unit, be configured as the diversity extracting object region based on the object properties pair corresponds to the object properties pair Feature；And

Recognition unit is configured for the object properties of the feature recognition subject area of extracted subject area.

2. equipment according to claim 1, wherein the subject area is face area, and the object properties are Face's attribute, and

Wherein, face's attribute is countenance.

3. equipment according to claim 1, wherein the subject area is to have been based on the spy being identified in subject area Levy the subject area of point alignment.

4. equipment according to claim 1, wherein the extraction unit includes：

Positioning unit is configured for positioning at least one of the template corresponding to the object properties pair in the subject area Block, the template characterize the diversity between the object properties pair；And

Feature extraction unit, be configured for positioned at least one block extract the subject area correspond to the object The feature of attribute pair.

5. equipment according to claim 4, wherein the positioning unit includes：

The auxiliary area being configured in the position positioning subject area dependent on the identified characteristic point in subject area Unit；And

It is configured for positioning different between the characterization object properties pair corresponding to object properties pair in the auxiliary area The unit of at least one block of the template of property.

6. equipment according to claim 4 or 5, wherein the phase between the characterization object properties pair of object properties pair The anisotropic template is formed in the following way：

Two mean object area images for corresponding respectively to two object properties that the object properties centering includes are divided into Multiple pieces to correspond to each other；

Corresponding with each object properties each divided mean object area image of extraction it is multiple it is in the block each Feature；

Determine the similitude between the feature of the corresponding blocks in the two divided mean object area images；And

Select the block as follows in the two divided mean object area images to form template, it is similar between block Property less than pre-defined threshold value.

7. equipment according to claim 4, wherein the feature extraction unit includes：

It is configured for from least one unit in the block for extracting feature in each in subject area, and

It is configured for unit of each piece will extracted of the feature connection as the feature of subject area.

8. equipment according to claim 1, wherein the recognition unit includes：

Identify unit, each object properties pair being configured in the set for pre-defined object properties, based on pair Two objects for including as the signature identification corresponding to the object properties pair in the region subject area and the object properties centering Which of attribute object properties are corresponding, and the score of the object properties corresponding to the subject area is increased and is made a reservation for Value, wherein whole object properties initial score having the same included in the set of the pre-defined object properties；With And

Attribute determining unit is configured for determining pair with top score in the set of the pre-defined object properties As the object properties that attribute is the subject area.

9. equipment according to claim 1, wherein the recognition unit includes：

Identify unit, an object properties pair being configured in the set for pre-defined object properties, based on pair As region includes corresponding to the signature identification of an object properties pair subject area and an object properties centering Which of two object properties object properties are corresponding, and

Attribute determining unit, the object properties being configured for corresponding to the subject area and the pre-defined object The remaining object properties in addition to an object properties pair in the set of attribute determine the object properties of the subject area,

Wherein, if the quantity of remaining object properties is equal to 0, it is right that the object properties corresponding to the subject area are confirmed as this As the object properties in region,

Otherwise, by the set of object properties and the pre-defined object properties corresponding to the subject area except this one Remaining object properties except a object properties pair are grouped into new object properties set again, and for the new object category Property set successively execute the mark operation and attribute determine operation.

10. a kind of object identifying method, including：

Multiple object properties pair are determined for the multiple subclass for including in the set of pre-defined object properties, wherein Each object properties of each object properties centering respectively from different subclass, and in the subclass attribute be face not Same expression；

The feature corresponding to the object properties pair in the diversity extracting object region based on the object properties pair；And

The object properties of feature recognition subject area based on the subject area extracted.