JP2007087089A - Gesture recognition device, gesture recognition program, and gesture recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To identify a gesture of a hand with periodicity, and more specifically, a gesture recognition device and a gesture capable of recognizing a gesture without using a hidden Markov model (HMM) without presenting the start point and end point of the gesture The present invention relates to a recognition program and a gesture recognition method.
The present invention regards the direction of movement of a human hand as an observation signal from an input image, inputs a fixed-length observation signal sequence to an HMM corresponding to the type of gesture, and calculates a likelihood value. The gesture type is recognized from the corresponding HMM for the likelihood value exceeding the threshold value.
[Selection] Figure 1

Description

  The present invention relates to periodic hand gesture recognition, and more specifically, a gesture recognition apparatus, a gesture recognition program, and a gesture recognition method capable of recognizing a gesture by using a hidden Markov model without presenting a start point and an end point of the gesture. About.

  Industrial robots have been used in manufacturing sites as substitutes for dangerous and simple tasks that humans have performed, and have contributed to improvements in safety and production efficiency. These have spread mainly in the automobile and electrical industries, and are one of the driving forces for strengthening the competitiveness of companies.

  Recently, however, the sophistication of robot technology has progressed, and the company has entered the fields of homes, offices, public facilities, etc. without stopping at the manufacturing site. For example, there are nursing robots for bedridden elderly people, pet robots for personal healing and entertainment, and security robots that detect and report the entry of suspicious individuals. These robots are positioned as next-generation robots that share a living space with humans and provide services while communicating with humans.

  Next-generation robots have technologies for control mechanisms (moving mechanisms, gripping mechanisms, etc.), sensing and recognition (detection based on detection and detection of vision, hearing, touch, etc.), and problem-solving capabilities (obstacle avoidance, language interpretation, etc.) It is an intelligent robot and consists of many elemental technologies. Gesture recognition technology for flexible communication with humans using visual information is an important elemental technology. As a gesture recognition method, a view-based method and a trajectory-based method are generally used. However, since a high-speed process is required for mounting on a mobile robot, a trajectory-based method is used. Often used. In addition, the use of a hidden Markov model (hereinafter referred to as HMM (Hidden Markov Model)) has been studied for gesture recognition (for example, Non-Patent Document 1).

There have been proposals for gesture recognition intended to be mounted on toy robots. In this method, a feature of a gesture is acquired from an image obtained by capturing a human gesture by image analysis, and the type of the gesture is recognized by comparing the feature with a plurality of gesture features whose types are known in advance ( For example, Patent Document 1).
Tie Yang and Yangsheng Xu, "Hidden Markov Model for Gesture Recognition", Carnegie Mellon University, Technique Report, CMLT-RI-TR-94 1 0, 1994. JP 2003-334389 A

There are the following problems in the recognition of a gesture using a HMM by detecting the movement locus of a human hand from a time-series image.
(1) It is necessary to present the start point and the end point of the gesture to the recognition system in order to remove the hand movement that becomes noise. For this purpose, for example, a constraint is required in which a gesture starts from an area determined on the screen and ends in an area determined on the screen.
(2) In the conventional gesture recognition, a recognition model corresponding to each gesture is stored in advance, and the movement trajectory of the hand observed by the camera is compared with these recognition models to identify the gesture. Yes. However, there is a problem that the probability of misrecognition is high if a movement trajectory is simply obtained and compared with a recognition model without removing gestures that are not in the recognition model and hand movement noise other than gestures. .
(3) When recognizing continuous gestures, it is necessary to detect the start and end points of each gesture and divide them into gesture units. Since this processing handles a large amount of data, the CPU load is heavy, and there is a problem that it is difficult to mount on a low-cost mobile robot that requires real-time performance.

  The invention of Patent Document 1 mentioned above does not touch on the problem (2).

  The present invention recognizes continuous gestures from the movement trajectory of the hand using the HMM without any restriction, and can be mounted on a robot that requires real-time performance and low cost. A gesture recognition apparatus, a program, and a method thereof are provided.

The gesture recognition device, the gesture recognition program, and the gesture recognition method of the present invention are configured as follows.
(1) 1st invention 1st invention replaces the motion of the hand between frames with an observation signal, inputs the acquired observation signal sequence into a plurality of HMMs corresponding to each type of gesture, calculates a likelihood value, The type of gesture is recognized based on the calculated likelihood value.

  The principle will be described with reference to FIG. As shown in FIG. 1, the gesture recognition apparatus according to the first aspect of the invention comprises an HMM database 10, a hand position detection means 20, an observation signal generation means 30, a likelihood value calculation means 40, and a gesture recognition means 50.

  The HMM database 10 is a database that stores HMMs corresponding to the types of hand gestures. There are a plurality of stored HMMs for each type of hand gesture.

  The hand position detection means 20 detects the position of the hand from the input frame image.

  The observation signal generation means 30 obtains the movement direction of the hand between frames from the hand position obtained by the hand position detection means according to the frame images input in time series, and sequentially creates observation signals corresponding to the movement direction. Then, a fixed-length observation signal sequence composed of a predetermined number of observation signals is generated.

  The likelihood value calculating means 40 inputs the fixed-length observation signal sequence generated by the observation signal generating means 30 to a plurality of HMMs stored in the HMM database. Then, a likelihood value is calculated in each HMM.

  The gesture recognizing unit 50 recognizes the type of gesture corresponding to the HMM that indicates the likelihood value when the likelihood value calculated by the likelihood value calculating unit 40 indicates a value equal to or greater than a predetermined threshold value. As a result. For example, when the likelihood value calculated by the HMM corresponding to the gesture of type A is greater than or equal to the threshold value, the gesture recognition device has recognized “a gesture of the type A”.

Note that the periodic gesture refers to a gesture in which the same gesture is repeated twice or more.
(2) Second invention The second invention is an invention for obtaining the positions of two kinds of hands in the hand position detecting means of the first invention. From the first hand position detecting means and the second hand position detecting means, Constitute.

  The first hand position detecting means creates a skin color likelihood map from the input image based on a skin color model prepared in advance, and sets a plurality of candidate regions of a hand region of a predetermined size at random positions of the skin color likelihood map. To do. A candidate area having an average of skin color likelihood values in the area equal to or greater than a predetermined value is identified as a hand area. The position of the hand is obtained from the hand area. Further, a color histogram of the hand region is created, and this color histogram is set as a reference color histogram. The reference color histogram serves as a reference for identifying the hand region in images input after the initial frame.

The second hand position detection means sets a plurality of candidate areas for a hand area of a predetermined size at random positions based on the tracking results in the previous frame for the image after the initial frame, and obtains a color histogram for each candidate area. create. Then, a candidate area in which the similarity obtained from the color histogram and the reference color histogram is a predetermined value or more is identified as a hand area, and the position of the hand is obtained from this area. There are one or more candidate regions identified in the hand region. Further, since the second hand position detecting means is applied to an image input after the first hand position is detected, the first hand position detecting means first detects the hand position, The subsequent hand position is detected by the second hand position detecting means.
(3) Third invention The third invention is another invention in the hand position detecting means of the second invention, and comprises a first position detecting means, a second hand position detecting means and a reference color histogram updating means. . Among these, the first position detecting means is the same as in the second invention.

  The second hand position detection means creates the color histogram from the hand region after obtaining the hand position as in the second invention. Since the hand region includes candidate regions identified as one or more hand regions, a color histogram is created from these regions.

The color histogram update unit replaces the color histogram obtained by adding a predetermined weight to the color histogram and the reference color histogram created by the second hand position detecting unit and replacing the reference color histogram. That is, the reference color histogram is updated. Therefore, when the hand region is obtained from the sequentially input frame images, the reference color histogram is also updated every time the frame image is input, and the hand region is identified by the updated reference color histogram.
(4) Fourth Invention The fourth invention is the gesture recognition program of the first invention.
(5) Fifth Invention The fifth invention is the gesture recognition method of the first invention.

  According to the first invention, it is possible to provide a gesture recognition device capable of recognizing a hand gesture without presenting a start point and an end point of a periodic hand gesture. .

  According to the second aspect of the present invention, it is possible to provide a gesture recognition device that can easily track a hand region and can recognize a gesture in real time at low cost because the processing load is light.

  According to the third aspect of the invention, the reference color histogram for tracking the hand region is updated for each frame, so that the hand region can be accurately tracked even when the light hitting the skin is changed by the gesture and the shape of the hand is changed. It is possible to provide a gesture recognition device that can perform a high recognition rate.

  According to the fourth and fifth inventions, it is possible to provide a gesture recognition program and a gesture recognition method that do not require presentation of the start and end points.

  An embodiment of the present invention will be described with reference to FIGS.

  FIG. 2 shows a configuration example of the gesture recognition apparatus 100. The CPU 110 controls programs, data, and input / output devices, the CCD camera 121 inputs time-series color images, and the input / output control unit controls the CCD camera 121. 120, a skin color model database 130 storing a skin color model, an HMM database 140 storing an HMM corresponding to a hand gesture type together with a gesture ID, a signal creation dictionary 150 storing an observation signal associated with the moving direction of the hand, and a gesture It consists of a recognition program 160.

  The gesture recognition program 160 includes a hand position detection unit 161, an observation signal generation unit 162, a likelihood value calculation unit 163, and a gesture recognition unit 164.

  The hand position detection unit 161 detects a hand region from the input color image, and performs processing for obtaining the hand position. For the first input image, a skin color likelihood map is created using a skin color model stored in advance in the skin color model database 130, and a hand region of a predetermined size is placed at a random position in the skin color likelihood map. A plurality of candidate areas are set. A candidate area having an average skin color likelihood value in the area equal to or greater than a predetermined value is identified as a hand area, and the hand position is obtained from the weighted average hand area of the skin color likelihood values of pixels in the hand area. Further, a color histogram of the pixels in the hand region is created and stored as a reference color histogram. For frame images that are input after the frame image for which the hand position is first obtained, a hand candidate area of a predetermined size is randomly set in the input image, and the color histogram and reference obtained for each candidate area The degree of similarity with the color histogram is examined, and a process for obtaining the position of the hand using a candidate area having a high degree of similarity as a hand area is performed.

  The observation signal generator 162 obtains the moving direction from the position of the hand between frames obtained by the hand position detector 161 for the color images sequentially input in time series. Then, the obtained moving direction is replaced with an observation signal with reference to the signal creation dictionary 150, and sequentially stored in a fixed-length signal storage area (not shown). The observation signal stored in the signal storage area is a fixed-length observation signal string.

  The likelihood value calculation unit 163 inputs the fixed-length observation signal sequence stored in the signal storage area to the HMM stored in the HMM database 140, and calculates the likelihood value in each HMM.

  The gesture recognition unit 164 compares the likelihood value acquired by the likelihood value calculation unit 163 with a predetermined threshold value, and recognizes the gesture type from the gesture ID of the HMM that indicates the threshold value or more.

  Next, a processing flow for gesture recognition will be described. FIG. 3 shows the overall processing flow. First, a color image is acquired from the CCD camera 121. Based on this image, the first hand position detection routine (details will be described later) determines the position of the first hand (S100, S110).

  Subsequently, the next color image is captured, and the position of the hand with respect to the image of this frame is obtained in a second hand position detection routine (details will be described later). The direction of movement from the position of each hand is calculated, and an observation signal is created by referring to this movement direction with reference to the signal creation dictionary. The created observation signal is stored in a fixed-length signal storage area (for example, storing 12 observation signals) (not shown). At this time, the first observation signal is stored in the signal storage area. Until the signal storage area is filled (12 observation signals are stored), the color image of the next frame is captured and the same processing is repeated. If the observation signal is filled in the signal storage area, the process proceeds to the next step (S120-S160).

  Next, the observation signal string (fixed length observation signal string) in the signal storage area is input to the HMM stored in the HMM database 140. Since a plurality of HMMs are stored, this observation signal string is input to the plurality of HMMs. As the observation signal sequence is input, each HMM calculates and outputs a likelihood value for each model of the gesture type. A check is made to see if any of the output likelihood values indicates a value greater than or equal to a preset threshold value. If so, a gesture in which the type of gesture of the HMM that output the likelihood value is recognized is recognized. Of the kind. Since the HMM and its gesture ID are stored in the HMM database 140, if the HMM is known, the type of gesture can be known from the gesture ID.

  If the likelihood value output from the HMM is less than the threshold value, the next frame image is taken in and the same calculation is performed. At this time, the observation signals stored in the signal storage area are twelve fixed-length observation signal strings in which one oldest observation signal is pushed out and one new observation signal is stored.

  After the gesture is recognized, if the captured frame image is not the last image, the next frame image is captured. If it is the last frame image, the process ends (S170-S200).

  Next, the processing flow of the first hand position detection routine will be described with reference to FIG. First, for the input color image, a skin color likelihood map is created by referring to the skin color model stored in the skin color model database 130. That is, the skin color likelihood map can be created by obtaining the likelihood value of the skin color model with respect to the hue and saturation of each pixel of the image and setting the obtained likelihood value at the position of each pixel (S300). .

  Subsequently, a candidate area that is a candidate for a hand area of a predetermined size is set at a random position on the skin color likelihood map. Then, with respect to the pixels in the candidate area, a ratio (that is, an occupancy ratio) of pixels having a likelihood value equal to or greater than a predetermined threshold to the pixels in the entire candidate area is calculated. If the occupation ratio is equal to or greater than a predetermined value, it is left as a good candidate area, and if it is less than the predetermined value, the candidate area is rejected. This process is performed for all candidate regions (S310, S320).

  After selecting whether the candidate area remains or rejects the candidate area on the skin color likelihood map, it is checked whether or not there is a good candidate area on the skin color likelihood map. If not, the color image of the next frame is captured because the hand area was not in the image. If there is a good candidate region on the skin color likelihood map, the position of the center pixel is obtained by the weighted average of likelihood values. The obtained position is set as the hand position, and this value is stored. The hand position obtained here is the hand position obtained first (S330-S350).

  Next, a color histogram is created from the excellent candidate area, and the created color histogram is stored as a reference color histogram. Since there are one or more excellent candidate regions, a color histogram is created from all of them (S360).

  Next, the processing flow of the second hand position detection routine will be described with reference to FIG. First, a candidate region of a hand region of a predetermined size is set at a random position of the input color image (S500).

  A color histogram is created from one of the set candidate areas, and a likelihood value between the created color histogram and the previously stored reference color histogram is obtained. If the likelihood value is equal to or greater than a predetermined threshold, the candidate area is left as a good candidate area. If it is less than the threshold, the candidate area is rejected. This is performed for all candidate areas for which this has been set (S510, S520).

  For the excellent candidate region, the position of the center pixel is obtained by the weighted average of likelihood values. The obtained position is set as the hand position, and this value is stored. Since there are one or more excellent candidate areas, the position is obtained by weighted average for all of them (S530).

  Subsequently, a color histogram of the excellent candidate region is created, and weights are respectively assigned to the color histogram and the reference color histogram (for example, 0.8 is assigned to the created color histogram, and 0.2 is assigned to the reference color histogram). And add up). The combined color histogram is replaced with the reference color histogram. That is, the reference color histogram is updated (S540).

  According to the description of the above processing flow, the position of the hand for each frame is detected based on the input color image, the movement direction is obtained from the position of the hand between the frames and replaced with an observation signal, and this is input to the HMM. The type of gesture can be obtained from the likelihood value. In this process, the gesture can be recognized without presenting the start and end points of the gesture. In addition, the skin color model was used when obtaining the first hand region, but thereafter, the hand region was obtained from the candidate region set by the color histogram. Thereby, compared with the process which calculates | requires a hand area | region using a skin color model for every flame | frame, a hand area | region can be identified easily and the subject of real-time identification can be solved at low cost. Further, by updating the color histogram with the color histogram of the hand region of the latest frame, the hand region can be recognized with a high recognition rate.

  Next, the skin color model used in the first hand position detection routine will be described. In this embodiment, as shown in FIG. 6, three regions Gauss-1 to Gauss-3 are set as a three-dimensional skin color model with respect to hue and saturation. Each model has a Gaussian distribution. The reason for setting a plurality of skin color models is to accurately identify the skin color region because the skin color varies depending on the difference in skin color between people and how light strikes.

  Next, an example used when creating an observation signal from the moving direction of the hand will be described with reference to FIG.

  FIG. 7A shows the movement of the hand gesture by a solid line, and A and B on the solid line are the positions of the hand obtained from the frame image (assuming that B is obtained after A). The direction of movement of B with respect to A is θ.

  FIG. 7B schematically shows the concept of the signal creation dictionary, and the value of the observation signal is set for each direction every 45 °. For example, this value sets the observation signal to “0” if the moving direction (“θ” in FIG. 7A) is in the range of −22.5 ° to + 22.5 °. The signal creation dictionary 150 stores a table in which the range of θ and the value of the observation signal are associated with each other.

  FIG. 7C shows an example of the observation signal sequence generated based on the hand movement direction obtained from the sequentially input frame images. In this example, a fixed-length observation signal sequence composed of 12 observation signals is shown, but the left observation signal “1” is the newest observation signal and the right observation signal “6” is the oldest observation signal. It is.

  Next, the process of the first hand position detection routine described above will be described with reference to FIG.

  FIG. 8A is a skin color likelihood map created from the input color image using the skin color model shown in FIG. The inside of the curve indicates that the skin color likelihood is high.

  FIG. 8B shows a candidate area set at a random position on the skin color likelihood map. The size of the candidate area is a predetermined size.

  The excellent candidate region in FIG. 8C has an occupancy ratio of pixels having a likelihood value equal to or greater than a predetermined threshold with respect to the pixels constituting one candidate region in each candidate region in FIG. This is a result of calculation, when the occupancy exceeds a predetermined occupancy, leaving it as a good candidate area on the skin color likelihood map, and rejecting the candidate areas that are not. The hand position indicates the position of the pixel obtained by calculating an average with the likelihood as a weight from the good candidate region as the hand position.

  FIG. 8D is a color histogram created as a reference color histogram from the good candidate regions. This color histogram is a three-dimensional histogram in which the other axis is the frequency (that is, the number of pixels) with respect to the two axes plane of saturation and hue.

  Next, the process of the second hand position detection routine described above will be described with reference to FIG.

  FIG. 9A shows an example in which candidate areas are set at random positions on the input color image. The candidate area has a predetermined size in the same way as the candidate area shown in FIG. A color histogram created in one of the candidate areas is shown on the right side of the figure. This color histogram is also a three-dimensional histogram with the frequency as one axis with respect to the biaxial plane of saturation and hue as in the reference color histogram shown in FIG.

  FIG. 9B shows that the similarity is obtained by comparing the color histogram created in FIG. 9A and the reference color histogram created in FIG. And the state which left only the thing more than the value with a similarity as an excellent candidate area | region is shown in the right figure.

  FIG. 9C shows the position of the pixel obtained by calculating the average with the likelihood as the weight from the excellent candidate area as the position of the hand. Further, the color histogram created from this excellent candidate region is added with the reference histogram of FIG. 8 and added together, the color histogram is replaced with the reference color histogram, and the updated reference color histogram is shown in the right figure.

  Next, an example in which the observation signal sequence described in the processing flow of FIG. 3 is input to the HMM to perform gesture recognition will be described with reference to FIG.

  FIG. 10 is a diagram in which twelve boxes at the top are connected to each other, schematically showing a signal storage area. Each box stores one observation signal. When storing an observation signal generated with the input of a frame image, the previously stored observation signal is moved to the right to open the leftmost box and store it there. Therefore, when a new observation signal is stored in a state where 12 observation signals are already stored, the observation signal stored in the rightmost box is lost from the signal storage area. 12 boxes have a fixed length.

  In the lower part of FIG. 10, three HMMs (HMM-A to C) are shown, and an arrow indicates a state in which an observation signal string composed of 12 signals stored in the signal storage area is input to the HMM. HMM-A is an HMM of gesture type A. Similarly, HMM-B and HMM-C are HMMs of gesture type B and gesture type C, respectively. Furthermore, the state where the likelihood value calculated by each HMM is output is also shown.

  Next, FIG. 11 shows a temporal change in the likelihood value output from the HMM. The figure shows a state in which the likelihood values output from the three HMMs change with time, and the subject's hand starts gesture A at time 0, but still contains noise and the like. what likelihood values from the HMM is low for. Although the likelihood value of HMM-A gradually increases with the passage of time, the likelihood values of HMM-B and HMM-C are still low. Further, the likelihood value of HMM-A increases with the passage of time, and is recognized as gesture A when the threshold value is exceeded. Assuming that the subject's hand moves to gesture B following gesture A, the likelihood of HMM-B increases this time and is recognized as gesture B when the threshold value is exceeded. When the HMM-A moves to the gesture B, the likelihood decreases and remains as it is. The human hand moves to gesture C following gesture B, and this state is also shown by the time variation of the likelihood of HMM-C.

In addition to the above examples, the following additional notes are disclosed.
(Appendix 1)
A gesture recognition device for recognizing a periodic gesture,
An HMM database storing a plurality of hidden Markov models corresponding to the types of hand gestures;
Hand position detecting means for determining the position of the hand from the input frame image;
The movement direction of the hand between the frames is obtained based on the position of the hand obtained by the hand position detection means according to the image of the frame input in time series, and a predetermined number of observation signals corresponding to the movement direction are sequentially generated to obtain a predetermined number of signals. An observation signal generating means for generating a fixed-length observation signal sequence composed of observation signals;
Likelihood value calculation means for inputting the fixed-length observation signal sequence to the plurality of hidden Markov models and calculating likelihood values in the respective hidden Markov models;
A gesture recognition unit that recognizes a gesture type corresponding to a hidden Markov model indicating the likelihood value when the calculated likelihood value has a likelihood value indicating a value equal to or greater than a predetermined threshold;
A gesture recognition device characterized by comprising:

(Appendix 2)
The hand position detecting means includes
A skin color likelihood map is created based on a skin color model prepared in advance from an image of an input frame, a plurality of candidate regions for a hand region of a predetermined size are set at random positions in the skin color likelihood map, and the candidate region A first hand position in which a candidate area having an average skin color likelihood value within a predetermined value is a hand area, a hand position is obtained from the hand area, and a color histogram of the hand area is created as a reference color histogram Detection means;
In the image of the frame input after the hand position is obtained by the first hand position detecting means, a plurality of candidate areas for a hand area of a predetermined size are set at random positions in the image, and the color of each candidate area Second hand position detecting means for creating a histogram, identifying a candidate area in which the similarity of the color histogram to the reference color histogram is a predetermined value or more as a hand area, and obtaining a hand position from the hand area; ,
The gesture recognition device according to appendix 1, wherein:

(Appendix 3)
Second hand position detection means of the hand position detection means,
In the image of the frame input after the hand position is obtained by the first hand position detecting means, a plurality of candidate areas for a hand area of a predetermined size are set at random positions in the image, and the color of each candidate area A histogram is created, a candidate area in which the similarity of the color histogram to the reference color histogram is equal to or greater than a predetermined value is identified as a hand area, the position of the hand is determined from the hand area, and the hand area is Create a color histogram,
The hand position detecting means further includes:
A reference color histogram updating means for replacing a color histogram created by adding a predetermined weight to the color histogram created from the hand region and the reference color histogram with the reference color histogram;
The gesture recognition apparatus according to appendix 2, characterized by comprising:

(Appendix 4)
A gesture recognition program for recognizing periodic gestures,
Computer
Hand position detecting means for determining the position of the hand from the input frame image;
The movement direction of the hand between the frames is obtained based on the position of the hand obtained by the hand position detection means according to the image of the frame input in time series, and a predetermined number of observation signals corresponding to the movement direction are sequentially generated to obtain a predetermined number of signals. An observation signal generating means for generating a fixed-length observation signal sequence composed of observation signals;
A likelihood value calculating means for inputting the fixed-length observation signal sequence to a plurality of hidden Markov models corresponding to hand gesture types prepared in advance, and calculating a likelihood value in each of the hidden Markov models;
A gesture recognition unit that recognizes a gesture type corresponding to a hidden Markov model indicating the likelihood value when the calculated likelihood value has a likelihood value indicating a value equal to or greater than a predetermined threshold;
Gesture recognition program for functioning.

(Appendix 5)
A gesture recognition method for recognizing periodic gestures,
A hand position detection procedure for obtaining a hand position from an input frame image;
The movement direction of the hand between the frames is obtained based on the hand position obtained in the hand position detection procedure according to the frame images input in time series, and predetermined observation signals corresponding to the movement direction are sequentially generated to obtain a predetermined number of signals. An observation signal generation procedure for generating a fixed-length observation signal sequence composed of observation signals;
A likelihood value calculation procedure for inputting the fixed-length observation signal sequence to a plurality of hidden Markov models corresponding to hand gesture types prepared in advance, and calculating a likelihood value with each of the hidden Markov models,
When there is a likelihood value indicating a value equal to or greater than a predetermined threshold in the calculated likelihood value, a gesture recognition procedure in which a gesture type corresponding to the hidden Markov model indicating the likelihood value is a recognition result;
A gesture recognition method characterized by comprising:

(Appendix 6)
The gesture recognition apparatus according to appendix 2 or appendix 3, wherein the skin color model is a Gaussian distribution of hue and saturation.

(Appendix 7)
The gesture recognition device according to appendix 2, appendix 3, or appendix 6, wherein the flesh color model is composed of one or more flesh color models.

It is a principle diagram of the invention. It is an example of composition of a gesture recognition device. It is an example of the processing flow of a gesture recognition apparatus. There is an example of a processing flow of a first hand position detection routine. There is an example of a processing flow of a second hand position detection routine. It is an example of a skin color model. It is an example of a production | generation of an observation signal sequence. There is a processing example of a first hand position detection routine. There is a processing example of a second hand position detection routine. It is the observation signal sequence of fixed length inputted, and an HMM example. It is an example of a time change of the likelihood value output from HMM.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 HMM database 20 Hand position detection means 30 Observation signal generation means 40 Likelihood value calculation means 50 Gesture recognition means 100 Gesture recognition apparatus 110 CPU
DESCRIPTION OF SYMBOLS 120 Input / output control part 121 CCD camera 130 Skin color model database 140 HMM database 150 Signal creation dictionary 160 Gesture recognition program 161 Hand position detection part 162 Observation signal generation part 163 Likelihood value calculation part 164 Gesture recognition part

Claims (5)

A gesture recognition device for recognizing a periodic gesture,
An HMM database storing a plurality of hidden Markov models corresponding to the types of hand gestures;
Hand position detecting means for determining the position of the hand from the input frame image;
The movement direction of the hand between the frames is obtained based on the position of the hand obtained by the hand position detection means according to the image of the frame input in time series, and a predetermined number of observation signals corresponding to the movement direction are sequentially generated to obtain a predetermined number of signals. An observation signal generating means for generating a fixed-length observation signal sequence composed of observation signals;
Likelihood value calculation means for inputting the fixed-length observation signal sequence to the plurality of hidden Markov models and calculating likelihood values in the respective hidden Markov models;
A gesture recognition unit that recognizes a gesture type corresponding to a hidden Markov model indicating the likelihood value when the calculated likelihood value has a likelihood value indicating a value equal to or greater than a predetermined threshold;
A gesture recognition device characterized by comprising: The hand position detecting means includes
A skin color likelihood map is created based on a skin color model prepared in advance from an image of an input frame, a plurality of candidate regions for a hand region of a predetermined size are set at random positions in the skin color likelihood map, and the candidate region A first hand position in which a candidate area having an average skin color likelihood value within a predetermined value is a hand area, a hand position is obtained from the hand area, and a color histogram of the hand area is created as a reference color histogram Detection means;
In the image of the frame that is input after the hand position is determined by the first hand position detecting means, a plurality of candidate areas for a hand area of a predetermined size are set at random positions in the image, and the color of each candidate area Second hand position detecting means for creating a histogram, identifying a candidate area where the similarity of the color histogram to the reference color histogram is a predetermined value or more as a hand area, and obtaining a hand position from the hand area; ,
The gesture recognition apparatus according to claim 1, comprising: Second hand position detection means of the hand position detection means,
In the image of the frame input after the hand position is obtained by the first hand position detecting means, a plurality of candidate areas for a hand area of a predetermined size are set at random positions in the image, and the color of each candidate area A histogram is created, a candidate area in which the similarity of the color histogram to the reference color histogram is equal to or greater than a predetermined value is identified as a hand area, the position of the hand is determined from the hand area, and the hand area is Create a color histogram,
The hand position detecting means further includes:
A reference color histogram updating means for replacing a color histogram created by adding a predetermined weight to the color histogram created from the hand region and the reference color histogram with the reference color histogram;
The gesture recognition device according to claim 2, comprising: A gesture recognition program for recognizing periodic gestures,
Computer
Hand position detecting means for determining the position of the hand from the input frame image;
The movement direction of the hand between the frames is obtained based on the position of the hand obtained by the hand position detection means according to the image of the frame input in time series, and a predetermined number of observation signals corresponding to the movement direction are sequentially generated to obtain a predetermined number of signals. An observation signal generating means for generating a fixed-length observation signal sequence composed of observation signals;
A likelihood value calculating means for inputting the fixed-length observation signal sequence to a plurality of hidden Markov models corresponding to hand gesture types prepared in advance, and calculating a likelihood value in each of the hidden Markov models;
A gesture recognition unit that recognizes a gesture type corresponding to a hidden Markov model indicating the likelihood value when the calculated likelihood value has a likelihood value indicating a value equal to or greater than a predetermined threshold;
Gesture recognition program for functioning. A gesture recognition method for recognizing periodic gestures,
A hand position detection procedure for obtaining a hand position from an input frame image;
The movement direction of the hand between the frames is obtained based on the hand position obtained in the hand position detection procedure according to the frame images input in time series, and predetermined observation signals corresponding to the movement direction are sequentially generated to obtain a predetermined number of signals. An observation signal generation procedure for generating a fixed-length observation signal sequence composed of observation signals;
A likelihood value calculation procedure for inputting the fixed-length observation signal sequence to a plurality of hidden Markov models corresponding to hand gesture types prepared in advance, and calculating a likelihood value with each of the hidden Markov models,
When there is a likelihood value indicating a value equal to or greater than a predetermined threshold in the calculated likelihood value, a gesture recognition procedure in which a gesture type corresponding to the hidden Markov model indicating the likelihood value is a recognition result;
A gesture recognition method characterized by comprising:

Images