JP2001307107A - Image processor, its method and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly accurately recognize a body gesture or a hand gesture. SOLUTION: An intra-area movement distribution calculation part 71 calculates the standard deviation of movement distribution in an area having the smallest evaluation value and the number of pixels having up or down movement respectively. An area gravity center movement checking part 72 calculates a distance between the center of gravity of an area having the smallest evaluation value of a current frame and the center of gravity of an area selected one frame before and checks the locus of the center of gravity from the calculated result. A grasping operation recognition state machine 73 recognizes grasping operation on the basis of the standard deviation of distribution of movement in the area, the number of pixels of the up or down direction and the locus of the center of gravity respectively calculated by the calculation part 71 and the checking part 72.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method, and a recording medium, and more particularly to an image processing apparatus and method capable of recognizing a user's gesture and hand gesture by image processing, and recording. Regarding the medium.

[0002]

2. Description of the Related Art Heretofore, there have been proposed some techniques for interfacing a person with a computer by incorporating a gesture or a hand gesture of a user as computer graphics.

For example, Japanese Patent Application Laid-Open No. H11-195140 discloses a technique for editing three-dimensional computer graphics data by operating a computer with gestures or hand gestures using a data glove.

[0004] For example, Japanese Patent Application Laid-Open No. 9-311759 discloses that a user makes a gesture by using a penlight that is turned on and blinks in a predetermined color, and the penlight is imaged and image-recognized by a camera. A technology for interfacing with a person is disclosed.

Further, for example, Japanese Patent Application Laid-Open No. 9-1020
No. 46 discloses a technique for estimating and recognizing a hand shape from a silhouette image captured by a camera, thereby performing an interface between a computer and a person.

Further, for example, Japanese Patent Application Laid-Open No. H08-2123
No. 27 discloses a technique for performing an interface between a computer and a person by recognizing a gesture using an orientation one-dimensional histogram of a hand image captured by a camera and a two-dimensional spatio-temporal orientation histogram. I have.

Further, for example, Japanese Patent Application Laid-Open No. 9-1799
No. 88 discloses a technique for recognizing a three-dimensional posture and gesture of a hand using two cameras to perform an interface between a computer and a person.

[0008]

In the case of the method using a data glove (Japanese Patent Laid-Open No. 11-195140), although the shape and movement of the hand can be recognized, the attachment and detachment of the data glove is inconvenient. In addition, there is a problem that a cable connecting the data glove and the computer becomes cumbersome.

In view of the above, a technique using a penlight has been proposed in consideration of the usability of the user (Japanese Patent Application Laid-Open No. 9-311759). However, the user must have the penlight, which makes the user troublesome. There is a problem that lacks versatility because it still remains and uses a special device (penlight).

[0010] Further, in order to reduce the inconvenience to the user, a technique for recognizing the shape of the hand picked up by a camera has been proposed (Japanese Patent Application Laid-Open Nos. 9-102046 and 8-212327). However, the shape of the hand may not be detected accurately due to changes in lighting, etc.
There was a problem of misrecognition.

In order to prevent erroneous recognition, a method of recognizing an image using a plurality of cameras has been proposed (Japanese Patent Laid-Open No. 9-179988), but there is a problem that it cannot be realized at low cost. Was.

SUMMARY OF THE INVENTION The present invention has been made in view of such a situation, and it is an object of the present invention to be able to recognize a user's gesture and hand gesture at low cost and with high accuracy by image processing.

[0013]

SUMMARY OF THE INVENTION An image processing apparatus according to the present invention is provided with an image pickup means for picking up an object to be gestured or shaken, an identification means for identifying an image picked up by the image pickup means, and an identification means for identifying the image. Dividing the divided image into predetermined regions, a moving direction detecting unit detecting a moving direction in the region divided by the region dividing unit, and a center of gravity in the region based on a detection result of the moving direction detecting unit. A center of gravity calculating means for calculating the evaluation value, an evaluation value calculating means for calculating an evaluation value in the area, an area selecting means for selecting an area having the smallest evaluation value from the calculation result of the evaluation value calculating means, and a selection by the area selecting means In the selected area, a gripping motion recognizing means for recognizing a gripping motion of the object, and a vibration for recognizing a swinging motion of the object based on a locus of the center of gravity in the area selected by the area selecting means. Characterized in that it comprises a motion recognizing means.

The image processing apparatus according to the present invention comprises a circumscribing frame detecting means for detecting a circumscribing frame of the area divided by the area dividing means, and an aspect ratio calculating for calculating an aspect ratio of the circumscribing frame detected by the circumscribing frame detecting means. Means, an area calculating means for calculating the area of the area by counting the number of pixels in the circumscribing frame detected by the circumscribing frame detecting means, and a pixel in the motion direction based on the detection result of the motion direction detecting means. A counting means for counting can be further provided, and the evaluation value calculating means calculates the evaluation value in the area based on the center of gravity, the aspect ratio, the area, and the number of pixels in the movement direction. be able to.

The image processing apparatus according to the present invention may further include a standard deviation calculating means for calculating a standard deviation of a pixel having a predetermined moving direction in the area selected by the area selecting means, The gripping motion recognizing means recognizes the gripping motion of the target object based on the calculation result of the standard deviation calculating means, the movement direction of the center pixel in the area selected by the area selecting means, and the calculation result of the center of gravity calculating means. You can do so.

The identifying means identifies an image whose pixel value is within a predetermined range, or determines whether the image pixel value is within a predetermined range and the image brightness value is within a predetermined range. Something can be identified.

The motion direction detecting means can detect the direction of motion in the area based on the luminance value of the central pixel in the area.

According to the image processing method of the present invention, an image pickup step of picking up an object to be gestured or hand-held, an identification step for identifying an image captured by the image pickup step, and an image identified by the identification step processing Is divided into predetermined regions, a motion direction detecting step for detecting a motion direction in the region divided by the process of the region dividing step, and an area in the region based on a detection result in the motion direction detecting step. A center of gravity calculating step of calculating a center of gravity, an evaluation value calculating step of calculating an evaluation value in the area, an area selecting step of selecting an area having the smallest evaluation value from the calculation result in the evaluation value calculating step, and an area selecting step A grip operation recognition step of recognizing a grip operation of the object in the area selected by the processing of Based on the centroid trajectory of the selected by management area, characterized in that it comprises a swinging motion recognition step recognizes the swinging motion of the object.

[0019] The program recorded on the recording medium of the present invention includes an imaging step of imaging an object to be gestured or gestured, an identification step of identifying an image captured by the processing of the imaging step, and a processing of the identification step. Dividing the image identified by the above into predetermined regions, a motion direction detecting step of detecting a motion direction in the region divided by the process of the region dividing step, and a motion direction detecting step. A center of gravity calculating step of calculating a center of gravity in the area; an evaluation value calculating step of calculating an evaluation value in the area; and an area selecting step of selecting an area having the smallest evaluation value from the calculation results in the evaluation value calculating step. And a gripping motion recognition step of recognizing a gripping motion of an object in the area selected by the processing of the area selecting step. , Based on the trajectory of the center of gravity of the area selected by the process of area selection step, characterized in that it comprises a swinging motion recognition step recognizes the swinging motion of the object.

In the image processing apparatus, the image processing method, and the program recorded on the recording medium according to the present invention,
An object to be gestured or gestured is imaged, the imaged image is identified, the identified image is divided into predetermined regions, the movement direction in the divided region is detected, and the center of gravity in the region is calculated. The evaluation value in the area is calculated, and the area with the lowest evaluation value is selected from the calculation result, and the gripping operation of the object is recognized in the selected area,
The swing motion of the object is recognized based on the locus of the center of gravity in the selected area.

[0021]

FIG. 1 is a block diagram showing an embodiment of an image processing system according to the present invention. The video camera 2 captures a gesture or hand gesture of the user 1, converts the captured image data into color data such as YUV, YCbCr, and RGB, and supplies the color data to the image processing device 3. The image processing device 3 performs predetermined processing such as image processing and gesture recognition processing on the image data supplied from the video camera 2, and outputs the processed data to the display / output device 4. The display / output device 4 starts an application on the screen or moves a window displayed on the screen based on image data corresponding to the gesture or hand gesture of the user 1 supplied from the image processing device 3. I do.

FIG. 2 is a block diagram showing the configuration of the image processing apparatus 3. The image processing unit 11 performs a skin color region extraction / division process, a center of gravity / motion detection process, and the like on the image data supplied from the video camera 2, and supplies the image processing result to the gesture recognition unit 12. . The gesture recognition unit 12 performs a gesture recognition process based on the image processing result supplied from the image processing unit 11, and supplies the recognition result to the display / output device 4.

FIG. 3 is a block diagram showing a detailed configuration of the image processing unit 11. The area dividing unit 21 includes the image storage unit 2
2 is read out from the image data of the previous frame stored in the image data of the current frame supplied from the video camera 2 and the image data of the current frame supplied from the video camera 2. ) Is extracted and divided, and labeling is performed for each extracted region. The image storage unit 22 stores the image data supplied from the video camera 2 for each frame.

The area processing section 23 detects the center of gravity and the direction of movement for each area labeled by the area dividing section 21, supplies the detection result to the gesture recognition section 12, and stores the result in the area storage section 24. . The region storage unit 24 stores the detection result of the center of gravity and the direction of movement for each region supplied from the region processing unit 23.

FIG. 4 is a block diagram showing a detailed configuration of the gesture recognition unit 12. As shown in FIG. The gripping motion recognition unit 31 recognizes the gripping / opening motion of the hand based on the detection result (center of gravity / direction of movement) supplied from the area processing unit 23 of the image processing unit 11, and displays and outputs the recognition result. 4 The gripping motion recognition unit 31 operates a gripping motion recognition state machine 73 (FIG. 8), which will be described later, to distribute the movement direction in the region and the position information of the center of gravity of several frames stored in the region storage unit 24. Recognize hand grip / open based on The center-of-gravity trajectory recognizing unit 32 recognizes an operation of waving up, down, left, and right based on the detection result (center of gravity / direction of movement) supplied from the area processing unit 23, and supplies the recognition result to the display / output device 4. .

FIG. 5 is a block diagram showing an area dividing section 21 of the image processing section 11.
FIG. 3 is a block diagram showing a detailed configuration of FIG. Color identification unit 41
Is based on the color data of the image data of the current frame supplied from the video camera 2, a pixel within a preset color (for example, flesh color) range is set to 1 and a preset color range Pixels that are not inside are binarized as 0 and supplied to the labeling unit 42. Labeling section 42
Performs a labeling process based on the binarized data supplied from the color identification unit 41, and supplies a processing result to the area processing unit 23 of the image processing unit 11.

FIG. 6 is a block diagram showing a detailed configuration of the area dividing section 21 of another example. In FIG. 6, portions corresponding to those in FIG. 5 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

The inter-frame difference calculation section 51 reads out the image data of the previous frame stored in the image storage section 22 and stores the image data of the previous frame and the video camera 2.
Calculate the luminance difference of each pixel of the image data of the current frame supplied from the pixel, and set a pixel having a luminance difference equal to or greater than a predetermined threshold to 1 and a pixel having a luminance difference less than the predetermined threshold to 0 to binarize the pixel, It is supplied to the color / motion evaluation section 52. The color / motion evaluation unit 52 determines a color within a set predetermined range based on the binary data supplied from the color identification unit 41 and the binary data supplied from the inter-frame difference calculation unit 51. A pixel having a skin color) and a luminance difference equal to or greater than a predetermined threshold value is extracted and supplied to the labeling unit 42. The labeling unit 42 performs a labeling process based on the extracted pixels supplied from the color motion evaluation unit 52, and supplies the labeling process to the area processing unit 23 of the image processing unit 11.

FIG. 7 shows the area processing unit 23 of the image processing unit 11.
FIG. 3 is a block diagram showing a detailed configuration of FIG. Bounding frame detector 6
1 detects a rectangle (circumscribed frame) surrounding each region from the outside based on the labeled image data supplied from the region dividing unit 21 of the image processing unit 11, and
2. Aspect ratio calculation unit 63 and area calculation unit 64
Respectively. The motion detector 62 compares the circumscribed frame one frame before supplied from the circumscribed frame detector 61 with the circumscribed frame of the current frame, detects the direction of motion in the circumscribed frame of the current frame, and uses the detection result as the center of gravity. It is supplied to the calculation unit 65 and the pixel number counting unit 66, respectively.

The aspect ratio calculation section 63 calculates the aspect ratio (aspect ratio) of the circumscribed frame supplied from the circumscribed frame detection section 61, and supplies the calculation result to the area evaluation section 67. The area calculation unit 64 calculates the area by counting the number of pixels included in the circumscribed frame supplied from the circumscribed frame detection unit 61, and supplies the calculation result to the area evaluation unit 67.

The center-of-gravity calculating section 65 calculates the center of gravity in the circumscribed frame based on the motion direction detection result supplied from the motion detecting section 62. The pixel counting section 66 includes a motion detection section 62.
The number of motion pixels is counted based on the supplied motion direction detection result. The area evaluation unit 67 calculates an evaluation value based on the aspect ratio, area, center of gravity, and number of motion pixels of the circumscribed frame supplied from the aspect ratio calculation unit 63 to the pixel number counting unit 66, and calculates the evaluation value. Select a small area.

FIG. 8 is a block diagram showing a detailed configuration of the gripping motion recognition unit 31 of the gesture recognition unit 12. As shown in FIG. The intra-region motion distribution calculation unit 71 calculates the standard deviation of the distribution of motion in the region having the smallest evaluation value supplied from the region evaluation unit 67,
Then, the number of pixels having upward or downward movement in a small block having upward and downward movements as central pixels is calculated. Area center of gravity shift check unit 7
2 calculates the distance between the center of gravity of the area of the current frame having the smallest evaluation value supplied from the area evaluation unit 67 and the center of gravity of the area selected one frame before, and moves the center of gravity (trajectory) from the calculation result. Check).

The grasping motion recognition state machine 73 includes an intra-region motion distribution calculating unit 71 and a region center-of-gravity movement checking unit 7.
The gripping motion is recognized based on the standard deviation of the motion distribution in the area, the number of pixels in the upward or downward direction, and the locus of the center of gravity calculated in step 2.

Next, referring to the flowchart of FIG.
A hand grip / shake motion recognition process executed by the image processing apparatus 3 will be described.

In step S1, the video camera 2
Captures a gesture or hand gesture of the user 1, converts the captured image data into, for example, YUV color data, and stores the data in the image storage unit 22. In step S2, the area dividing unit 21 extracts and divides a preset color area (skin color).

Here, the area dividing process will be described with reference to FIG. In step S21, the color identification unit 41 of the area division unit 21 identifies a color from the YUV color data of the image data captured in step S1. That is, the color identification unit 41 determines that Y _min <Y <Y _max , U _min <U <
U _max, and 1 pixel is within the range of V _min <V <V _max, the other pixels to 0. Note that the variables Y _min , Y
_max , U _min , U _max , V _min , and V _max are arbitrarily set in advance.

In step S21, the color discriminating section 41 sets one pixel having color data within the above-described range to one.
And the other pixels are set to 0, and corresponding to whether the distance from a predetermined color distribution is equal to or less than a predetermined threshold,
It is also possible to identify the colors. For example, when the skin color image data is distributed in the YUV space as shown in FIG. 11, first, the color distribution is approximated by a straight line L _C. Next, the color discriminating unit 41 compares the point (Y, U, V) in the YUV space with the straight line L _C
Is calculated according to the following formulas (1) to (4), and the pixel whose calculated distance d is equal to or smaller than a predetermined threshold is set to 1, and the pixel which is not equal to or smaller than the predetermined threshold is set to 0.

(Equation 1) _{t = y V Y + u V} U + v V V ··· (2) (y C, u C, v C) = t (Y, U, V) ··· (3)

(Equation 2)

Returning to FIG. 10, in step S22, the color identification section 41 extracts a pixel having one data (skin color pixel) from the pixels identified in step S21. In step S23, the labeling unit 42 of the area dividing unit 21 performs labeling processing on the pixels in the skin color area extracted in the processing in step S22, and returns to step S3 in FIG.

Further, another example of the area dividing process will be described with reference to the flowchart of FIG. Step S
In 31, the color identification unit 41 identifies a color from the YUV color data of the image data captured in step S <b> 1 by the above-described processing. In step S32, the inter-frame difference calculation unit 51 calculates the difference (Y) between the luminance value (Y) of the image data of the current frame captured in the process of step S1 and the image data of the previous frame stored in the image storage unit 22. Hereinafter, this is referred to as a luminance difference), and 1 is substituted for a pixel whose luminance difference is equal to or more than a predetermined threshold, and 0 is substituted for a pixel whose luminance difference is not equal to or more than the predetermined threshold.

In step S33, the color / motion estimating section 52 selects the pixel position (x _m ) of one data pixel calculated in step S32 from among the one data pixel identified in step S31. , y _m ), that is, the coordinates (x, y) are x _m −Δx <x
<X _m + Δx, y _m −Δy <y <y _m + Δy The pixel within the range of 1 is set to 1, and the other pixels are set to 0. The shift amounts Δx and Δy are arbitrarily set in advance. In step S34, the labeling unit 42 determines in step S3
Labeling processing is performed on the pixels in the flesh color region extracted in the processing of step 3, and the process returns to step S3 in FIG.

In step S3 in FIG. 9, the circumscribed frame detection unit 61 of the area processing unit 23 detects a circumscribed frame (rectangle) surrounding each area divided in step S2 from the outside. In step S4, the motion detection unit 62 detects a motion direction in the circumscribed frame of the current frame detected in the process of step S3.

Here, the motion direction detecting process will be described with reference to the flowchart of FIG. FIG.
As shown in the figure, the description will be made on the assumption that the area of the current frame composed of 3 × 3 pixels is compared with the area one frame before to detect the motion direction of the area of the current frame.

In step S41, the motion direction detector 62 substitutes 0 for variables h and v indicating the direction of motion.
In step S42, the movement direction detection unit 62 determines whether movement in both the vertical and horizontal directions has been detected,
If it is determined that motion in both the vertical and horizontal directions has not been detected yet, the process proceeds to step S43, where the center pixel P (i, j) of the current frame and the center pixel P 'of the previous frame are determined.
It is determined whether or not the luminance difference of (i, j) is larger than a predetermined positive threshold. Note that i represents the y coordinate, and j represents the x coordinate.

In step S43, P (i, j)-
If it is determined that P ′ (i, j) is larger than the predetermined positive threshold, the process proceeds to step S44, and the movement direction detection unit 62
Is to determine whether or not to detect horizontal motion, that is, to determine whether or not vertical motion has been detected, and to determine to detect horizontal motion (when vertical motion has been detected. If it is determined that there is, the process proceeds to step S45. In step S45, the movement direction detection unit 62 determines whether or not the luminance difference between the center pixel P (i, j) and the pixel P (i, j + 1) is larger than a predetermined threshold, and determines whether P (i, j) − P
When it is determined that (i, j + 1) is larger than the predetermined threshold, the process proceeds to step S46, where 1 is substituted for the variable h, and the process returns to step S42.

In step S45, P (i, j)-
If it is determined that P (i, j + 1) is not larger than the predetermined threshold, the process proceeds to step S47, where the motion direction detector 6
2 further includes a central pixel P (i, j) and a pixel P (i, j).
It is determined whether or not the luminance difference of -1) is larger than a predetermined threshold. If it is determined that P (i, j) -P (i, j-1) is larger than the predetermined threshold, the process proceeds to step S48. Variable h
Is substituted for -1, and the process returns to step S42. Step S4
7, if it is determined that P (i, j) -P (i, j-1) is not larger than the predetermined threshold, step S4
Return to 2.

If it is determined in step S44 that a vertical motion is detected instead of a horizontal motion, the process proceeds to step S49, where the motion direction detector 62 determines whether the center pixel P (i, j) It is determined whether or not the luminance difference of P (i + 1, j) is larger than a predetermined threshold, and P (i, j)
If it is determined that −P (i + 1, j) is larger than the predetermined threshold, the process proceeds to step S50, where 1 is substituted for a variable v, and the process returns to step S42.

In step S49, P (i, j)-
When it is determined that P (i + 1, j) is not larger than the predetermined threshold, the process proceeds to step S51, and the movement direction detecting unit 6
2 further includes a central pixel P (i, j) and a pixel P (i−
It is determined whether or not the luminance difference of (1, j) is larger than a predetermined threshold. If it is determined that P (i, j) -P (i-1, j) is larger than the predetermined threshold, the process proceeds to step S52. , -1 is substituted for the variable v, and the process returns to step S42. Step S
If it is determined in step S51 that P (i, j) -P (i-1, j) is not larger than the predetermined threshold, step S
Return to 42.

In step S43, P (i,
If it is determined that j) −P ′ (i, j) is not larger than the predetermined positive threshold, the process proceeds to step S53, and the motion direction detection unit 62 further determines the center pixel P (i,
j) and whether or not the luminance difference between the center pixel P ′ (i, j) one frame before is smaller than a predetermined negative threshold.

In step S53, P (i, j)-
If it is determined that P ′ (i, j) is smaller than the predetermined negative threshold, the process proceeds to step S54, and the movement direction detection unit 62
Is to determine whether or not to detect horizontal motion, that is, to determine whether or not vertical motion has been detected, and to determine to detect horizontal motion (when vertical motion has been detected. If it is determined that there is, the process proceeds to step S55. In step S55, the movement direction detection unit 62 determines whether the pixel P
It is determined whether or not the luminance difference between (i, j + 1) and the central pixel P (i, j) is larger than a predetermined threshold, and P (i, j + 1)
If it is determined that −P (i, j) is larger than the predetermined threshold, the process proceeds to step S56, where 1 is substituted for a variable h, and the process returns to step S42.

In step S55, P (i, j +
1) If it is determined that -P (i, j) is not larger than the predetermined threshold, the process proceeds to step S57, and the motion direction detection unit 62 further determines the center pixel P (i, j-1) and the pixel P
It is determined whether or not the luminance difference of (i, j) is larger than a predetermined threshold. If it is determined that P (i, j-1) -P (i, j) is larger than the predetermined threshold, the process proceeds to step S58. Then, -1 is substituted for the variable h, and the process returns to step S42. If it is determined in step S57 that P (i, j-1) -P (i, j) is not larger than the predetermined threshold, the process returns to step S42.

If it is determined in step S54 that a vertical motion is detected instead of a horizontal motion, the process proceeds to step S59, where the motion direction detection unit 62 determines whether the pixel P (i + 1, j) and the central pixel It is determined whether the luminance difference of P (i, j) is larger than a predetermined threshold, and P (i + 1, j) is determined.
j) If it is determined that -P (i, j) is larger than the predetermined threshold, the process proceeds to step S60, where 1 is substituted for a variable v, and the process returns to step S42.

In step S59, P (i + 1,
If it is determined that j) -P (i, j) is not larger than the predetermined threshold, the process proceeds to step S61, where the motion direction detection unit 62 further determines the pixel P (i-1, j) and the central pixel P
It is determined whether or not the luminance difference of (i, j) is greater than a predetermined threshold. If it is determined that P (i-1, j) -P (i, j) is greater than the predetermined threshold, the process proceeds to step S62. Then, -1 is substituted for the variable v, and the process returns to step S42. If it is determined in step S61 that P (i-1, j) -P (i, j) is not larger than the predetermined threshold, the process returns to step S42.

When it is determined in step S42 that both vertical and horizontal motions have been detected, or in step S53, P (i, j) -P '
When it is determined that (i, j) is not smaller than the predetermined negative threshold, the process returns to step S4 in FIG.

In step S4 in FIG. 9, the movement direction detecting section 62 determines the variables h and v calculated by the above-described processing.
Is associated with the corresponding diagram shown in FIG. 15, and a pixel value (index value) indicating the motion direction is set.

In the example of FIG. 15, LU (Left Upper) is
U (Upper) indicates that the pixel has moved upward, RU (Right Upper) indicates that the pixel has moved upward, and L (Left) indicates that the pixel has moved left. Indicates that the pixel has moved, N (No move) indicates that the pixel has not moved, R (Right) indicates that the pixel has moved to the right, and LD (Left Down) indicates that the pixel has moved to the lower left. D (Down) indicates that the pixel has moved down, and RD (Right Down) indicates that the pixel has moved down and to the right.

In FIG. 15, for example, N = 0, LU =
1, U = 2, RU = 3, L = 4, R = 5, LD = 6, D = 7, RD
= 8. Both the calculated variables h and v are-
When the pixel value is 1, the pixel value indicating the movement direction is set to 1. When the calculated variable h is 0 and the variable v is -1,
The pixel value indicating the movement direction is set to 2. When the calculated variable h is 1 and the variable v is -1, the pixel value indicating the motion direction is set to 3. When the calculated variable h is −1 and the variable v is 0, the pixel value indicating the motion direction is set to 4. If both the calculated variables h and v are 0,
The pixel value indicating the movement direction is set to 0. When the calculated variable h is 1 and the variable v is 0, the pixel value indicating the motion direction is set to 5. The calculated variable h is -1 and the variable v
Is 1, the pixel value indicating the motion direction is set to 6. When the calculated variable h is 0 and the variable v is 1, the pixel value indicating the motion direction is set to 7. Then, when both the calculated variables h and v are 1, the pixel value indicating the motion direction is set to 8.

That is, when the pixel value is 1, the pixel has moved to the upper left, and when the pixel value is 2, the pixel has moved upward. When the pixel value is 3, the pixel has moved to the upper right. When the pixel value is 4, the pixel has moved to the left, and when the pixel value is 0, the pixel has not moved, and the pixel value is 5. If
If the pixel has moved to the right and the pixel value is 6, then
If the pixel value is 7, it means that the pixel has moved down, and if the pixel value is 8, it means that the pixel has moved down right.

The present invention is not limited to this, and it is also possible to detect the direction of movement using a technique such as block matching.

Returning to FIG. 9, in step S5, the aspect ratio calculating section 63 calculates the aspect ratio (aspect ratio) of the circumscribed frame detected in the processing in step S3. In step S6, the area calculation unit 64 calculates the area by counting the number of pixels included in the circumscribed frame detected in the processing in step S3. In step S7,
The pixel number counting unit 66 counts the number of motion pixels based on the motion direction detection result calculated in the process of step S4. In step S8, the center-of-gravity calculation unit 65 calculates the center of gravity in the circumscribed frame based on the motion direction detection result calculated in the process of step S4.

Here, a method of calculating the center of gravity will be described with reference to FIG. In the case of the example of FIG. 16A, the average position is calculated using the pixels in the circumscribed frame, and the calculated average position is set as the center of gravity C.

In the case of the example shown in FIG.
The weighted average is calculated, and is set as the center of gravity C. Sand
The coordinates (x_ul, y_ul), Lower right coordinates
(X_br, y _br), The weight corresponding to a certain pixel (x, y)
Is calculated according to the following equation (5), and the calculation result is
(6) to calculate a weighted weighted average. In addition,
This example shows that when the hand is
Takes advantage of the high probability that
Is weighted. (Y_br−y) / (Y_br-Y_ul) ・ ・ ・ (5)

[0062]

(Equation 3)

In the case of the example shown in FIG. 16C, the first time
As shown in the example of FIG. 16A, the average position C1 is calculated using the pixels in the circumscribed frame. In the second time, the average position C2 is calculated using pixels located above the screen from the average position C1. Similarly, the third and subsequent average positions are sequentially and repeatedly calculated. Then, the n-th average position Cn is set as the final center of gravity C.

Returning to FIG. 9, in step S8, the center-of-gravity calculator 65 selects the center of gravity in the direction in which the number of motion pixels counted in the processing in step S7 is the largest as the center of gravity in this area. In step S9, the area evaluation unit 67
Is calculated from the aspect ratio, area, center of gravity, and number of motion pixels of the circumscribed frame calculated in the processing of steps S5 to S8.
The evaluation value is calculated according to the following equation (7). Φ (i) = α _c | C _i −C _p | + α _a | A _i −A _p | + α _s | S _i −S _p | −α _m Nm _i (7)

[0065] In the above formula (7), C _i is the center of gravity of the area i, A _i is the aspect ratio of the area i, S _i is the area of region i, Nm _i represents the number of motion pixels in the region i, C _p ,
_Ap and _Sp are the barycentric positions of the region p selected in the previous frame,
Represents the aspect ratio and area, respectively, α _c , α _a ,
α _s and α _m are weighting coefficients indicating the importance of the position of the center of gravity, the aspect ratio, the area, and the number of motion pixels.

In step S10, the area evaluation section 67
Selects an area having the smallest evaluation value among the evaluation values calculated in the processing of step S9. In step S11, the grip operation confirmation unit 31 of the gesture recognition unit 12
In the region selected in the process of step S10, the hand grip / open operation is recognized.

Here, the gripping motion recognition processing will be described with reference to the flowchart of FIG. Step S7
In step 1, the intra-region motion distribution calculation unit 71 determines in step S
The standard deviation of the distribution of the pixels having the upward movement in the bw × bh small block (FIG. 18) centering on the pixels having the upward movement in the region selected in the processing of step 10 is It is calculated according to equation (8). Note that N represents the total number of pixels, and x bar and y bar are average coordinates of pixels having an upward movement.

[0068]

(Equation 4)

In step S72, the area motion distribution calculation section 71 counts the number of pixels having an upward motion in the small block. In step S73, the area motion distribution calculation unit 71 calculates the standard deviation of the distribution of the pixels having the downward movement in the bw × bh small block (FIG. 18) centering on the pixels having the downward movement. It is calculated according to equation (8). In step S74, the area motion distribution calculation unit 71 counts the number of pixels having a downward motion in the small block.

In step S75, the area motion distribution calculation unit 71 selects a small block having the largest number of pixels from the number of pixels counted in the processing in steps S72 and S74. In step S76, the area distribution calculation unit 71 determines whether the standard deviation of the small block selected in the processing in step S75 is within a predetermined threshold,
If it is determined that the standard deviation of the small block is within the predetermined threshold, the process proceeds to step S77, where the area distribution calculation unit 71
Determines whether or not the center pixel of the selected small block has an upward movement.

If it is determined in step S77 that the central pixel of the selected small block has an upward movement, the process proceeds to step S78, in which a Release flag is set for the small block. If it is determined in step S77 that the central pixel of the selected small block does not have an upward movement, that is, has a downward movement, the process proceeds to step S79, and the small block is assigned Grasp data. Set the flag.

If it is determined in step S76 that the standard deviation of the small block is not smaller than the predetermined threshold, the process skips steps S77 to S79 and proceeds to step S80.

After the processing in step S76, S78 or S79, in step S80, the area center-of-gravity shift check unit 72 determines the center of gravity of the image area of the current frame selected in the processing of step S10 and the image area of the previous frame. Calculate the distance between the centers of gravity. In step S81,
The region center-of-gravity shift checking unit 72 calculates whether or not the distance between the center of gravity of the image region of the current frame calculated in the process of step S80 and the center of gravity of the image region one frame before is within a predetermined threshold value, If it is determined that the difference is within the predetermined threshold, the process proceeds to step S82, and the Stable flag is set in the image area.

If it is determined in step S81 that the distance between the center of gravity of the image area of the current frame and the center of gravity of the image area one frame before is not within a predetermined threshold value, the processing of step S82 is skipped and step S83 is performed. Proceed to.

After the processing in step S81 or S82,
In step S83, the grasp operation confirmation state machine 73 sets the Release flag or Grasp flag set in the small block in the processing in step S78, S79, or S82, and the Stab set in the image area of the current frame.
The grip operation is recognized from the le flag, and step S1 in FIG.
Return to 2.

Here, referring to FIGS. 19 and 20,
The operation of the gripping motion recognition state machine 73 will be described in more detail.

FIG. 19 shows a gripping motion recognition state machine 7.
FIG. 6 is a diagram for explaining a state transition of No. 3; In the figure, Open
Move represents a state in which the hand is open (a state in which the user wants to grasp), OpenStopCount represents a small stop state before grasping the hand, CloseRecog represents a state in which the operation of grasping the hand is recognized, CloseMove indicates a state of moving while holding the hand, CloseStopCount indicates a small stop state after holding the hand, and OpenRecog indicates a state of recognizing the opening operation from the state of holding the hand (holding the hand). Is released). Also, the upper row of each column,
The condition for the transition is shown, and the lower part of each column shows the operation at that time.

FIG. 20 is a diagram for explaining the state transition of the gripping operation recognition state machine 73 of another example.

For example, when the distance between the center of gravity of the image area of the current frame and the center of gravity of the image area one frame before is determined not to be within the predetermined threshold value by the processing of step S81 in FIG. Since the processing is skipped, the stable flag is not set in the image area of the current frame. That is, based on the condition that the state is not Stable, the grasping operation recognition state machine 73 transits to the state of OpenMove from the transition state table shown in FIG.
In the case of 0, the transition state is L1).

Further, for example, when the gripping motion recognition state machine 73 is in the state of OpenMove, step S in FIG.
When the Stable flag is set in the image area of the current frame by the process of 82, that is, because of the condition of Stable, the gripping motion recognition state machine 73 changes from the transition state table shown in FIG. 19 to the state of OpenStopCount. (In the case of the example of FIG. 20, the transition state L2). At this time, the variable
1 is assigned to Open_count.

Further, for example, when the gripping motion recognition state machine 73 is in the state of OpenStopCount, the distance between the center of gravity of the image area of the current frame and the center of gravity of the image area one frame before is determined by the process of step S81 in FIG. From the transition state table shown in FIG. 19 when it is determined that the value is not within the threshold of
The gripping motion recognition state machine 73 transits to the OpenMove state (transition state L3 in FIG. 20). At this time,
0 is assigned to the variable Open_count (the variable Open_count is reset).

Further, for example, when the gripping motion recognition state machine 73 is in the state of OpenStopCount, the processing of step S82 in FIG.
If the le flag is set, that is, from the condition of Stable, the grasping operation recognition state machine 73 transits to the state of OpenStopCount from the transition state table shown in FIG. 19 (in the example of FIG. 20, the transition state L4). At this time, 1 is incremented in the variable Open_count.

Also, for example, when the gripping motion recognition state machine 73 is in the state of OpenStopCount, the variable Open_coun
Based on the condition that t is larger than a predetermined threshold, the grasping motion recognition state machine 73 is obtained from the transition state table shown in FIG.
Transitions to the state of CloseRecog (transition state L5 in the case of FIG. 20).

Further, for example, when the gripping motion recognition state machine 73 is in the CloseRecog state, the gripping motion recognition state machine 73 can be determined from the transition state table shown in FIG.
(In the example of FIG. 20, the transition state L
6).

For example, if the standard deviation of the small block is determined not to be within the predetermined threshold value by the processing of step S76 in FIG. 17 while the gripping motion recognition state machine 73 is in the CloseRecog state, the processing of step S78 is performed. It is determined that the Release flag has been set for the small block, or that the distance between the center of gravity of the image area of the current frame and the center of gravity of the image area one frame before is not within a predetermined threshold value by the processing of step S81. In other words, the grip state recognition state machine 73 transits to the state of CloseRecog from the transition state table shown in FIG. 19 based on the condition that the state is not Grasp or not Stable (in the case of FIG. 20, Transition state L7).

For example, when the Grasp flag is set for the small block by the processing of step S79 in FIG. 17 while the gripping motion recognition state machine 73 is in the CloseRecog state, and by the processing of step S81, the current frame If the Stable flag is set in the image area, that is, from the transition state table shown in FIG. 19, the gripping motion recognition state machine 73 transits to the CloseMove state based on the condition of Grasp and Stable ( FIG.
In the case of the example of 0, the transition state is L8). At this time, the gripping motion recognition state machine 73 recognizes the gripping motion.

For example, when the gripping motion recognition state machine 73 is in the CloseMove state, the distance between the center of gravity of the small block of the current frame and the center of gravity of the small block one frame before is determined by the processing of step S81 in FIG. Is determined not to be within the threshold value, that is, from the condition that the state is not Stable, the gripping motion recognition state machine 73
Transitions to the CloseMove state (transition state L9 in FIG. 20).

Further, for example, if the Stable flag is set in the image area of the current frame by the processing of step S82 in FIG. 17 while the gripping motion recognition state machine 73 is in the CloseMove state, that is, from the condition of Stable. From the transition state table shown in FIG. 19, the gripping motion recognition state machine 73 transits to the state of CloseStopCount (transition state L10 in the example of FIG. 20). At this time,
1 is assigned to the variable Close_count.

For example, when the gripping motion recognition state machine 73 is in the state of CloseStopCount, the distance between the center of gravity of the image area of the current frame and the center of gravity of the image area one frame before is determined by the processing of step S81 in FIG. From the transition state table shown in FIG. 19 when it is determined that the value is not within the threshold of
The gripping motion recognition state machine 73 transitions to the CloseMove state (transition state L11 in FIG. 20). At this time, 0 is assigned to the variable Close_count (the variable Close_cou
nt is reset).

Further, for example, when the gripping motion recognition state machine 73 is in the CloseStopCount state, the processing of step S82 in FIG.
When the le flag is set, that is, from the condition of Stable, the grasping motion recognition state machine 73 transits to the state of CloseStopCount from the transition state table shown in FIG. 19 (in the example of FIG. 20, the transition state L12). At this time, 1 is incremented to the variable Close_count.

Further, for example, when the gripping motion recognition state machine 73 is in the CloseStopCount state, the variable Close_co
Based on the condition that unt is greater than a predetermined threshold, the transition state table shown in FIG.
Transitions to the OpenRecog state (transition state L13 in FIG. 20).

For example, when the release flag is set in the small block by the processing in step S78 in FIG. 17 while the gripping motion recognition state machine 73 is in the state of OpenRecog, and in the processing in step S82, If the Stable flag is set on the image area,
That is, from the transition state table shown in FIG. 19, the gripping motion recognition state machine 73 transits to the OpenMove state based on the condition of Release and Stable (transition state L14 in the example of FIG. 20). .

For example, in the transition state table shown in FIG. 19, when the gripping motion recognition state machine 73 is in the OpenRecog state and the moving distance is larger than a predetermined threshold, the gripping motion recognition state machine 73 indicates that the CloseMov.
e state (in the example of FIG. 20, the transition state L1
5).

Further, for example, when it is determined that the standard deviation of the small block is not within the predetermined threshold value by the processing of step S76 in FIG. 17 while the gripping motion recognition state machine 73 is in the OpenRecog state, step S7
If the Grasp flag is set for the small block by the process of step 9 or the distance between the center of gravity of the image region of the current frame and the center of gravity of the image region one frame before is within a predetermined threshold value by the process of step S81. If it is determined that the state is not “Release” or “Stable”, the gripping motion recognition state machine 73 transits to the OpenRecog state from the transition state table shown in FIG. 19 (the example in FIG. 20). , The transition state L16).

As described above, the grasping motion recognition state machine 73 recognizes the grasping or opening motion from the Release flag or Grasp flag set in the small block and the Stable flag set in the image area of the current frame. be able to.

Returning to FIG. 9, in step S12,
The center-of-gravity trajectory recognition unit 32 recognizes an operation of waving the hand up, down, left, and right based on the trajectory of the center of gravity of the area selected in the process of step S10.

Here, with reference to the flowcharts of FIGS. 21 and 22, the processing of recognizing a hand movement will be described.

In step S101, the center-of-gravity trajectory recognizing unit 32 substitutes the center-of-gravity position of the region selected in each frame into a ring buffer composed of ten buffers as shown in FIG. In step S102, the center-of-gravity trajectory recognition unit 32 updates the center-of-gravity positions X, Y of the current frame and the center-of-gravity positions X ', Y' of the previous frame, which are substituted in the processing of step S101. In this case,
The center of gravity of the current frame is updated to X = X ₅ , Y = Y ₅ , and the center of gravity of one frame before is updated to X ′ = X ₄ , Y ′ = Y ₄ .

In step S103, the center-of-gravity trajectory recognizing unit 32 determines whether or not the absolute value of XX 'is larger than a predetermined positive threshold, that is, the X coordinate of the current frame and n frames (one frame in this case). ) Compare the previous X 'coordinates,
It is determined whether or not there has been a change, and if it is determined that the absolute value of XX ′ is not larger than the predetermined positive threshold, that is, if it is determined that there is no change, the process proceeds to step S104.

In step S104, the center-of-gravity trajectory recognizing unit 32 determines whether the ring buffer has made one round. If it is determined that the ring buffer has not yet made one round, the process returns to step S102 to return to the current frame. Of the center of gravity X, Y ′ of the current frame and the center of gravity X ′, Y ′ of the previous frame are updated (in this case, the center of gravity X = X ₄ , Y of the current frame)
= Y ₄ , center of gravity position X ′ = X ₃ , Y ′ = Y ₃ one frame before
And repeat the subsequent processing. That is, X
The center of gravity X, Y of the current frame and the center of gravity X one frame before, which are traced back by one frame until the coordinates change.
', Y'. If it is determined in step S104 that the ring buffer has made one round, the process proceeds to step S128.

In step S103, XX
If it is determined that the absolute value of 'is larger than the predetermined positive threshold, the process proceeds to step S105,
The coordinates X 'and Y' at the time of the change are substituted for Xs and Ys, respectively, and the variable max is initialized to 0 and the variable BigSkip is initialized to FALSE. In step S106, the center-of-gravity locus recognizing unit 32 determines whether XX ′ is greater than 0,
If it is larger (XX 'is a positive value), the process proceeds to step S107, and it is determined whether XX' is larger than a predetermined positive threshold.

If it is determined in step S107 that XX 'is larger than the predetermined positive threshold, the process proceeds to step S107.
Proceeding to 108, the center-of-gravity trajectory recognition unit 32 sets the variable BigSkip to T
Substitute RUE. In step S107, X
If it is determined that −X ′ is not larger than the predetermined positive threshold, the process proceeds to step S109, and the center of gravity locus recognizing unit 32
Further determines whether or not XX ′ is smaller than a predetermined negative threshold. If it is determined that XX ′ is smaller than the predetermined negative threshold, the process proceeds to step S123 and determines that the value is not smaller than the predetermined negative threshold. If it is determined, the process proceeds to step S110.

After the processing in step S108 or S109, in step S110, the center-of-gravity trajectory recognition unit 32
Determines whether or not the absolute value of X-Xs is greater than the absolute value of variable max (in this case, 0). If it is determined that the absolute value of X-Xs is greater than the absolute value of variable max, step Proceeding to S111, the value of X-Xs is substituted for the variable max. If it is determined in step S110 that the absolute value of X-Xs is not larger than the absolute value of the variable max, the process skips step S111 and proceeds to step S112.

Step S110 or step S111
In step S112, the center-of-gravity trajectory recognition unit 32 determines whether or not the absolute value of Y-Ys is smaller than a predetermined threshold. If it is determined that the absolute value is smaller than the predetermined threshold, the process proceeds to step S113. If it is determined that the absolute value of Y-Ys is not smaller than the predetermined threshold, step S1
Proceed to 28.

In step S113, the center-of-gravity trajectory recognizing unit 32 determines whether or not the ring buffer has made one round. 32 is
The center-of-gravity positions X and Y of the current frame and the center-of-gravity positions X 'and Y' of the previous frame are updated, and the process returns to step S107.
In this case, the barycentric position of the current frame is updated to X = X ₄ , Y = Y ₄ , and the barycentric position X ′ = X ₃ , Y ′ = Y ₃ of the previous frame, which is advanced by one frame. Step S
If it is determined in step 113 that the ring buffer has made one round, the process proceeds to step S123.

In step S106, XX
If it is determined that ′ is not greater than 0 (if XX ′ is determined to be a negative value), the process proceeds to step S115, and the center-of-gravity trajectory recognition unit 23 determines that XX ′ is a predetermined negative value. It is determined whether or not the threshold value is smaller than a predetermined threshold value. If it is determined that the threshold value is smaller than a predetermined negative threshold value, the process proceeds to step S116. In step S116, the center-of-gravity trajectory recognition unit 32 determines that the variable BigS
Assign TRUE to kip. If it is determined in step S115 that XX ′ is not smaller than the predetermined negative threshold, the process proceeds to step S117, and the center-of-gravity locus recognizing unit 32 further determines that XX ′ has a predetermined positive threshold. It is determined whether the value is larger than the predetermined positive threshold. If it is determined that the value is not larger than the predetermined positive threshold, the process proceeds to step S118. If it is determined that the value is larger than the predetermined positive threshold, the process proceeds to step S123.

After the processing in step S116 or S117, in step S118, the center-of-gravity trajectory recognition unit 32
Determines whether or not the absolute value of X-Xs is greater than the absolute value of the variable max. If it is determined that the absolute value of X-Xs is greater than the absolute value of the variable max, the process proceeds to step S119, where the variable max is Substitute the value of X-Xs. In step S118, the absolute value of X-Xs is set to the variable max.
If it is determined that the value is not larger than the absolute value of
Step 119 is skipped, and the process proceeds to step S120.

Step S118 or step S119
In step S120, the center-of-gravity trajectory recognition unit 32 determines whether or not the absolute value of Y-Ys is smaller than a predetermined threshold, and proceeds to step S121 if it is determined that the absolute value is smaller than the predetermined threshold. If it is determined that the absolute value of Y-Ys is not smaller than the predetermined threshold, step S1
Proceed to 28.

In step S121, the center-of-gravity trajectory recognition unit 32 determines whether or not the ring buffer has completed one round. If it is determined that the ring buffer has not yet completed one round, the process proceeds to step S122, and the center-of-gravity trajectory recognition unit 32 proceeds. 32 is
The center-of-gravity positions X and Y of the current frame and the center-of-gravity positions X 'and Y' of the previous frame are updated, and the process returns to step S115.
In this case, the barycentric position of the current frame is updated to X = X ₄ , Y = Y ₄ , and the barycentric position X ′ = X ₃ , Y ′ = Y ₃ of the previous frame, which is advanced by one frame. Step S
If it is determined in 121 that the ring buffer has made one round, the process proceeds to step S123.

After the processing of step S109, 113, 117 or step S121, in step S123, the center-of-gravity trajectory recognition unit 32 determines whether or not the variable BigSkip is FALSE, and determines that the variable BigSkip is FALSE. If so, the process proceeds to step S124, step S124
, The center-of-gravity locus recognizing unit 32 determines whether or not the value of the variable max is greater than a predetermined positive threshold, and
Is determined to be larger than the predetermined positive threshold, the process proceeds to step S125, and it is recognized that the right-hand swing has been performed.
It returns to step S13.

In step S124, the variable m
If it is determined that ax is not larger than the predetermined positive threshold value, the process proceeds to step S126,
Further, it is determined whether or not the variable max is smaller than a predetermined negative threshold, and if it is determined that the value of the variable max is smaller than the predetermined negative threshold, the process proceeds to step S127, and it is recognized that a left swing has been performed. It returns to step S13 of FIG.

In step S123, the variable BigSkip
If it is determined that is not FALSE, step S126
In step S104, it is determined that the variable max is not smaller than the predetermined negative threshold.
2, or after the processing of step S120, step S1
At 28, the center-of-gravity trajectory recognition unit 32 recognizes that the hand has not been shaken, and returns to step S13 in FIG.

Further, another example of the swinging motion recognition process will be described with reference to the flowchart of FIG. In step S141, the center-of-gravity trajectory recognition unit 32 sets the curr_g
Substitute Right, Left, or None for es. That is,
Steps S125, S127 or S12 in FIG.
Processing result at 8 (right, left, or no hand gesture)
Is assigned. In step S142, the center-of-gravity locus recognizing unit 32 compares the time of the current frame with the reference time and calculates the elapsed time of the current frame. In step S143, the center-of-gravity trajectory recognition unit 32 determines whether the current frame has moved rightward or leftward. If it is determined that the current frame has moved rightward or leftward, the process proceeds to step S1.
Go to 44. In step S144, the center-of-gravity locus recognizing unit 32 updates the reference time for calculating the elapsed time to the time of the current frame.

If it is determined in step S143 that there has been no rightward or leftward movement, the process in step S144 is skipped, and the flow advances to step S145. In step S145, the center-of-gravity trajectory recognition unit 32
Determines whether or not the elapsed time of the current frame calculated in the process of step S142 is longer than a predetermined threshold. If it is determined that the elapsed time of the current frame is longer than the predetermined threshold, the process proceeds to step S146, where the variable Output the gesture (right or left) stored in prev_ges.

If it is determined in step S145 that the elapsed time of the current frame is not longer than the predetermined threshold, the process proceeds to step S147, and the center-of-gravity trajectory recognition unit 32 determines
Gesture of the current frame (turn right or left)
Output curr_ges. In step S148, the center-of-gravity locus recognizing unit 32 substitutes (saves) curr_ges, which is the gesture of the current frame, output in step S147, into prev_ges.

In step S149, the center-of-gravity locus recognizing unit 32 recognizes the swinging motion from the gesture output in step S146 or S147, and returns to step S13 in FIG.

In the above-described processing, when one motion (for example, leftward motion) changes to another motion (for example, rightward motion) in a short time, one (leftward) motion is excluded. Since the other (rightward) movement is recognized, the processing can be sped up. It should be noted that the hand gesture in the vertical direction can be recognized in the same manner as in the processing in FIG. 21, FIG. 22, or FIG.

In step S13 of FIG. 9, the gesture recognition section 3 of the image processing device 3 supplies the recognition results of steps S11 and S12 to the display / output device 4, and displays the gesture (gesture or hand gesture) of the user 1. Let it. In step S14, the image processing device 3 determines whether or not the gesture / hand gesture of the user 1 has ended, and if it determines that the gesture / hand gesture has not ended yet,
Returning to step S1, the above processing is repeated. If it is determined in step S14 that the gesture / hand gesture has ended, the process ends.

As described above, since the gesture and hand gesture of the user can be recognized from the image data captured by the video camera, the computer or the like can be easily operated from a remote place without using an input device such as a mouse or a keyboard. Can be operated.

Further, since a special device and a plurality of video cameras are not used, a low-cost system can be realized.

In the above description, not only the hand grip / opening operation but also the hand movement (trajectory) operation and the hand swinging operation can be recognized. Move the hand (move the mouse), hold the hand (icon selection), move the hand (drag while selecting the icon), open the hand (deselect icon) or swing left (mouse left Button click) and rightward swing (right mouse button click). Therefore,
The user can start an application on the screen or move a window displayed on the screen by an intuitive operation without using a mouse.

The above-described series of processing can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software can execute various functions by installing a computer built into dedicated hardware or installing various programs. It is installed from a recording medium into a possible general-purpose personal computer or the like.

This recording medium is separate from the computer.
A magnetic disk (including a floppy disk) on which the program is recorded, an optical disk (CD-ROM (Compact Disk-Read) distributed to provide the program to users.
It is not only composed of package media consisting of magneto-optical disks (including MD (Mini-Disk)), semiconductor memory, etc., including Only Memory), DVD (Digital Versatile Disk), but also pre-installed in the computer. And a hard disk or the like provided with the program for recording the program.

[0124] In this specification, the step of describing a program recorded on a recording medium may be performed in a chronological order according to the described order. This also includes processing executed in parallel or individually.

In this specification, the system is
It represents the entire device composed of a plurality of devices.

[0126]

As described above, the image processing apparatus of the present invention
According to the image processing method and the program recorded on the recording medium, an image of an object to be gestured or shaken is captured, the captured image is identified, the identified image is divided into predetermined regions, and the divided image is divided. The direction of movement within the region
Calculate the center of gravity in the area, calculate the evaluation value in the area, select the area with the smallest evaluation value from the calculation result, recognize the gripping operation of the object in the selected area, and select the area. Since the swing motion of the target object is recognized based on the trajectory of the center of gravity in the region, the user's gesture and hand gesture can be recognized at low cost and with high accuracy by image processing.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration example of an image processing system to which the present invention has been applied.

FIG. 2 is a block diagram illustrating a configuration of an image processing apparatus 3.

FIG. 3 is a block diagram illustrating a detailed configuration of an image processing unit 11;

FIG. 4 is a block diagram illustrating a detailed configuration of a gesture recognition unit 12;

FIG. 5 is a block diagram showing a detailed configuration of a region dividing unit 21.

FIG. 6 is a block diagram illustrating a detailed configuration of a region dividing unit 21 of another example.

FIG. 7 is a block diagram showing a detailed configuration of an area processing unit 23.

FIG. 8 is a block diagram illustrating a detailed configuration of a gripping motion recognition unit 31.

FIG. 9 is a flowchart illustrating hand grip / shake motion recognition processing.

FIG. 10 is a flowchart illustrating an area dividing process in step S2 of FIG. 9;

FIG. 11 is a diagram illustrating color distribution.

12 is a flowchart illustrating another example of the area dividing process in step S2 of FIG. 9;

FIG. 13 is a flowchart illustrating a motion direction detection process in FIG. 9;

FIG. 14 is a diagram illustrating a frame for detecting a motion direction.

FIG. 15 is a correspondence diagram for setting a pixel value indicating a movement direction.

FIG. 16 is a diagram illustrating an operation of calculating a center of gravity.

FIG. 17 is a flowchart illustrating a gripping motion recognition process in step S11 in FIG. 9;

FIG. 18 is a diagram illustrating a small block for calculating a standard deviation.

FIG. 19 is a diagram illustrating a state transition of the gripping motion recognition state machine 73.

FIG. 20 is a diagram illustrating a state transition of the gripping motion recognition state machine 73 of another example.

FIG. 21 is a flowchart illustrating a swing motion recognition process in step S12 of FIG. 9;

FIG. 22 is a flowchart following FIG. 21;

FIG. 23 is a diagram illustrating a ring buffer.

FIG. 24 is a flowchart illustrating a swing motion recognition process in another example of step S12 in FIG. 9;

[Explanation of symbols]

2 video camera, 3 image processing device, 4 display /
Output device, 11 image processing unit, 12 gesture recognition unit, 21 area division unit, 22 image storage unit, 2
3 area processing section, 24 area storage section, 31 grasping motion recognition section, 32 center of gravity locus recognition section, 41 color identification section, 42 labeling section, 51 inter-frame difference calculation section, 52 color / motion evaluation section, 61 circumscribed frame detection section , 62 motion detector, 63 aspect ratio calculator,
64 area calculation section, 65 center of gravity calculation section, 66 pixel number calculation section, 67 area evaluation section, 71 area motion distribution calculation section, 72 area center of gravity movement check section, 73
Grasping motion recognition state machine

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5L096 AA02 AA06 BA08 BA18 CA04 CA21 FA15 FA16 FA54 FA59 FA60 FA66 FA67 GA08 GA19 GA28 GA34 HA05 JA11

Claims (7)

[Claims] 1. An image pickup means for picking up an object to be gestured or gestured, an identification means for identifying an image picked up by the image pickup means, and dividing the image identified by the identification means into a predetermined area. Area dividing means; moving direction detecting means for detecting a moving direction in the area divided by the area dividing means; and a center of gravity calculating a center of gravity in the area based on a detection result of the moving direction detecting means. Means, evaluation value calculation means for calculating an evaluation value in the area, area selection means for selecting the area having the smallest evaluation value from the calculation result of the evaluation value calculation means, selection by the area selection means A grip operation recognizing unit for recognizing a grip operation of the object in the selected area; and a locus of the center of gravity in the area selected by the area selecting unit. Te, an image processing apparatus comprising: a recognizing swinging motion recognizing means pretended operation of the object. 2. A circumscribing frame detecting means for detecting a circumscribing frame of the area divided by the area dividing means, and an aspect ratio calculating means for calculating an aspect ratio of the circumscribing frame detected by the circumscribing frame detecting means. An area calculation unit that calculates the area of the region by counting the number of pixels in the circumscribed frame detected by the circumscribed frame detection unit, and a motion direction based on a detection result of the motion direction detection unit. A counting unit for counting pixels, wherein the evaluation value calculating unit calculates an evaluation value in the area based on the center of gravity, the aspect ratio, the area, and the number of pixels in the movement direction. The image processing apparatus according to claim 1, wherein: 3. A standard deviation calculating means for calculating a standard deviation of a pixel having a movement direction in a predetermined direction in the area selected by the area selecting means, wherein the gripping motion recognizing means comprises the standard deviation. Recognizing the gripping operation of the object based on the calculation result of the calculation means, the movement direction of the center pixel in the area selected by the area selection means, and the calculation result of the center of gravity calculation means. The image processing device according to claim 1. 4. The image processing apparatus according to claim 1, wherein the identification unit identifies an image whose pixel value is within a predetermined range, or determines whether the pixel value of the image is within a predetermined range and a luminance value of the image. 2. The image processing apparatus according to claim 1, wherein the image processing apparatus identifies an object within a predetermined range. 5. The image processing apparatus according to claim 1, wherein the motion direction detecting means detects a direction of motion in the area based on a luminance value of a central pixel in the area. 6. An image capturing step of capturing an object to be gestured or gestured, an identification step of identifying an image captured by the processing of the image capturing step, and the image identified by the processing of the identification step being a predetermined image. An area dividing step of dividing into areas; a motion direction detecting step of detecting a motion direction in the area divided by the processing of the area dividing step; A center of gravity calculating step of calculating a center of gravity of the area; an evaluation value calculating step of calculating an evaluation value in the area; an area selecting step of selecting the area having the smallest evaluation value from the calculation result in the evaluation value calculating step And a grip for recognizing a gripping operation of the object in the area selected by the processing of the area selecting step. Image processing comprising: a crop recognition step; and a swing motion recognition step of recognizing a swing motion of the object based on a locus of the center of gravity in the area selected by the processing of the area selection step. Method. 7. An image capturing step of capturing an object to be gestured or gestured, an identification step of identifying an image captured by the processing of the image capturing step, and An area dividing step of dividing into areas; a motion direction detecting step of detecting a motion direction in the area divided by the processing of the area dividing step; A center of gravity calculating step of calculating a center of gravity of the area; an evaluation value calculating step of calculating an evaluation value in the area; an area selecting step of selecting the area having the smallest evaluation value from the calculation result in the evaluation value calculating step And a grip for recognizing a gripping operation of the object in the area selected by the processing of the area selecting step. A computer comprising: a crop recognition step; and a swing motion recognition step of recognizing a swing motion of the object based on a trajectory of the center of gravity in the area selected by the processing of the area selection step. A recording medium on which a readable program is recorded.