CN101437124A - Method for processing dynamic gesture identification signal facing (to)television set control - Google Patents

Abstract

The signal processing method of dynamic gesture recognition for TV control involves detecting motion from the image captured by the built-in camera of the TV, recognizing the motion of the human hand and converting it into TV control signals (such as channel switching, volume adjustment and more complex functions). A signal processing method for menu operations, etc.). The method uses the built-in camera of the TV to collect the action images of the TV user, analyzes the collected images to obtain the motion area, extracts the information of the hand object from the motion area and generates a trajectory, and then judges the instruction corresponding to the trajectory to generate a TV control command. The present invention proposes a method for controlling a TV with dynamic gestures without manual assistance in locating an opponent. The method obtains the user's sign language by means of motion detection and building a multi-target human body model, and generates corresponding TV control instructions. It provides a more complete remote control method for the field of sign language TV. This remote control method will significantly promote the development of the TV application field.

Description

Signal Processing Method for Dynamic Gesture Recognition Oriented to TV Control

technical field

The invention relates to motion detection from images captured by a built-in camera of a TV, recognition of human hand movements and conversion into a signal for TV control signals (such as channel switching, volume adjustment, and more complex menu operations, etc.) Approach. It belongs to the technical field of television control devices.

Background technique

The current modes of remote control of the TV mainly include:

1. Infrared TV Remote Controller (Infrared TV Remote Controller) remote control,

2. Remote control input similar to mouse,

3. Use the remote control handwriting input of the tablet,

Although there is already a patent on remote control of TV with sign language actions, such as Chinese patent 02101991.6, this patent is based on the recognition of infrared information of human fingertips, and is not an ideal technical solution in terms of feasibility, cost and accuracy. Other similar patents also have some key technical problems to be solved, such as the need to improve the anti-noise ability, or the need for artificial assistance in positioning the opponent, and so on.

Existing technical problems or areas to be improved

1. The current TV remote controllers are all the same in terms of appearance and input operation, which has caused aesthetic fatigue to users and cannot highlight the characteristics of the product in appearance.

2. As TV functions become more and more complicated, the remote control operation is too cumbersome, and the functions used by consumers are still based on the 80%-20% principle, that is, the commonly used functions are only 20%.

3. Watching TV itself is leisure and entertainment. When you need to control the TV (such as adjusting the volume), you need to find the remote control in a hurry, and there are often multiple remote controls at home, such as TV, DVD, air conditioner, which one? One is the TV remote?

4. Disadvantages of current sign language TV: users need to manually locate, not humanized and intelligent enough

Contents of the invention

Technical problem: The object of the present invention is to realize a TV control method combining motion detection and dynamic gesture recognition, by which the TV can be controlled by the operator's hand movements (for example, waving), such as adjusting the volume, switching channels, etc. , thus saving the process of looking for and operating the remote control, freeing the user from dependence on the remote control, and controlling the TV with a wave of the hand.

Technical solution: The present invention’s dynamic gesture recognition signal processing method for TV control relates to a signal processing method combining motion detection and hand shape recognition. Obtain the motion area, extract the information of the hand target from the motion area and generate a trajectory, and then judge the instruction corresponding to the trajectory to generate a TV control command. The method specifically includes the following steps:

Step 1: Use the built-in camera in the TV to collect video data, add image sequences,

Step 2: If the reference background image is not constructed, use the reference background construction method to construct a reference background image for the image sequence collected in step 1,

Step 3: If the reference background image has been constructed, continue to collect the image in real time, and make a point-by-point difference with the reference background image to generate a motion description image,

Step 4: If the reference background image has been constructed, start to collect the image in real time, compare it with the reference background image, generate a motion description image, and convert it into a binary motion mask image,

Step 5: Utilize the reference background construction method to update the reference background image,

Step 6: Carry out target segmentation and judgment on the binary moving image, construct and update a multi-target human body model based on time and space by using the method of multi-target tracking, in which each target corresponds to the segmented human body movement part,

Step 7: If there is a hand target that has been tracked in the multi-target mannequin, after updating the trajectory of the target, check whether the trajectory matches the preset trajectory and generate an operation instruction; otherwise, check whether the trajectory of a specific target matches the one that needs to be tracked The hand object feature, if any, identifies it,

Step 8: If an operation command is generated, it is sent to the built-in software module of the TV, and the corresponding operation is completed after it is processed.

Step 9: Repeat step 3 until the TV is turned off.

in:

The reference background construction method in step 2 is to select different background construction methods according to the difference in motion speed and amplitude: when the motion amplitude and speed are large, the latest image is used as the reference background; The frame image average method is used to calculate the reference background.

The generation method of the binary motion mask image in step 4 is to binarize the motion description image through the set threshold value, and use the set of points larger than a certain threshold as the motion area; other point sets as the non-motion area.

In the sixth step, the method of multi-target tracking is used to construct a multi-target human body model for the motion detection result, and a hand target conforming to a specific motion trajectory is selected as the identification object.

In step seven, the identified trajectory of the hand object is identified using a pattern matching method based on the trajectory library, compared with the preset operation command trajectory, and if there is a match, a corresponding control command is generated.

Beneficial effects: the greatest contribution of the present invention is to propose a method of dynamic gesture control TV without manual assistance in positioning the opponent. This method obtains the user's sign language by means of motion detection and construction of a multi-target human body model, and generates a corresponding TV Control instruction. The invention provides a more perfect remote control mode for the field of sign language television. This remote control method will significantly promote the development of the TV application field.

Description of drawings

Figure 1 is an example of a gesture recognition signal processing flow chart, in which the key steps are identified by numbers ① to ⑧, corresponding to step 1 to step 8 in claim 1, such as the identification ① is to collect video data, and the identification ② is Construct a reference background image, mark ③ to generate a motion description image, mark ④ to generate an improved motion mask image, mark ⑤ to update the reference background, mark ⑥ to generate a multi-target human body model, mark ⑦ to identify the trajectory of the hand object, and mark ⑧Recognition of gestures and generation of TV control commands.

Detailed ways

1. Motion detection

Motion detection consists of three steps:

1. Reference background construction

Before motion detection, it is necessary to construct a reference background based on the images captured by the camera unit. Here are two background construction methods for different situations:

Method 1: multi-frame image statistics method. This method is suitable for slow hand movement. The specific implementation method is to take multiple frames of continuous grayscale images, and use the average value of each point in the image over time as the value of the reference background at that point.

Method 2: Inter-frame reference method. This method is suitable for the case of fast hand movement, and the specific implementation method is to use the last captured image as the reference background of the currently captured image.

2. Motion Region Extraction

After the reference background is constructed successfully, the motion contour is obtained by making the difference between the gray value of each point in the currently captured image and the gray value of each point in the reference background. The specific implementation method is:

If the difference of a certain point is greater than a certain threshold, it is considered that the point belongs to the motion area.

For the reference background constructed by Method 1, a statistical threshold was used. Right now ${\mathrm{\δ}}_i=\frac{2}{\mathrm{no}}\underset{t=1}{\overset{\mathrm{no}}{\mathrm{\Σ}}}|{\mathrm{\μ}}_i\left(t\right)-{\mathrm{\μ}}_i|,$ Where δ _i is the decision threshold of point i, μ _i (t) is the gray value of the image used to construct the reference background at point i in frame t, and μ _i is the gray value of the reference background at point i.

For the background constructed by the second method, a dynamic threshold is adopted.

Usually, the difference distribution between the currently captured image and the reference background satisfies the bimodal characteristic. If μ1 and μ2 are the mean values of the frame-to-frame differences between the static background part and the moving area respectively, set an initial threshold value, and then dynamically adjust the threshold value as follows:

If the current threshold G is greater than (μ1+μ2)/2, then reduce G by one step; otherwise, increase G by one step, and specify G∈(Gmin, Gmax).

3. Erosion operation

Both the motion area and the non-motion area obtained in 2 need to be corroded.

The corrosion calculation method of the motion area:

If a point belongs to a moving object, it can be assimilated into a non-moving area when one of the following conditions is met:

① The two points adjacent to the left and right of this point belong to the non-moving area;

② The two adjacent points above and below this point belong to the non-moving area;

③ The two points adjacent to the upper left and lower right of this point belong to the non-moving area;

④ The two points adjacent to the lower left and upper right of this point belong to the non-moving area;

Corrosion calculation method of non-moving area:

If a point belongs to the non-moving area, the point can be assimilated into the non-moving area when one of the following conditions is met:

① At least one of the two adjacent points to the left and right of this point belongs to the non-moving area;

②At least one of the two points adjacent to this point belongs to the non-moving area;

③ At least one of the two points adjacent to the upper left and lower right of the point belongs to the non-moving area;

④At least one of the two points adjacent to the lower left and upper right of the point belongs to the non-moving area;

4. Background update

In the case of using method 1 in 1 to construct the reference background, in order to ensure the adaptability of the reference background, the reference background needs to be updated, so that objects that suddenly move can be found, and objects that stop moving can be integrated into the reference background. , the erosion operation mentioned in 2 can guarantee the realization of this.

The specific update method is:

${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; \&alpha;</span>}{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>& {\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\\ {\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>& {\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right.$

${{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; \&alpha;</span>}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; )</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}& {\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\\ {{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>& {\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right.$

Among them, I _i (t) is the gray value of point i in the image at time t; D _i =0 means that point i belongs to the non-moving area, D _i =1 means that point i belongs to the moving area; α is a forgetting factor, and the reference value is 0.8. Other definitions are the same as in 1.

2. Dynamic Gesture Recognition

The recognition of dynamic gestures includes the following three parts: multi-target tracking, human body modeling, trajectory discrimination and dynamic gesture recognition.

In order to make the system have higher robustness and recognition ability, we use the method of multi-target tracking to build a human body model for the motion detection results, and perform trajectory discrimination and dynamic gesture recognition on the basis of the model, and also use judgment and recognition The results are used as feedback to correct the human body model. Commonly used multi-target tracking methods include Joint Probabilistic Data Association Filter (JPDAF), Multiple Hypothesis Tracking (MHT) algorithm, dynamic multi-dimensional allocation algorithm, infinite Kalman filter and particle filter, etc. These methods have their own advantages and disadvantages, and are suitable for different applications. It is used in different scenarios and different environments, and its principles are not described here.

The motion detection of real-time video data can get a series of moving objects in each frame. We use the method of multi-target tracking to find the corresponding targets for these moving objects, and according to their trajectories to match the characteristics of human body parts. effective target for building a human model. In the multi-target human body model, each effective target has its own space-time trajectory. No matter in the time axis or in the space coordinate system, the effective targets have the characteristics of continuity, and different targets have their own different Manifestations. For example, the movement of the hand object is stronger than that of the head object, and the spatial distance between the hand object and the head object is always maintained at a certain interval; the spatio-temporal characteristics of the head object are often maintained at the same height and do not appear frequently jitter, and so on. These features provide the basis for us to build a multi-objective human model.

In the multi-target human body model, we select objects that meet specific trajectory characteristics as the marked hand targets, and identify their trajectories to determine whether to generate control commands. Commonly used trajectory recognition methods include feature-based recognition, rule-based recognition, and rule-based pattern matching. In the process of trajectory discrimination and dynamic gesture recognition, we can use the preset trajectory library as the object to be compared. Once If the target trajectory conforms to the characteristics of the specific trajectory in the predetermined trajectory library, they are considered to be the same trajectory, and corresponding instructions can be generated. In addition, the trajectory library can also support the function of dynamic update, and can record the trajectory of specific functions according to the needs of users to achieve personalized needs.

After trajectory discrimination and dynamic gesture recognition are completed, the results of discrimination and recognition can also be used as feedback for the correction of multi-target human body models. When wrong identification or misjudgment occurs, the results of trajectory discrimination and recognition can be used as high-weight The information corrects the existing logo, and the human body model system is also further improved in continuous revision.

Claims (5)

1. A dynamic gesture recognition signal processing method for TV control, characterized in that: the built-in camera of the TV is used to gather the TV user's action image, and the captured image is analyzed to obtain the motion area, and the hand target is extracted from the motion area. information and generate a trajectory, and then judge the instruction corresponding to the trajectory to generate a TV control command. The method specifically includes the following steps: Step 1: Use the built-in camera in the TV to collect video data, add image sequences, Step 2: If the reference background image is not constructed, use the reference background construction method to construct a reference background image for the image sequence collected in step 1, Step 3: If the reference background image has been constructed, continue to collect the image in real time, and make a point-by-point difference with the reference background image to generate a motion description image, Step 4: If the reference background image has been constructed, start to collect the image in real time, compare it with the reference background image, generate a motion description image, and convert it into a binary motion mask image, Step 5: Utilize the reference background construction method to update the reference background image, Step 6: Carry out target segmentation and judgment on the binary moving image, construct and update a multi-target human body model based on time and space by using the method of multi-target tracking, in which each target corresponds to the segmented human body movement part, Step 7: If there is a hand target that has been tracked in the multi-target mannequin, after updating the trajectory of the target, check whether the trajectory matches the preset trajectory and generate an operation instruction; otherwise, check whether the trajectory of a specific target matches the one that needs to be tracked The hand object feature, if any, identifies it, Step 8: If an operation command is generated, it is sent to the built-in software module of the TV, and the corresponding operation is completed after it is processed. Step 9: Repeat step 3 until the TV is turned off. 2. A kind of dynamic gesture recognition signal processing method facing TV control according to claim 1, characterized in that the reference background construction method in step 2 is to select different background construction methods according to the difference of motion speed and amplitude: when motion In the case of large amplitude and speed, the latest image is used as the reference background, and in the case of gentle motion, the method of averaging multiple frames of images is used to calculate the reference background. 3. a kind of dynamic gesture recognition signal processing method facing TV control according to claim 1, it is characterized in that the generation method of binary motion mask image in the step 4 is to carry out motion description image by the threshold value of setting Binarization, the set of points larger than a certain threshold is regarded as a motion area; other point sets are regarded as a non-motion area. 4. A kind of dynamic gesture recognition signal processing method oriented to TV control according to claim 1, characterized in that in the step 6, the multi-target tracking method is used to construct a multi-target human body model for the motion detection result, and therefrom select a target body model that meets a specific motion. The hand target of the trajectory is used as the identification object. 5. a kind of dynamic gesture recognition signal processing method facing TV control according to claim 1, it is characterized in that in the step 7, adopt the pattern matching method based on the trajectory storehouse to distinguish the hand object movement track that goes out, and preset Comparing the trajectories of operation instructions, if there is a match, a corresponding control command is generated.