KR20250131372A - Distance adaptive hand gesture recognition method and apparatus based on radar and deep learning - Google Patents

Abstract

The present invention proposes a radar and deep learning-based distance-adaptive hand gesture recognition method and device, which includes a process of generating a plurality of learning models by learning data according to a plurality of distances between a radar and a user's hand, a process of transmitting a continuous wave from the radar toward the user's hand, a process of receiving a continuous wave reflected according to the user's hand gesture at the radar, a process of generating a spectrogram for the received data of the radar and calculating a distance from the radar to the user's hand, and a process of selecting a learning model according to the calculated distance from among the plurality of learning models and performing inference.

Description

Distance-adaptive hand gesture recognition method and apparatus based on radar and deep learning

The present invention relates to a hand gesture recognition method, and more particularly, to a hand gesture recognition method and device based on a frequency shift keying (FSK) radar and deep learning.

Sensor technology is used to acquire user or surrounding information and enable users to perform desired actions. With the recent rise of IoT (Internet of Things) technology, research is actively underway on using sensors to control smart devices. Among these, the need for HMI (human-machine interaction), which recognizes human movement and controls machines, is emerging in various applications such as smart cars and smart homes. In particular, active research is being conducted on hand gesture recognition systems that can efficiently control devices by recognizing hand gestures.

Existing methods for capturing user hand gestures have utilized devices that capture hand gestures, or multiple optical sensors or cameras. However, these methods have limitations in their usability due to environmental influences, brightness, and obstacles. Furthermore, while these methods can easily measure large hand gestures, they struggle to measure finer movements.

To overcome these shortcomings, hand gesture recognition systems using radar have become necessary. Currently, most hand gesture recognition systems using radar utilize continuous wave (CW) radar. CW radar generates Doppler frequencies according to the movement of an object, which can be used to determine the object's speed. However, it cannot measure the distance between the object and the radar. Consequently, the recognition rate for hand gestures decreases as the distance exceeds a preset distance. Therefore, hand gesture recognition systems using CW radar require limited positions between the user and the radar, which can be inconvenient for users.

Meanwhile, artificial intelligence has been increasingly used in radar-based hand gesture recognition systems. Hand gesture recognition systems utilizing CW radar and AI generate spectrograms using data received via CW radar, and then use these spectrograms as input for inference using a convolutional neural network (CNN).

Korean Patent Publication No. 10-2017-0012422

The present invention provides a frequency shift keying (FSK) radar capable of measuring the distance between an object and a radar and a deep learning-based hand gesture recognition method.

The present invention provides a hand gesture recognition method capable of improving the recognition rate of hand gestures by training a model according to various distances between a radar and a hand.

The present invention relates to a hand gesture recognition method and device capable of improving the recognition rate of hand gestures by measuring the distance between a radar and a hand and then using a learning model based on the distance measured during inference.

According to one embodiment of the present invention, a distance-adaptive hand gesture recognition method based on radar and deep learning includes a process of generating a plurality of learning models by learning data according to a plurality of distances between a radar and a user's hand, a process of transmitting a continuous wave from the radar toward the user's hand, a process of receiving a continuous wave reflected according to the user's hand gesture at the radar, a process of generating a spectrogram for the received data of the radar and calculating a distance from the radar to the user's hand, and a process of selecting a learning model according to the calculated distance from among the plurality of learning models and performing inference.

The process of generating the above multiple learning models includes a process of making the distance between the radar and the user's hand different in multiple ways, transmitting a continuous wave from the radar and then receiving the continuous wave reflected from the user's hand at each distance to measure hand movements according to each distance, a process of generating a spectrogram at each of the multiple distances using a short-term Fourier transform, and a process of generating a multiple learning model by training using the spectrogram generated at each of the multiple distances between the radar and the hand movement.

Data from two adjacent streets are combined and trained.

The above radar uses a frequency shift keying (FSK) radar with carrier frequencies of f _a and f _b .

The distance between the above radar and the user's hand is calculated by [Mathematical Formula 1].

[Mathematical Formula 1]

Here, f _a and f _b are the carrier frequencies of the radar, c is the speed of light, and Δφ is the phase difference between the two received signals.

According to another embodiment of the present invention, a radar and deep learning-based distance-adaptive hand gesture recognition device includes a radar that generates and transmits a continuous wave and receives a signal reflected according to a user's hand gesture, a spectrogram generation unit that generates a spectrogram through a short-term Fourier transform, a distance calculation unit that calculates a distance between the radar and the user's hand, a learning unit that learns data according to a plurality of different distances between the radar and the user's hand and generates a plurality of learning models, and an inference unit that selects a learning model according to the distance calculated from among the plurality of learning models from the learning unit and makes an inference.

The above radar uses a frequency shift keying radar having carrier frequencies of f _a and f _b .

The distance between the above radar and the user's hand is calculated by [Mathematical Formula 1].

[Mathematical Formula 1]

Here, f _a and f _b are the carrier frequencies of the radar, c is the speed of light, and Δφ is the phase difference between the two received signals.

The above learning unit learns by combining data from two adjacent streets into one.

The present invention recognizes hand gestures using an FSK radar capable of measuring the distance between the radar and a target. Furthermore, the present invention trains multiple models based on various distances between the FSK radar and the target. At this time, data from two adjacent locations can be combined for training. These trained models can be used for inference using a CNN. During inference, the distance between the radar and the hand is measured, a spectrogram is generated using the measured data, and then the hand gesture is inferred using the trained model corresponding to the measured distance. In other words, the present invention selects the optimal model based on the distance between the radar and the hand to infer the hand gesture.

Therefore, the present invention can improve recognition rates at all targeted locations because it selects the optimal model based on distance. Furthermore, because data from two adjacent locations is combined and trained, recognition rates for locations between them can also be improved.

FIG. 1 is a flowchart of a radar and deep learning-based distance-adaptive hand gesture recognition method according to an embodiment of the present invention.
FIG. 2 is a block diagram of a radar and deep learning-based distance-adaptive hand gesture recognition device according to an embodiment of the present invention.
Figure 3 is a diagram illustrating an example of a spectrogram.
Figure 4 is a waveform diagram of a continuous wave of an FSK radar having carrier frequencies of f _a and f _b .
Figure 5 is a photograph showing examples of various hand gestures of a user.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided solely to ensure complete disclosure of the present invention and to fully inform those of ordinary skill in the art of the scope of the invention.

FIG. 1 is a flowchart of a distance-adaptive hand gesture recognition method based on radar and deep learning according to an embodiment of the present invention, and FIG. 2 is a block diagram of a device therefor. That is, FIG. 2 is a device for distance-adaptive hand gesture recognition based on radar and deep learning according to an embodiment of the present invention, and the method of FIG. 1 can be implemented using the device of FIG. 2. The hand gesture recognition method and device according to an embodiment of the present invention will be described below using FIGS. 1 and 2.

Referring to FIG. 1, a distance-adaptive hand gesture recognition method based on radar and deep learning according to an embodiment of the present invention may include a process of learning data according to various distances between a radar and a hand (S110), a process of transmitting a continuous wave from a radar toward a user's hand (S120), a process of performing a hand gesture (S130), a process of receiving data of a continuous wave reflected from a hand from a radar (S140), a process of calculating a distance from the radar to the hand (S150), and a process of selecting and inferring a model according to the calculated distance (S160). That is, a hand gesture recognition device for a hand gesture recognition method may include a radar (100) that generates and transmits a continuous wave and receives a reflected signal according to a user's hand gesture, as illustrated in FIG. 2, a spectrogram generation unit (200) that generates a spectrogram through a short-term Fourier transform (STFT), a distance calculation unit (300) that calculates a distance from the radar to the hand, a learning unit (400) that learns data according to various distances between the radar and the hand, and an inference unit (500) that selects and infers a model according to the calculated distance from the learning unit. The radar and deep learning-based distance-adaptive hand gesture recognition method according to one embodiment of the present invention will be described in more detail for each process as follows.

S110: A plurality of data according to various distances between the radar and the user's hand is trained in the artificial intelligence learning unit (400). Training is a process of training an artificial intelligence model capable of distinguishing hand movements. For example, a CNN (Convolutional Neural Network) can be used as the artificial intelligence model. CNN is a type of deep learning. Deep learning is a field of machine learning that utilizes an artificial neural network to learn and recognize complex patterns and features, and builds a deep neural network to perform automated feature extraction and discrimination for various tasks. In order to train a plurality of data according to various distances between the radar and the user's hand, a process of measuring hand movement data according to various distances, a process of generating a spectrogram, and a process of training a model can be performed.

First, the distance between the radar (100) and the user's hand is varied, and hand movements according to each distance are measured. That is, hand movements are measured using the radar while maintaining various distances between the radar (100) and the user's hand. The present invention uses an FSK radar capable of measuring distances as the radar (100). For example, hand movements can be measured using the FSK radar while maintaining the distance between the FSK radar and the hand at 30 cm intervals. If the goal is to cover 30 cm to 180 cm, measurements are made at distances of 30 cm, 60 cm, 90 cm, 120 cm, 150 cm, and 180 cm, respectively. That is, after the user's hand is moved away from the radar (100) at 30 cm intervals, a continuous wave is transmitted from the radar (100), and the continuous wave reflected from the user's hand is received to measure the hand movements.

Next, hand movements are measured at various distances, and spectrograms are generated for each distance. Spectrograms can be generated using the Short Term Fourier Transform (STFT). The STFT is used to transform a time-domain signal into the frequency domain, allowing one to understand how the signal's frequency characteristics change over time. The STFT can be performed through a series of processes, including time segmentation, performing a Fourier transform on each time interval, and generating a time-frequency spectrogram from the resulting data. First, time segmentation divides the original time-series data into small time intervals. The size of the time interval determines the resolution of the STFT, and small time intervals can detect rapidly changing frequencies. Next, a Fourier transform is performed on each time interval to obtain the frequency characteristics for that time interval. The Fourier transform is used to extract the frequency components of each interval. A time-frequency spectrogram is then generated to visually represent how the frequency characteristics of the original signal change over time. A spectrogram generated by the STFT is data that visually represents frequency changes over time. A spectrogram is typically represented as a 2D matrix, with the horizontal axis representing time and the vertical axis representing frequency. The color of each cell represents the signal intensity at that time and frequency. Typically, spectrograms use a color map to represent the intensity of each cell, with higher intensities represented by darker colors. This primarily represents the intensity of frequency components resulting from the Fourier transform. An example of a spectrogram is shown in Figure 3.

It is trained using spectrograms generated at each distance between the radar and the hand gesture. At this time, data from two adjacent locations can be combined and trained. In the embodiment of the present invention, since the distance of 30 cm to 180 cm is measured at 30 cm intervals, a total of 7 models can be trained. That is, 7 models can be trained, including a 30 cm model (model_30), a 30 cm to 60 cm model (model_3060), a 60 cm to 90 cm model (model_6090), a 90 cm to 120 cm model (model_90120), a 120 cm to 150 cm model (model_120150), a 150 cm to 180 cm model (model_150180), and a 180 cm model (model_180). Here, model_30 is a CNN model trained using only 30 cm data, and model_3060 is a CNN model trained using data between 30 cm and 60 cm. model_6090 is a CNN model trained using data between 60cm and 90cm, model_90120 is a CNN model trained using data between 90cm and 120cm, model_120150 is a CNN model trained using data between 120cm and 150cm, model_150180 is a CNN model trained using data between 150cm and 180cm, and model_180 is a CNN model trained using data at 180cm. In other words, data from two adjacent locations can be combined and trained as one.

S120: The present invention uses a Frequency Shift Keying (FSK) radar. That is, a continuous wave is transmitted using an FSK radar as a radar. The FSK radar according to the present invention alternately transmits continuous waves of two frequencies, and each continuous wave has a carrier frequency of f _a and f _b , as illustrated in FIG. 4. At this time, the FSK radar modulates information bits with different frequencies to transmit digital data. That is, the FSK radar is a radar that alternately transmits continuous waves of two frequencies. The FSK radar transmits binary data to transmit continuous waves. That is, the digital data is represented as a binary signal of 0 and 1, and each bit is modulated with a different frequency. At this time, the frequency modulation assigns two different frequencies to each bit of the binary data, for example, 0 is assigned a low frequency, and 1 is assigned a high frequency. The transmitting side of the FSK radar changes the frequency according to the binary data bit and transmits it. In other words, an FSK radar may include a transmitter, antenna, and receiver. The transmitter generates a continuous-wave transmission frequency waveform and modulates the frequency by assigning two different frequencies to each bit of binary data. The modulated data is then transmitted to the target, i.e., the user's hand, via the antenna.

S130: The process of performing a hand gesture can be performed by moving the user's hand in various forms. That is, the hand gesture can be performed by moving the hand up, down, left, and right. For example, the hand can be moved from up to down, or conversely, from down to up. In addition, the hand can be moved from right to left, or conversely, from left to right. Examples of such hand gestures are illustrated in FIG. 5. That is, FIG. 5 is a photograph illustrating examples of various hand gestures applied to the present invention. FIG. 5(a) is a photograph illustrating an up to down movement of a hand (UP to DOWN), and FIG. 5(b) is a photograph illustrating an movement of a hand from right to left (RIGHT to LEFT). In addition, FIG. 5(c) is a photograph illustrating an movement of a hand from left to right (LEFT to RIGHT), and FIG. 5(d) is a photograph illustrating an movement of a hand from down to up (DOWN to UP).

S140: The process of receiving data from radar involves receiving data reflected from a continuous wave transmitted from the radar according to the user's hand movements. In other words, the receiver receives data reflected according to hand movements through the antenna of the FSK radar. The receiver detects frequency changes in the signal received through the antenna and decodes each frequency into digital data. Meanwhile, the FSK radar detects frequency shift modulation by calculating the frequency difference between the transmitted continuous wave and the received reflected wave. This function is performed by the mixer of the FSK radar. That is, the mixer inputs the frequency waveform transmitted from the transmitter and the frequency waveform received from the receiver, calculates the frequency difference between the two signals, and detects frequency shift modulation accordingly. The frequency information obtained through the mixer is transmitted to the signal processing unit, which performs signal processing to convert it into digital data. In this way, the FSK radar for hand gesture detection uses frequency shift keying to transmit data and measures the distance and speed of an object through frequency changes. That is, FSK radar has the advantage of being able to measure distance precisely without limitations in distance resolution because it alternately transmits two different frequencies and calculates distance using the phase difference caused by the different frequencies.

S150: By using FSK radar in this way, the distance between the user and the radar can be measured. The distance (R) between the user and the radar can be calculated as in [Mathematical Formula 1]. That is, the distance calculation unit (200) can calculate the distance between the user and the radar using [Mathematical Formula 1]. FSK radar can calculate the distance between the radar and a moving target in this way, and by using this distance, a distance-adaptive hand gesture recognition system can be implemented without limiting the user's position, unlike when using a conventional CW radar.

Here, c is the speed of light and Δφ is the phase difference between the two received signals.

S160: The inference unit (500) selects a model according to the calculated distance between the radar and the user's hand, performs inference, and recognizes hand movements. Inference is the process of recognizing actual hand movements, and a previously learned model is used. In addition, since FSK radar is used, it has the characteristic of utilizing distance information. First, data on hand movements are received from the radar. Since the example covers 30 cm to 180 cm, the movement is performed within this range. Then, a spectrogram for the radar data is generated using STFT, and the distance from the radar to the hand is calculated using the radar data, and then a model according to the calculated distance is selected. That is, the inference unit (500) selects a model according to the distance calculated by the distance calculation unit (300) from the learning unit (400). At this time, a different model of the learning unit (400) is selected depending on the calculated distance. For example, if the distance between the radar and the hand is 0 cm to 30 cm, select model_30, if it is 30 cm to 60 cm, select model_3060, and if it is 60 cm to 90 cm, select model_6090. Then, if the distance between the radar and the hand is 90 cm to 120 cm, select model_90120, if it is 120 cm to 150 cm, select model_120150, if it is 150 cm to 180 cm, select model_150180, and if it is 180 cm, select model_180. In other words, the inference process calculates the distance between the radar and the hand, selects the learning model corresponding to the calculated distance, and performs inference to recognize the hand gesture. Since the optimal model is selected according to the distance in this way, the recognition rate can be increased at all targeted locations. In addition, if data from two adjacent locations are combined and trained, the recognition rate for the locations in between can also be increased.

The table below shows the results showing the accuracy of distance estimation according to the present invention.

distance 30cm 40cm 50cm 60cm 70cm 80cm 90cm 100cm Comparative example 81.67% 92.08% 77.50% 86.67% 87.08% 85.42% 88.33% 95.00% Example 1 93.28% 94.14% 75.00% 85.59% 86.19% 84.87% 93.94% 93.72% Example 2 92.44% 94.98% 82.92% 91.53% 92.47% 91.18% 92.21% 96.65%

distance 110cm 120cm 130cm 140cm 150cm 160cm 170cm 180cm average Comparative example 95.00% 90.83% 91.25% 92.92% 90.00% 84.58% 86.25% 91.25% 88.49% Example 1 92.02% 95.30% 92.05% 85.17% 93.15% 74.67% 77.49% 92.13% 88.05% Example 2 94.12% 96.15% 86.61% 90.68% 94.52% 86.90% 80.95% 93.06% 91.08%

For each distance, 170 training data and 60 inference data were used for each movement. The training data were measured at 30, 60, 90, 120, 150, and 180 cm, and the test data were all measured at distances of 10 cm and compared with the actual measurements from the FSK radar.

In the comparative example, data of 30, 60, 90, 120, 150, and 180 cm were combined into one, trained as one model, and the resulting model model_all was created, and hand movements were measured using this. In Example 1, six resulting models (model_30, model_60, model_90, model_120, model_150, model_180) corresponding to each distance were derived through data of 30, 60, 90, 120, 150, and 180 cm, and at the time of inference, a model that is close in terms of distance was selected based on the distance estimation value of the FSK radar (e.g., if the distance estimation result is 40 cm, the close model model_30 is selected and inferred). In Example 2, data of close distances among data of 30, 60, 90, 120, 150, and 180 cm were combined and learned to derive the resulting models model_30, model_3060, model_6090, model_120150, model_150180, and model_180. At the time of inference, the corresponding model was selected based on the distance estimation result of the FSK radar. For example, if it is estimated to be 40 cm, model_3060 was selected because it is between 30 and 60 cm, if it is 100 cm, model_90120 was selected because it is between 90 and 120 cm, and if it is 25 cm, model_30 was selected because it is ~30 cm. The experimental results show that Example 1 had good performance at close range, and that Example 2 had good performance overall.

As described above, the present invention recognizes hand gestures using an FSK radar capable of measuring the distance between the radar and the target. In addition, the present invention generates spectrograms according to various distances between the FSK radar and the target and trains multiple models according to the distances using these. The models trained in this way can be used for inference using a CNN. During inference, the distance between the radar and the hand is measured, a spectrogram is generated using the measured data, and then the hand gesture can be inferred using the trained model corresponding to the measured distance. Therefore, the present invention can increase the recognition rate at all targeted locations because it selects the optimal model according to the distance. In addition, because the data from two adjacent locations are combined and trained, the recognition rate for locations in between can also be increased.

While the technical concepts of the present invention have been specifically described through the above-described embodiments, it should be noted that the embodiments are intended for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, those skilled in the art will appreciate that various embodiments are possible within the scope of the technical concepts of the present invention.

100: Radar 200: Spectrogram generator
300: Distance calculation unit 400: Learning unit
500: Inference Department

Claims (9)

A process of creating multiple learning models by learning data according to multiple distances between the radar and the user's hand,
A process of transmitting a continuous wave from the radar toward the user's hand,
A process of receiving a continuous wave reflected according to the user's hand movements from the radar,
A process of generating a spectrogram for the received data of the above radar and calculating the distance from the radar to the user's hand,
A radar and deep learning-based distance-adaptive hand gesture recognition method including a process of selecting and inferring a learning model according to the distance produced from among multiple learning models. In claim 1, the process of generating the plurality of learning models comprises:
A process of changing the distance between the radar and the user's hand to multiple different distances, transmitting a continuous wave from the radar, and then receiving the continuous wave reflected by the user's hand at each distance to measure hand movements according to each distance,
A process of generating spectrograms at each of multiple distances using short-term Fourier transform,
A radar and deep learning-based distance-adaptive hand gesture recognition method comprising a process of creating multiple learning models by training using spectrograms generated at each of multiple distances between radar and hand gestures. In claim 2, a radar and deep learning-based distance-adaptive hand gesture recognition method that combines data from two adjacent distances into one and learns. In claim 1 or claim 3, the radar uses a frequency shift keying (FSK) radar having carrier frequencies of f _a and f _b and a deep learning-based distance-adaptive hand gesture recognition method. In claim 4, a radar and deep learning-based distance-adaptive hand gesture recognition method wherein the distance between the radar and the user's hand is calculated by [Mathematical Formula 1].
[Mathematical Formula 1]

Here, f _a and f _b are the carrier frequencies of the radar, c is the speed of light, and Δφ is the phase difference between the two received signals. A radar that generates and transmits continuous waves and receives reflected signals according to the user's hand movements,
A spectrogram generation unit that generates a spectrogram through short-term Fourier transform,
A distance calculation unit that calculates the distance between the radar and the user's hand,
A learning unit that learns data according to multiple different distances between the radar and the user's hand to generate multiple learning models;
A radar and deep learning-based distance-adaptive hand gesture recognition device including an inference unit that selects and infers a learning model according to a distance produced from among multiple learning models from the above learning unit. In claim 6, the radar is a radar using a frequency shift keying radar having carrier frequencies of f _a and f _b and a deep learning-based distance-adaptive hand gesture recognition device. In claim 7, a radar and a deep learning-based distance-adaptive hand gesture recognition device, wherein the distance between the radar and the user's hand is calculated by [Mathematical Formula 1].
[Mathematical Formula 1]

Here, f _a and f _b are the carrier frequencies of the radar, c is the speed of light, and Δφ is the phase difference between the two received signals. In claim 6 or claim 8, the learning unit is a radar and deep learning-based distance-adaptive hand gesture recognition device that learns by combining data from two adjacent distances.