JP2019003366A - Information input device, information input method and information input system - Google Patents

Abstract

To provide an information input device, an information input method, and an information input system capable of protecting user's privacy with high gesture recognition accuracy and without occupying both hands and eyes.SOLUTION: An information input device 1 includes a signal processing unit 120. The signal processing unit 120 includes a reception processing unit 121, a contact detection unit 122, a state estimation unit 123, and an operation information generation unit 124. The reception processing unit 121 receives detection signals from each of a first detection unit 111 and a second detection unit 112. Based on the detection signals from the first detection unit 111 and the second detection unit 112, the contact detection unit 122 specifies a flat contact period in which a user's fingertip of one hand is in flat contact with the palm of the other hand. The state estimation unit 123 estimates the moving state of the fingertip of the one hand in the palm of the other hand based on the detection signals from the second detection unit 112 in the flat contact period. The operation information generation unit 124 generates operation information for operating an operation target terminal based on the estimation result of the state estimation unit 123.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information input apparatus, an information input method, and an information input system that accept various input information operations performed by a user.

  In recent years, it has been considered to provide information that assists a person in accordance with the behavior and state when the person is performing a normal activity. In such an application, it is bothersome to take out the terminal each time information is provided, perform operation input using both hands, and point the line of sight toward the terminal. Therefore, a hands-free and eyes-free information input device that does not occupy both hands and eyes is considered necessary.

  As this type of information input device, an information input device using voice recognition or an information input device using gesture input is used. Voice recognition allows voice input operations and does not occupy both hands and eyes.

  However, with voice recognition, there is a possibility that the content of the user's utterance may be heard by a third party, which may not be able to protect privacy, and may cause trouble for the surrounding people due to the accompanying sound. .

  On the other hand, in an information input device using gesture input, a finger trajectory (Patent Documents 1 and 2) drawn in a three-dimensional space, or a forearm or wrist surface movement (Patent Documents 3 and 4) is detected by a sensor. Or by tracing the back of the hand with a fingertip to draw a character or the like using optical means (Patent Document 5).

  Since the gesture input does not accompany the generated sound, it is unlikely to cause trouble for the surrounding people. However, with the techniques described in Patent Documents 1 and 2, there is a possibility that a gesture can be seen by a third party, and the privacy of the user may not be protected. In addition, it is difficult to accurately detect the precise three-dimensional position, moving direction, and movement of the fingertip, and the gesture recognition accuracy is not sufficient.

  In addition, the techniques disclosed in Patent Documents 3 and 4 are not user-friendly techniques because of the low recognition accuracy, and there are few types of gestures that can be recognized. Further, with the technique described in Patent Document 5, there is a possibility that the user's operation cannot be accurately detected depending on the presence or absence of an obstacle.

US Patent Application Publication No. 2010/0066664 US Patent Application Publication No. 2004/0263473 US Patent Application Publication No. 2013/0232095 Republished 2015/033152 JP 2014-106765 A

  The present invention has been made in view of such a situation, and an object of the present invention is to provide an information input device and an information input method that can protect the user's privacy and have high gesture recognition accuracy without occupying both hands and eyes. And providing an information input system.

In order to achieve the above object, an information input device according to the present invention includes:
A receiving processing unit that receives a detection signal from at least one of the first detection unit and the second detection unit, and a contact body that makes a planar contact with the object based on the detection signal of at least one of the first detection unit and the second detection unit A contact detection unit that identifies a planar contact period that is being performed, a state estimation unit that estimates a moving state of the contact body on the surface of the object, based on a detection signal of the second detection unit in the planar contact period, And a signal processing unit including an operation information generation unit configured to generate operation information for operating the operation target terminal based on the estimation result of the state estimation unit.

In order to achieve the above object, an information input method according to the present invention includes:
Based on the reception processing step of receiving a detection signal from at least one of the first detection unit and the second detection unit, and the detection signal of at least one of the first detection unit and the second detection unit, the contact body makes a planar contact with the object A contact detection step for identifying a planar contact period, and a state estimation step for estimating a moving state of the contact body on the surface of the object based on a detection signal of the second detection unit in the planar contact period; And an operation information generation step of generating operation information for operating the operation target terminal based on the estimation result of the state estimation unit.

In order to achieve the above object, an information input system according to the present invention includes:
A receiving processing unit that receives a detection signal from at least one of the first detection unit and the second detection unit, and a contact body that makes a planar contact with the object based on the detection signal of at least one of the first detection unit and the second detection unit A contact detection unit that identifies a planar contact period that is being performed, a state estimation unit that estimates a moving state of the contact body on the surface of the object, based on a detection signal of the second detection unit in the planar contact period, An information input device having a signal processing unit having an operation information generation unit for generating operation information for operating the operation target terminal based on the estimation result of the state estimation unit, and transmitted from the information input device And a terminal that operates based on the operation information.

  In the present invention, the movement state of the contact body on the surface of the object is estimated during the plane contact period in which the contact body is in plane contact with the object. More specifically, for example, when the user performs an operation of drawing a character or the like while tracing the surface of the object (such as the user's palm or desk) with a contact object (such as the user's fingertip or pen), the contact object becomes an object. In the plane contact period in which the plane is in contact, the moving state of the contact body that moves on the surface of the object along the trajectory of the character or the like is estimated. Then, operation information based on the estimation result is generated, and the terminal can be operated based on the operation information.

  Such an operation can be performed intuitively, and it is not always necessary to direct a line of sight to the surface of the object or use both hands, and does not occupy both hands and eyes.

  In addition, the user can perform the above operation without being seen by a third party by moving the contact body while hiding it on the surface of the object, thereby protecting the privacy.

  In addition, it is easier to detect the position, moving direction, and movement of a contact that moves on the surface (two-dimensional plane) of an object than when detecting the position, moving direction, and movement of a fingertip that moves in a three-dimensional space. It is possible to improve the accuracy of gesture recognition. Furthermore, since the gesture recognition accuracy is increased, it is possible to recognize many types of gestures, and provide a user-friendly technique.

  Preferably, the first detection unit detects a vibration generated when the contact body makes a planar contact with the object, and the second detection unit detects a motion of the contact body. That is, in the present invention, the plane contact period is specified based on at least one of the vibration waveform signal output from the first detection unit and the motion waveform signal output from the second detection unit. When the contact body makes a flat contact with the object, the vibration at that time appears rapidly in the vibration waveform as a peak. Further, when the contact body moves, the acceleration, angular velocity, and the like at that time appear rapidly in the movement waveform as a peak. Therefore, according to the present invention, the plane contact period can be accurately obtained based on the peak of each waveform.

  Preferably, the state detection unit includes, from the planar contact period, a contact start time at which the contact body starts planar contact with the object and a contact end time at which the contact body finishes planar contact with the object. A movement period excluding a nearby period is specified, and the movement state is estimated based on a detection signal of the second detection unit in the movement period. According to this aspect, it is possible to prevent the accuracy of the estimation result of the moving state of the contact body from being deteriorated due to the influence of the shock wave generated at the contact start time and the contact end time and including an error in the specified plane contact period. it can.

  Preferably, the surface of the object has a substantially planar shape. In this case, the contact body moves on a substantially two-dimensional plane. Therefore, it is possible to accurately estimate the moving state of the contact body as compared with the case where the contact body moves on the surface of the uneven object. Therefore, the gesture recognition accuracy can be further increased.

FIG. 1 is a diagram showing a configuration of a gesture input system having an information input device according to an embodiment of the present invention. FIG. 2 is a block diagram showing a functional configuration of the information input apparatus shown in FIG. FIG. 3A is a flowchart showing the contents of processing executed by the signal processing unit of the information input device. FIG. 3B is a flowchart showing the contents of processing executed by each unit of the signal processing unit of the information input device in the recognition phase. FIG. 3C is a flowchart showing the contents of processing executed by each unit of the signal processing unit of the information input device in the calibration phase. FIG. 4 is a diagram showing a waveform of a detection signal of the detection unit of the information input device. FIG. 5 is another diagram showing a waveform of a detection signal of the detection unit of the information input device. FIG. 6 is a diagram for explaining the contents of processing executed by the state estimation unit in the signal processing unit of the information input device. FIG. 7 is another diagram for explaining the contents of the processing executed by the state estimation unit in the signal processing unit of the information input device. FIG. 8 is a diagram for explaining the calibration.

  Hereinafter, the present invention will be described based on embodiments shown in the drawings.

  FIG. 1 is a diagram showing a configuration of a gesture input system 1 having an information input device 10 according to an embodiment of the present invention. As shown in FIG. 1, the information input system 1 includes an information input device 10 and a terminal 20. The information input device 10 provides a gesture UI (user interface) to the terminal 20. In this gesture UI, the operation of the fingertip is recognized as an input operation to the terminal 20, and information on the input operation is input to the terminal 20.

  More specifically, in the present embodiment, as an input operation to the terminal 20, the user traces the palm of the other hand with a fingertip of one hand so as to draw a predetermined locus (hereinafter referred to as “hand gesture”). ”). Here, examples of the predetermined trajectory include a trajectory of a figure such as a quadrangle, a circle, or an arc, or a trajectory of a character such as “B” or “D”.

  The terminal 20 includes all kinds of electronic devices, electrical products, cars, robots, and drones that accept operation instructions.

  As shown in FIG. 1, the information input device 10 has a ring shape and is fixed to a mounting portion 30 that can be attached to and detached from the wrist, and is mounted on a predetermined mounting site (in the illustrated example, the user's wrist). . As a mounting part of the information input device 10, an elbow and a forearm can be exemplified in addition to the wrist. The user can also perform a hand gesture by tracing a desk or the like with a writing instrument such as a pen held with one hand so as to draw a predetermined locus. In this case, the information input device 10 is attached to a predetermined part such as a writing instrument such as a pen.

  FIG. 2 is a block diagram showing a functional configuration of the information input device 10 shown in FIG. As illustrated in FIG. 2, the information input device 10 includes a detection unit 110, a signal processing unit 120, and a communication unit 130. FIG. 2 shows a terminal 20 in addition to the information input device 10.

  The detection unit 110 includes a first detection unit 111 and a second detection unit 112. The first detection unit 111 detects vibration generated when a contact body (in this embodiment, the fingertip of one hand of the user) makes a flat contact with an object (in this embodiment, the palm of the other hand of the user). And a vibration sensor 111a. The vibration sensor 111a is provided on the back side of the mounting portion 30 (the side in contact with the user's wrist). In addition, as an aspect of the detection unit 110, an aspect in which each of the first detection unit 111 and the second detection unit 112 is incorporated in two different modules that are physically separated, or the first detection unit 111 and the second detection unit are configured. Various modes are conceivable, such as a mode in which the unit 112 is built in one module.

  The vibration sensor 111a detects wrist vibration that occurs when the user performs a hand gesture, and outputs a detection signal having a voltage value corresponding to the vibration. More specifically, when the user makes the fingertip of one hand touch the palm of the other hand during the hand gesture, vibration is generated in the fingertip and the palm, and vibration of about 10 to 40 Hz is propagated to the wrist. . The vibration sensor 111a detects the vibration propagated to the wrist.

  The first detection unit 111 performs A / D conversion on the detection signal of the vibration sensor 111a, and outputs the detection waveform sample V1 to the signal processing unit 120.

  The 2nd detection part 112 is a means for detecting the motion of a contact body, and has the acceleration sensor 112a and the angular velocity sensor 112b. The acceleration sensor 112a detects accelerations in the three axes (x-axis, y-axis, and z-axis) directions of the user's wrist when the hand gesture is executed. The angular velocity sensor 112b detects angular velocities about three axes (x axis, y axis, z axis) that are coordinate axes fixed to the user's wrist when the hand gesture is executed.

  The second detection unit 112 performs A / D conversion on the detection signals of the acceleration sensor 112a and the angular velocity sensor 112b, and outputs the signals to the signal processing unit 120 as the acceleration waveform sample V2a and the angular velocity waveform sample V2b, respectively.

  The signal processing unit 120 includes a reception processing unit 121, a contact detection unit 122, a state estimation unit 123, and an operation information generation unit 124. The processing executed by the signal processing unit 120 is roughly divided into a calibration phase and a recognition phase. In the recognition phase, mainly processing for recognizing the contents of the hand gesture by the user is performed, and in the calibration phase, processing for mainly improving recognition accuracy of the hand gesture recognized in the recognition phase is performed.

  FIG. 3A is a flowchart showing the contents of processing executed by the signal processing unit 120. As shown in FIG. 3A, the signal processing unit 120 first executes processing in the calibration phase (step S10). Next, the signal processing unit 120 executes processing in the recognition phase (step S20). Next, the signal processing unit 120 determines the presence / absence of a reception signal (detection signal of the detection unit 110) (step S30). If the determination result of step S30 is Yes, the signal processing unit 120 ends the process in the recognition phase. On the other hand, when the determination result of step S30 is No, the signal processing unit 120 executes the process in the recognition phase again.

  FIG. 3B is a flowchart illustrating the contents of processing executed by each unit of the signal processing unit 120 in the recognition phase. Details of the calibration phase will be described later.

  Each time the reception processing unit 121 receives the vibration waveform sample V1 from the first detection unit 111, the reception processing unit 121 buffers the received vibration waveform sample V1 (step SA100). Similarly, each time the reception processing unit 121 receives the acceleration waveform sample V2a and the angular velocity waveform sample V2b from the second detection unit 112, the reception processing unit 121 buffers the received waveform samples V2a and V2b (step SA100).

  The contact detection unit 122 performs contact detection processing (step SA110). More specifically, the contact detection unit 122 calculates a statistical value for each frame with a predetermined number of buffered vibration waveform samples V1 as one frame. Similarly, the contact detection unit 122 calculates a statistical value for each frame by setting a predetermined number of buffered acceleration waveform samples V2a as one frame. Similarly, the contact detection unit 122 calculates a statistical value for each frame by using a predetermined number of buffered angular velocity waveform samples V2b as one frame.

  The statistical value includes the maximum value, displacement amount (variation width), peak value of each waveform sample V1, V2a, V2b, or the difference between the statistical value of each frame and the statistical value of the previous frame. Is included.

  Next, the contact detection unit 122 refers to the statistical value for each frame, and the frame in the plane contact period in which the contact body (the fingertip of one hand of the user) is in plane contact with the object (the palm of the other hand of the user). Is identified. Here, the plane contact period is given as a period in which the user's hand is in motion and in contact with some object.

  Specifically, the contact detection unit 122 collects a predetermined number of frames into one window and shifts the window unit to determine whether or not the statistical value of each frame included in the window exceeds a predetermined threshold value. (Or whether it is below). And the contact detection part 122 specifies the flame | frame in a plane contact period based on the determination result.

  FIG. 4 is a diagram illustrating a waveform of a detection signal of the detection unit 110. In FIG. 4, a waveform V1 is a vibration waveform of a detection signal of the vibration sensor 111a. Waveforms V2a_x, V2a_y, and V2a_z are acceleration waveforms in the three-axis (x-axis, y-axis, and z-axis) directions of the detection signal of the acceleration sensor 112a. Waveforms V2b_x, V2b_y, and V2b_z are angular velocity waveforms around three axes (x axis, y axis, and z axis) of the detection signal of the angular velocity sensor 112b.

  As shown in FIG. 4, in the period B, a plurality of peaks appear in the acceleration waveforms V2a_x, V2a_y, V2a_z and the angular velocity waveforms V2b_x, V2b_y, V2b_z, which indicates that the user's hand is in an exercise state. In period B, a plurality of peaks appear in the vibration waveform V1, indicating that the user's hand is in contact with some object. That is, period B is a period in which the user's hand is in motion and in contact with some object (a plane contact period in which the fingertip of one hand of the user is in plane contact with the palm of the other hand) The periods A and C correspond to periods before and after the plane contact period (stationary period). The peak of the vibration waveform sample V1 in the period b is due to surface friction vibration (friction vibration) that occurs when the fingertip of one hand of the user is moving the palm of the other hand. That is, the waveform of the vibration waveform sample V1 in the period C is a waveform corresponding to the friction vibration generated by the friction between the fingertip of one hand of the user and the palm of the other hand.

  FIG. 5 is another diagram illustrating waveforms of detection signals of the first detection unit 111 and the second detection unit 112. The waveforms shown in the figure are the vibration waveform V1, acceleration waveforms V2a_x, V2a_y, V2a_z and angular velocity observed when the user lifts the cup with one hand and moves it in the air and then places the cup on the desk. The waveforms are V2b_x, V2b_y, and V2b_z.

  In FIG. 5, a period A is a period when the user lifts the cup with one hand and moves it in the air. Period B is a period before and after the user places the cup on the desk. In period B, a plurality of peaks appear in the acceleration waveforms V2a_x, V2a_y, V2a_z and the angular velocity waveforms V2b_x, V2b_y, V2b_z, indicating that the user's hand is in an exercise state. In period B, a plurality of peaks appear in the vibration waveform V1, and it can be seen that the cup was in contact with the desk. That is, the period B corresponds to the plane contact period.

  Next, the contact detection unit 122 is based on a frame in the plane contact period and a frame in a period before and after (frame in which the user's hand was in a stationary state) (hereinafter collectively referred to as “a series of frames”). The contact start time Tcs when the fingertip (contact body) of one hand of the user starts to contact the palm (surface of the object) of the other hand, and the fingertip (contact body) of the user's one hand is the palm of the other hand. The contact end time Tce for ending contact with the (object surface) is specified.

  More specifically, in FIG. 4, the contact detection unit 122 refers to a statistical value of a series of frames including a frame in the plane contact period B and frames before and after the stationary periods A and C, and the plane contact period B The time when the peak is observed in the vibration waveform sample V1 is specified. Then, when there is a certain period (period A) during which no other peak is observed before that time, the contact detection unit 122 identifies that time as the contact start time Tcs. In addition, when there is a certain period (period C) in which no other peak is observed after that time, the contact detection unit 122 specifies that time as the contact end time Tce.

  The acceleration waveforms V2a_x, V2a_y, V2a_z and the angular velocity waveforms V2b_x, V2b_y, V2b_z may include peaks due to hand movements that are not intended by the user. Therefore, if the contact start time Tcs and the contact end time Tce are specified based on the peaks of the acceleration waveforms V2a_x, V2a_y, V2a_z and the angular velocity waveforms V2b_x, V2b_y, V2b_z, these values may include an error. In this regard, in the present embodiment, since each time Tcs, Tce is specified based on the peak of the vibration waveform V1, these values can be obtained accurately.

  When the contact detection unit 122 specifies the contact start time Tcs and the contact end time Tce, the contact detection unit 122 gives information indicating the times Tcs and Tce to the state estimation unit 123 together with a series of frames.

  The state estimation unit 123 segments a plane contact period in which the fingertip of one hand of the user is in plane contact with the palm of the other hand, that is, a period from the contact start time Tcs to the contact end time Tce into a plurality of periods. (Step SA120).

  Specifically, the state estimation unit 123 takes out the acceleration waveform sample V2a and the angular velocity waveform sample V2b from the series of frames, and removes high-frequency noise superimposed on the waveform samples V2a and V2b with a low-pass filter and smoothes them. To do.

  Next, the state estimating unit 123 starts the movement start time Tms at which the fingertip (contact body) of the other hand starts moving on the palm of one hand of the user based on the acceleration waveform sample V2a and the angular velocity waveform sample V2b. And the movement end time Tme at which the fingertip (contact body) of the other hand ends moving on the palm of one hand of the user. Then, the state estimation unit 123 estimates the movement state of the fingertip (contact body) of the other hand in the palm of one user's hand (the surface of the object) during the movement period from the movement start time Tms to the movement end time Tme. To do.

  More specifically, in FIG. 4, the state estimation unit 123 performs statistics of the acceleration waveform sample V2a and the angular velocity waveform sample V2b in the plane contact period B, and the acceleration waveform sample V2a and the angular velocity waveform sample V2b in the periods A and C before and after that. With reference to the values, the time at which peaks are observed in the waveform samples V2a and V2b in the vicinity of the contact start time Tcs and the contact end time Tce is specified. Then, when the peak exceeds a predetermined threshold and is observed in the vicinity of the contact start time Tcs, the state estimation unit 123 starts moving at that time (or the time immediately before or immediately after that time). The time is specified as Tms. Further, when the peak exceeds a predetermined threshold and is observed in the vicinity of the contact end time Tce, the contact detection unit 122 finishes moving the time (or the time immediately before or immediately after that time). The time Tme is specified.

  In this way, by specifying the movement period from the movement start time Tms to the movement end time Tme, shock waves in the vicinity of the respective times Tcs and Tce (from the plane contact period from the contact start time Tcs and the contact end time Tce ( It is possible to obtain a period excluding a period affected by a shock wave generated when the fingertip of one hand of the user contacts the palm of the other hand. In other words, the movement period corresponds to a period (friction period) when the fingertip of one hand of the user is rubbing the palm of the other hand.

  Further, the state estimation unit 123 determines the contact period as a stationary period, a surface movement start period, a surface movement period, a surface movement end period, and an air movement based on the statistical values of the acceleration waveform sample V2a and the angular velocity waveform sample V2b in the contact period. Segment into a start period, air movement period and air movement end period. Each period is determined on the basis of the magnitude of each direction component by decomposing each waveform sample V2a, V2b into a component in a direction parallel to a predetermined plane and a component in a direction perpendicular thereto. .

  And the state estimation part 123 performs sensor attitude | position correction | amendment using the acceleration waveform sample V2a and angular velocity waveform sample V2b of a stationary period (step SA130).

  FIG. 6 is a diagram for explaining the contents of processing executed by the state estimation unit 123. The acceleration sensor 112a is usually mounted on the mounting unit 30 in a state where the acceleration sensor 112a is tilted at a certain tilt angle with respect to the horizontal plane of the earth. Therefore, the value of the acceleration waveform sample V2a belongs to the sensor coordinate system depending on the tilt angle as shown in FIG.

  Therefore, the state estimation unit 123 calculates the gravity vector based on the acceleration waveform sample V2a, calculates the angular displacement by integrating the amplitude value of the angular velocity waveform sample V2b, and based on the calculated gravity vector and angular displacement, Sensor attitude correction is performed on the acceleration waveform sample V2a. As a result, the sensor coordinate system is converted into a world coordinate system with the normal to the horizontal plane of the earth as the Z axis.

  Next, the state estimation unit 123 integrates the amplitude value of the acceleration waveform sample V2a, and calculates the speed of the fingertip of the other hand when the palm of one hand is moving. Further, the state estimation unit 123 integrates the calculated speed and calculates the three-dimensional position coordinates of the fingertip of the other hand that moves the palm of one hand. At that time, the state estimation unit 123 performs zero speed correction in order to prevent accumulation of errors due to integration (step SA140).

  Next, the state estimation unit 123 projects the three-dimensional coordinate data onto a predetermined two-dimensional plane, and executes a plane coordinate series conversion process (step SA150). Thereby, the three-dimensional coordinate data of the three-dimensional world coordinate system is converted into the two-dimensional coordinate data of the two-dimensional standard coordinate system (planar coordinate system) (see FIGS. 6 and 7). The predetermined two-dimensional plane is calculated in a calibration phase described later.

  The state estimation unit 123 performs inverse transformation on the two-dimensional coordinate data based on a predetermined transformation matrix to correct distortion (distortion component) superimposed on the locus indicated by the two-dimensional coordinate data, and A direction is specified (step SA160). The predetermined transformation matrix is calculated in a calibration phase described later.

  Then, based on the corrected trajectory (character trajectory information) of the two-dimensional coordinate data, the state estimating unit 123 detects the palm of one hand (the surface of the object) from the movement start time Tms to the movement end time Tme. ) Is estimated, and the result of the estimation is given to the operation information generator 124.

  The operation information generation unit 124 generates operation information for operating the operation target terminal 20 based on the estimation result, and provides the operation information to the communication unit 130 (step SA170).

  Upon receiving the operation information, the communication unit 130 gives this to the terminal 20 wirelessly or by wire. Upon receiving the operation information, the terminal 20 executes various processes according to the operation information, such as displaying characters on the display unit, based on the operation information.

  Next, the calibration phase will be described. In the calibration phase, a predetermined two-dimensional plane used in the plane coordinate conversion process in step SA150 and a predetermined conversion matrix used in the direction / distortion correction in step SA160 are obtained.

  FIG. 3C is a flowchart illustrating the contents of processing executed by each unit of the signal processing unit 120 in the calibration phase. As shown in FIG. 3C, in the calibration phase, each unit of the signal processing unit 120 performs plane detection processing (step SB145) and direction / distortion detection (step SB155) in addition to the processing in the recognition phase (FIG. 3B). And execute.

  In the calibration phase, the user inputs an information input device so that a predetermined character (hereinafter referred to as “calibration character”) is drawn on the palm of the other hand with a single stroke with the fingertip of one hand. Inspired by 10

  As the calibration character, a character having two or more directions and capable of uniquely specifying a plane is used. For example, in addition to characters such as “L”, “X”, and “ko”, various figures ( For example, squares and triangles are also included. In this embodiment, a square is used as the calibration character.

  FIG. 8 is a diagram for explaining the calibration. As shown in FIG. 8A, the user moves the fingertip of one hand in the directions shown in (i) to (vi) in the figure, and draws a square that is a calibration character on the palm of the other hand. .

  When the user executes the above-described operation, the processing from step SA100 to step SA140 is executed, and the state estimation unit 123 acquires three-dimensional coordinate data corresponding to the calibration character. Note that the movement start time Tms and the movement end time Tme specified by the state estimation unit 123 in step SA120 are during the movement of the fingertip of the user's hand in the directions shown in (i) to (vi) in FIG. It is specified from the period.

  Next, the state estimation unit 123 performs a plane detection process (step SB145). More specifically, the state estimation unit 123 obtains a plane to which a large number belongs to a part of the three-dimensional coordinate data in step SA140. In the present embodiment, the orientation of the plane is substantially in accordance with the orientation of the user's palm. That is, when the user performs a hand gesture with the palm of one hand tilted, the plane is slightly tilted with respect to the horizontal plane of the earth. In addition, when the user performs an operation of tracing a wall standing perpendicular to the horizontal plane of the earth with the fingertips of one hand instead of the palm, the plane is a plane perpendicular to the horizontal plane of the earth. . The three-dimensional coordinate data in step SA140 is projected onto the plane obtained in step SB145 (step SA150).

  Here, even if the user draws a square as shown in FIG. 8A, the trajectory indicated by the two-dimensional coordinate data obtained in step SA150 is not a square, but as shown in FIG. 8B. It becomes a shape. This is due to the following reason. That is, as shown in FIG. 1 and the like, since the acceleration sensor 112a is arranged not around the fingertip of the user's hand but around the wrist, the two-dimensional coordinate data obtained in step SA150 is obtained at the position of the user's fingertip. Instead, it indicates the two-dimensional coordinates of the wrist position. Therefore, as shown in FIG. 8B, distortion occurs in the calibration character.

  Therefore, the state estimation unit 123 detects distortion superimposed on the trajectory (trajectory corresponding to the calibration character) indicated by the two-dimensional coordinate data (step SB155), and generates distortion correction information for correcting the distortion. . In addition, the state estimation unit 123 acquires information for specifying the vertical and horizontal directions based on the two-dimensional coordinate data corresponding to the calibration character. As shown in FIG. 8B, the distortion correction information is obtained as a transformation matrix for correcting a distortion character (trapezoid) with distortion to a calibration character (square) without distortion. . This transformation matrix is used in step SA160 in the recognition phase.

  Then, the state estimation unit 123 performs inverse transformation based on the transformation matrix on the two-dimensional coordinate data obtained by the plane coordinate transformation processing in step SA150, and corresponds to the trajectory (corresponding to the calibration character) indicated by the two-dimensional coordinate data. The distortion superimposed on the locus to be corrected is corrected (step SA160). As a result, as shown in FIG. 8B, when the trapezoidal calibration character before the calibration is converted into a square, it is considered that the calibration is successful, and the conversion matrix is also used in the recognition phase. Will be used.

  The above is the configuration of the present invention. Next, a usage example of the information input device 10 according to the present embodiment will be described.

(1) The user came up with a product that had to be purchased while walking. The user thought to take note of the product, but it is dangerous to operate the notepad app on the smartphone while walking, and it takes time and effort to take out the paper and ballpoint pen. Therefore, the user traces the name (character) of the product with the fingertip of one hand on the palm of the other hand. Thereby, the process (refer FIG. 3A) of step S10-step S30 was performed, and the text data which shows the name of goods was memorize | stored in the memo pad application of the smart phone. After arriving at the store, the user started the earphone application on the smartphone, played the text data, and was able to finish shopping easily.

(2) In a certain store, a user who has an utterance disorder tries to convey a product he / she wants to purchase to the store clerk, but does not have paper and a pen and cannot communicate his / her intention to the store clerk. Therefore, the user activates the speech-to-speech application on the smartphone and traces the name of the product (characters) on the counter of the store with the fingertip of one hand. Thereby, the process (refer FIG. 3A) of step S10-step S30 was performed, and the information which shows the name of goods was transmitted to the voice reading application. As a result, the name of the product was read aloud and I was able to convey my intention to the store clerk.

(3) In a busy downtown area, the user wants to execute a voice navigation application for a smartphone. However, the voice input mode is difficult to use from the viewpoint of privacy protection. Also, since it is strange to perform gesture input in the air in the public, the gesture input mode is difficult to use. Therefore, the user taps the palm of the other hand with the fingertip of one hand. Thereby, the process (refer FIG. 3A) of step S10-step S30 was performed, the starting command of the voice navigation application was transmitted to the smart phone, and the voice navigation application started. As a result, the user was able to start the voice navigation application without taking out the smartphone, and listen to the voice navigation application only with the hand gesture and the earphone.

  Thereafter, the user traced the first few letters of the name of the destination with the fingertip of one hand and the palm of the other hand in order to make the smartphone read out the target value candidates. Thereby, the process (refer FIG. 3A) of step S10-step S30 was performed, and the candidate of the several destination was read aloud by the smart phone. Therefore, the user traced the number corresponding to the desired candidate with the fingertip of one hand to select the target value. Thereby, the process of step S10-step S30 (refer FIG. 3A) was performed, the target value was selected, and navigation started. As a result, the user was able to listen to the voice guidance to the target value using only the hand gesture and the earphone without taking out the smartphone.

(4) The user thought to adjust the volume of the TV through the voice dialogue robot, but gives the voice dialogue robot instructions such as “larger”, “no, a little smaller”, “a little bigger”. Felt annoying. Therefore, the user gives an instruction “change volume” to the voice interactive robot, and then the information input device 1 controls the voice interactive robot. In order to adjust the volume of the television, the user traces the palm of the other hand with the fingertip of one hand so that the volume knob is rotated clockwise or counterclockwise by a predetermined rotation amount. Thereby, the process of step S10-step S30 (refer FIG. 3A) was performed, and the volume according to the said rotation amount was adjusted with the voice dialogue robot.

  As described above, by using the information input device 1, it is possible to easily give an instruction of an analog amount that is difficult in voice dialogue. In addition, as an analog amount, in addition to the volume of a device such as a television, the brightness of a light / screen, position designation, the wind direction / volume / strength of an air conditioner, the screen direction of a device such as a television, and the like may be exemplified. it can.

(5) The user operates a personal computer with a graphical user interface. Assume that the eye tracking function is added to the personal computer. Instead of moving the pointer with the mouse, the user looks at the mouse pointer. Then, the shape of the pointer changed, and the personal computer gave feedback to the user that the user was selecting the pointer as an operation target. Therefore, the user traces on the desk in the direction in which he / she wants to move. The mouse pointer moves the indicated distance in that direction. Also, look at the file icon on the toolbar of an application. Instead of clicking with the mouse, tap the desk with your finger to open the file menu. Look at the command you want to execute in the menu. Then, instead of clicking the mouse, tap the desk with your finger to execute the command. In this way, a graphical object is selected as an operation target by viewing, and an instruction is given with a finger. Such a user interface is quicker and more natural than operating a mouse because it looks at an object and operates with a finger.

(6) The user performed a predetermined hand gesture after looking at the air conditioner that is the target device of the eye tracking device. Thereby, the process of step S10-step S30 (refer FIG. 3A) is performed, and the "selection" command is transmitted to the apparatus which controls household appliances, and came to be able to perform on-off control of an air conditioner. In addition, the user performs a predetermined hand gesture to turn on the air conditioner. Thereby, the process of step S10-step S30 (refer FIG. 3A) was performed, the "ON" command was transmitted to the air conditioner, and the air conditioner started.

(7) The user performs a predetermined hand gesture in order to take a zoom photograph of a distant object using a line-of-sight tracking device with a camera. Thereby, the process of step S10-step S30 (refer FIG. 3A) was performed, and the zoom command was transmitted to the gaze tracking camera. Since the line of sight moved in an unstable manner and the three-dimensional position of the subject to be imaged was not determined, the user performed a predetermined hand gesture to fix the three-dimensional position. Thereby, the process of step S10-step S30 (refer FIG. 3A) was performed, the position fixing command was transmitted to the gaze tracking camera, and the three-dimensional position of imaging | photography object was fixed.

  However, since the three-dimensional position of the photographing target is not the intended position, the user performs a predetermined hand gesture. Thereby, the process of step S10-step S30 (refer FIG. 3A) was performed, and the distance and direction to move were designated. Then, the user traced the palm of the other hand up and down with the fingertip of one hand in order to adjust the three-dimensional position of the object to be imaged. Thereby, the process of step S10-step S30 (refer FIG. 3A) was performed, and the three-dimensional position of imaging | photography object moved up and down. Thereafter, the user took a zoom photograph and thought to end the shooting, and performed a predetermined hand gesture. As a result, the processing of steps S10 to S30 (see FIG. 3A) was executed, a “zoom imaging” command was transmitted to the line-of-sight tracking camera, and then an end command was transmitted.

  The above three-dimensional position indication can also be used for the drone controller. As shown in the above usage example, by using the information input device 1 as a user interface, it is possible to control an arbitrary device such as a robot, a drone, or a head mounted display, as well as a voice interactive device and a line-of-sight tracking device. be able to.

  In the present embodiment, the movement state of the fingertip of one hand in the palm of the other hand is estimated during the plane contact period in which the fingertip of one hand of the user is in plane contact with the palm of the other hand. More specifically, when the user performs an operation of drawing a character or the like while tracing the palm of the other hand (the surface of the object) with the fingertip (contact body) of one hand, the fingertip of one hand moves to the other hand. In the plane contact period in which the palm is in plane contact, the movement state of the fingertip that moves the palm along the trajectory of the character or the like is estimated. Then, operation information based on the estimation result is generated, and the terminal can be operated based on the operation information.

  The flat thing to contact is not limited to the palm of the other hand, and may be anything such as the outside of the thigh of the trousers or the flat part of the bag. An operation of writing on a certain plane with one hand can be performed intuitively, and it is not always necessary to direct a line of sight to the surface of the object or use both hands, and does not occupy both hands and eyes.

  In addition, the user can perform the above operation without being seen by a third party by moving the fingertip while hiding the fingertip with a palm, thereby protecting the privacy.

  In addition, it is easier to detect the position, moving direction, and movement of the fingertip that moves the palm (two-dimensional plane) than when detecting the position, moving direction, and movement of the fingertip that moves in the three-dimensional space. Gesture recognition accuracy can be increased. Furthermore, since the gesture recognition accuracy is increased, it is possible to recognize many types of gestures, and provide a user-friendly technique.

  FIG. 7 is another diagram for explaining the contents of the process executed by the state estimation unit 123. As shown in FIG. 7, in order to specify the position of the fingertip moving in the three-dimensional space, it is necessary to search for a plurality of correct patterns. On the other hand, when the position of the fingertip that moves on the two-dimensional plane is specified, the correct answer pattern is uniquely determined, so that high gesture recognition accuracy can be realized.

  In the present embodiment, the plane contact period is specified based on the contact start time Tcs and the contact end time Tce. In this plane contact period, the movement period is specified based on the movement start time Tms and the movement end time Tme. Is done. Since the movement period is a period in which the fingertip of one hand of the user is moving the palm (two-dimensional plane) of the other hand, the movement state of the user's fingertip during the movement period is estimated, thereby providing a high gesture. Recognition accuracy can be realized.

  Moreover, in this embodiment, the 1st detection part 111 detects the vibration which generate | occur | produced when a fingertip contacts a palm, and the 2nd detection part 112 detects the motion of a fingertip. That is, in this embodiment, the plane contact period is specified based on the vibration waveform signal V1 output from the first detection unit 111, the acceleration waveform signal V2a output from the second detection unit 112, and the angular velocity waveform signal V2b. When the fingertip of one hand of the user makes a plane contact with the palm of the other hand, the vibration at that time appears rapidly in the vibration waveform as a peak. Further, when the fingertip of one hand of the user moves, the acceleration, angular velocity, and the like at that time quickly appear as peaks in the acceleration waveform and the angular velocity waveform. Therefore, according to the present embodiment, the plane contact period can be accurately obtained based on the peak of each waveform.

  Further, in the present embodiment, the state detection unit 123 determines that the fingertip of one hand of the user starts the plane contact with the palm of the other hand from the plane contact period and the fingertip of the one hand is the above. A movement period excluding a period near each of the contact end times Tce for ending the flat contact with the palm of the other hand, and the movement state based on the detection signal of the second detection unit 112 in the movement period Is estimated. Therefore, an error is included in the specified plane contact period due to the influence of the shock wave generated at the contact start time Tcs and the contact end time Tce, and the accuracy of the estimation result of the movement state of the fingertip of the one hand is prevented from being lowered. be able to.

  Further, in the present embodiment, the contact detection unit 122 detects both the detection signal (vibration waveform sample V1) of the first detection unit 111 and the detection signal (acceleration waveform sample V2a and angular velocity waveform sample V2b) of the second detection unit 112. A period in which the fluctuation range exceeds a predetermined threshold is specified as a plane contact period. Therefore, the plane contact period can be obtained more accurately.

  In the present embodiment, the fingertip of one hand of the user moves on a substantially two-dimensional plane (the palm of the other hand of the user). Therefore, the movement state of the fingertip can be accurately estimated as compared with the case where the fingertip moves on the surface of the object having the unevenness. Therefore, the gesture recognition accuracy can be further increased.

  The present invention is not limited to the above-described embodiment, and can be variously modified within the scope of the present invention.

  In the above embodiment, a Web server may be used based on the operation information from the operation information generation unit 124 using a Web server instead of the terminal 20. For example, the characters indicated by the operation information may be stored in a Web server as text, and the text may be provided to a notepad application, a search application, or the like in response to a request from the user terminal.

  In the above embodiment, the vibration sensor 111a may be omitted, and the detection unit 111 may be configured with only a motion sensor (for example, an acceleration sensor, each speed sensor, or the like). According to the experiment of the present inventor, it has been clarified that the detection signal of the motion sensor includes a vibration component of frictional vibration in the vicinity of 10 to 40 Hz. Therefore, it is possible to specify the frame in the plane contact period by frequency-resolving the detection signal of the motion sensor and detecting the vibration component of the frictional vibration superimposed on the detection signal. In this case, the first detection unit 111 can be omitted, and the configuration of the detection unit 110 can be simplified.

  In the above embodiment, the first detection unit 111 is configured by the vibration sensor 111a, but may be configured by a piezoelectric sensor, a capacitor microphone, or the like. Or you may comprise the 1st detection part 111 combining the said some sensor.

  In the said embodiment, although a user's fingertip was illustrated as a contact body, the example of a contact body is not limited to this. For example, a writing instrument such as a ballpoint pen or a pencil may be used as a contact body, and a motion sensor and a vibration sensor may be fixed to these writing instruments. In this aspect, when the user writes a predetermined character on a notebook or the like (the surface of the object) using the writing instrument, the processing of steps S10 to S30 is executed (see FIG. 3A), and the predetermined character is handwritten. It is recognized in the character recognition process (FIG. 3B: step SA170). Therefore, for example, the present invention can be applied to educational toys and the like by digitizing the recognized predetermined characters and displaying them on a display or reading them as voices.

  Further, since the tip portion (for example, the pen tip) of the writing instrument as described above is harder than the fingertip or the like, the friction vibration is easily transmitted to the detection unit 110 when the surface of the object is rubbed with the writing instrument. Therefore, in the contact detection process (FIG. 3B: step SA110), friction vibration can be easily detected, and the plane contact period can be specified with high accuracy.

  In addition, when the motion sensor is fixed to the writing instrument, the distance between the contact body and the motion sensor is shorter than when the motion sensor is fixed to the wrist. Therefore, the distortion detected in step SB155 (see FIG. 3C) is reduced, and the accuracy of the estimation result of the moving state of the contact body by the state estimation unit 123 can be increased.

DESCRIPTION OF SYMBOLS 1 ... Gesture input system 10 ... Information input device 100 ... Signal processing device 110 ... Detection part 111 ... 1st detection part
111a ... Vibration sensor 112 ... Second detector
112a Acceleration sensor
112b ... Angular velocity sensor 120 ... Signal processing unit 121 ... Reception processing unit 122 ... Contact detection unit 123 ... State estimation unit 124 ... Operation information generation unit 130 ... Communication unit 30 ..Mounting part 20 ... Terminal

Claims (6)

A reception processing unit that receives a detection signal from at least one of the first detection unit and the second detection unit;
A contact detection unit that identifies a plane contact period in which the contact body is in plane contact with the object based on at least one detection signal of the first detection unit and the second detection unit;
A state estimation unit that estimates a moving state of the contact body on the surface of the object based on a detection signal of the second detection unit in the planar contact period;
An information input device comprising: a signal processing unit including: an operation information generation unit configured to generate operation information for operating an operation target terminal based on an estimation result of the state estimation unit. The first detection unit detects vibration generated when the contact body makes a flat contact with the object,
The information input device according to claim 1, wherein the second detection unit detects a movement of the contact body.   The state detection unit includes a period near each of a contact start time when the contact body starts planar contact with the object and a contact end time when the contact body finishes planar contact with the object from the plane contact period. 3. The information input device according to claim 1, wherein a movement period excluding the movement period is specified, and the movement state is estimated based on a detection signal of the second detection unit during the movement period.   The information input device according to claim 1, wherein the surface of the object has a substantially planar shape. A reception processing step of receiving a detection signal from at least one of the first detection unit and the second detection unit;
A contact detection step for identifying a plane contact period in which the contact body is in plane contact with the object based on at least one detection signal of the first detection unit and the second detection unit;
A state estimation step of estimating a moving state of the contact body on the surface of the object based on a detection signal of the second detection unit in the planar contact period;
An information input method comprising: a signal processing step including: an operation information generation step for generating operation information for operating an operation target terminal based on an estimation result of the state estimation unit. A reception processing unit that receives a detection signal from at least one of the first detection unit and the second detection unit;
A contact detection unit that identifies a plane contact period in which the contact body is in plane contact with the object based on at least one detection signal of the first detection unit and the second detection unit;
A state estimation unit that estimates a moving state of the contact body on the surface of the object based on a detection signal of the second detection unit in the planar contact period;
An information input device having a signal processing unit having an operation information generating unit for generating operation information for operating the operation target terminal based on the estimation result of the state estimating unit;
A terminal that operates based on the operation information transmitted from the information input device;
An information input system comprising:

Images