JP7561319B2 - Motion recognition system, method, and program - Google Patents

Description

The present invention relates to a motion recognition system, method, and program.

Conventionally, cameras and sensors are installed on production lines in factories, etc., and the movements of workers are analyzed and evaluated based on the information on the movements of the workers obtained by these cameras and sensors. For example, in the following Patent Document 1, the presence or absence of non-standard work that deviates from standard work is determined based on information obtained by motion capture, etc.

WO 2018/131630

Incidentally, when evaluating a worker's proficiency, it is possible to focus on tasks in which the worker performs different actions with their right and left hands. For example, in the process of tightening a screw with a screwdriver, the screwdriver is used with the right hand, which is the dominant hand, while the left hand grasps a part and does not use the screwdriver. When evaluating such tasks, it is preferable to recognize and evaluate the actions performed independently by the left and right hands separately.

However, in Patent Document 1, the movement during the task is perceived as a whole and the entire movement is recognized as the subject of evaluation, and no consideration is given to the individual recognition of movements performed independently by the left and right parts of the body.

Here, in Patent Document 1, when analyzing the movements of a worker, the work content of the worker is recognized by comparing the work content (learned data) that has been learned in advance in a learning content recording unit with the work content of the worker. For example, to accurately recognize the movements of a process for tightening a screw with a screwdriver based on such learning data, it is necessary to learn data of left-handed workers in addition to data of right-handed workers. However, since the proportion of left-handed people is low, it is difficult to collect data on left-handed workers.

The present invention provides a motion recognition system, method, and program that can improve the accuracy of individually recognizing motions performed independently by the left and right parts of a worker.

The motion recognition system according to one aspect of the present disclosure includes an acquisition unit that acquires motion information including elapsed time of work, individual motions to be recognized, and skeletal data of a worker; a conversion unit that uses an axis formed by connecting coordinates corresponding to a plurality of predetermined reference parts among the coordinates corresponding to each of a plurality of body parts included in the skeletal data as a reference axis and converts the coordinates corresponding to each of the body parts into coordinates that are linearly symmetrical with respect to the reference axis; and a learning unit that trains a model for motion recognition that outputs information indicating a motion of a worker that corresponds to any of the individual motions based on the motion information converted by the conversion unit.

According to this aspect, for example, among the coordinates corresponding to each of the multiple body parts included in the skeletal data of a right-handed worker, an axis formed by connecting the coordinates corresponding to the neck and waist, which are the reference parts, is used as a reference axis, and the coordinates corresponding to each body part are converted to coordinates that are linearly symmetrical with respect to the reference axis, thereby generating skeletal data for a left-handed worker. Then, a learning model for motion recognition can be trained using this skeletal data for a left-handed worker.

In the above aspect, the learning unit may further train the model based on the motion information acquired by the acquisition unit.

According to this aspect, a model for motion recognition can be further trained based on motion information based on time-series information, such as video of the motions of right-handed or left-handed workers.

In the above aspect, the acquisition unit may further acquire time-series information regarding the movements of the worker to be recognized, input skeletal data corresponding to the time-series information acquired by the acquisition unit into a trained model trained by the learning unit, and further include a motion recognition unit that recognizes the movements of the right hand and the left hand of the worker to be recognized based on information indicating the movements of the worker that correspond to any of the individual movements output from the trained model.

According to this aspect, a trained model trained using skeletal data for a right-handed worker and skeletal data for a left-handed worker can be used to recognize whether the movements of the right hand and left hand of the worker to be recognized correspond to any of the individual movements.

In the above aspect, the reference site may include at least one of the nose, neck, and lower back.

According to this embodiment, the reference axis can be formed in accordance with the central axis of the body facing directly.

A method according to another aspect of the present disclosure is a method executed by a processor, and includes acquiring motion information including the elapsed time of the task, the individual motion to be recognized, and skeletal data of the worker, converting the coordinates corresponding to each of the body parts into coordinates that are linearly symmetrical with respect to the reference axis by using an axis formed by connecting the coordinates corresponding to a plurality of predetermined reference parts among the coordinates corresponding to each of the plurality of body parts included in the skeletal data as a reference axis, and learning a model that outputs information indicating the motion of the worker that corresponds to any of the individual motions based on the converted motion information.

According to this aspect, for example, among the coordinates corresponding to each of the multiple body parts included in the skeletal data of a right-handed worker, an axis formed by connecting the coordinates corresponding to the neck and waist, which are the reference parts, is used as a reference axis, and the coordinates corresponding to each body part are converted to coordinates that are linearly symmetrical with respect to the reference axis, thereby generating skeletal data for a left-handed worker. Then, a learning model for motion recognition can be trained using this skeletal data for a left-handed worker.

A program according to another aspect of the present disclosure causes a computer to function as an acquisition unit that acquires motion information including the elapsed time of a task, an individual motion to be recognized, and skeletal data of a worker, a conversion unit that uses an axis formed by connecting coordinates corresponding to a plurality of predetermined reference parts among the coordinates corresponding to each of a plurality of body parts included in the skeletal data as a reference axis and converts the coordinates corresponding to each of the body parts into coordinates that are linearly symmetrical with respect to the reference axis, and a learning unit that learns a model that outputs information indicating a motion of a worker that corresponds to one of the individual motions based on the motion information after conversion by the conversion unit.

According to this aspect, for example, among the coordinates corresponding to each of the multiple body parts included in the skeletal data of a right-handed worker, an axis formed by connecting the coordinates corresponding to the neck and waist, which are the reference parts, is used as a reference axis, and the coordinates corresponding to each body part are converted to coordinates that are linearly symmetrical with respect to the reference axis, thereby generating skeletal data for a left-handed worker. Then, a learning model for motion recognition can be trained using this skeletal data for a left-handed worker.

The present invention provides a motion recognition system, method, and program that can improve the accuracy of individually recognizing motions performed independently by the left and right parts of a worker.

1 is a diagram illustrating an example of an overview of an action recognition system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a functional configuration of an action recognition system and an action recognition device. FIG. 4 is a diagram illustrating an example of action information stored in the action recognition device. 13 is a diagram illustrating an example of skeletal data in which a reference axis is formed. FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the action recognition device. 10 is a flowchart illustrating an example of an operation of the action recognition device. 7 is a flowchart for explaining the procedure of the conversion process shown in FIG. 6 .

Below, an embodiment of one aspect of the present invention (hereinafter, referred to as "this embodiment") will be described with reference to the drawings. Note that in each drawing, parts with the same reference numerals have the same or similar configurations.

§1 Application Example First, an example of a situation in which the present invention is applied will be described with reference to Fig. 1. In the action recognition system 100 according to this embodiment, the action of a worker A performed in a certain working area R is captured by image sensors 20a, 20b, and 20c, and the action recognition device 10, which acquires the captured video, recognizes the actions of the right and left hands of the worker A using a trained model.

The trained model is a model for motion recognition that is trained to input motion information including skeletal data of a right-handed worker and skeletal data generated for a left-handed worker, and output information indicating the motion of the worker. The skeletal data for a left-handed worker that is input during training is generated from skeletal data of a right-handed worker, as described below.

This allows the motion recognition model to learn skeletal data for left-handed workers in addition to skeletal data for right-handed workers. Therefore, the motion recognition device 10 according to this embodiment can improve the accuracy of individually recognizing motions performed independently by the right and left hands of a worker.

§2 Configuration example [Functional configuration]
Next, an example of the functional configuration of the action recognition system 100 and the action recognition device 10 according to the present embodiment will be described with reference to FIG. The action recognition device 10 includes the following functions: The system includes, for example, an acquisition unit 11, a conversion unit 12, a learning unit 13, a motion recognition unit 14, and a storage unit 19. The storage unit 19 stores, for example, a video 19a, motion information 19b, and a trained model 19c. do.

In this embodiment, the motion recognition device 10 is described as having a function of learning a model for motion recognition (learning mode) and a function of recognizing the motion of a worker (motion recognition mode), but each function may be distributed and provided in an independent, separate device.

The details of each functional configuration of the action recognition system 100 and the action recognition device 10 are explained below.

<Image sensor>
The image sensor 20 is, for example, a general-purpose camera, and captures a video including a scene in which the worker A is performing an action in the working area R. The image sensor 20 has, for example, a detection unit as a functional configuration. The detection unit detects the action of the worker A, and outputs a video showing the action as time-series information.

Here, the time series information is not limited to video. For example, it may be information about coordinates indicating the movements of worker A measured by a motion capture device that can be provided in place of the image sensor 20.

Each of the image sensors 20a, 20b, 20c is positioned so that it can capture the entire work area R and the entire body of the worker A. In this case, for example, each of the image sensors 20a, 20b, 20c may be positioned so that it can capture the entire work area R and the entire body of the worker A, or each of the image sensors 20a, 20b, 20c may be positioned so that it can capture a portion of the work area R and the worker A, and the respective videos can be combined to cover the entire work area R and the entire body of the worker A. Also, each of the image sensors 20a, 20b, 20c may capture the work area R and the worker A at different magnifications. It is not necessary to provide three image sensors 20, and it is sufficient to provide at least one or more.

<Acquisition Department>
The acquisition unit 11 acquires time-series information (video in this embodiment) related to the actions performed by the worker A from the image sensor 20. The time-series information acquired by the acquisition unit 11 is transmitted to the storage unit 19 and stored as a video 19a. The acquisition unit 11 also acquires the video 19a stored in the storage unit 19.

The acquisition unit 11 extracts skeletal data indicating the movement of the worker's skeleton from the images of the video 19a. The skeletal data can be represented by coordinates (x, y) corresponding to multiple body parts. In this embodiment, the case where the coordinates are two-dimensional coordinates (x, y) is described, but the same can be applied to the case where the coordinates are three-dimensional coordinates (x, y, z). Furthermore, information indicating the accuracy of the coordinate values may be added in addition to the coordinate values.

The skeletal data is transmitted to the storage unit 19 and stored as part of the motion information 19b. The acquisition unit 11 also acquires the motion information 19b stored in the storage unit 19.

Movement information 19b will be described with reference to FIG. 3. Movement information 19b is configured to include, as data items, an elapsed time item, a right hand movement item, a left hand movement item, and a skeleton data item, for example. The elapsed time item stores the elapsed time from the reference time, which is the time when the first process of all the processes to be worked on was started. The interval of the elapsed time can be set arbitrarily, and may be set, for example, in units of frames of a video, or may be set at a predetermined interval such as every second.

The right hand movement item stores information indicating which of the individual movements to be recognized (described below) the right hand movement corresponds to. The left hand movement item stores information indicating which of the individual movements to be recognized corresponds to the left hand movement. The information stored in the right hand movement item and left hand movement item can be registered, for example, by referring to a video, checking the right and left hand movements at each elapsed time, and inputting the confirmed movement details.

Examples of individual actions include grasping, transporting, and adjusting. Grasping is defined as the action of moving the hand to the work object and grabbing it, transport is defined as the action of moving the work object to the desired location, and adjustment is defined as the action of moving the work object to the target state.

Each individual action includes, for example, the following actions: Gripping includes the action of grasping the base and cover in the case fitting process, and the action of grasping the electric screwdriver in the screw tightening process. Transporting includes the action of carrying the cover to the base in the case fitting process, and the action of carrying the electric screwdriver to the screw threads in the screw tightening process. Adjusting includes the action of engaging the hook in the case fitting process, and the action of tightening the screw with the electric screwdriver in the screw tightening process.

The skeleton data item of the motion information 19b stores skeleton data extracted from the video corresponding to the elapsed time.

The motion information in the first row shown in Figure 3 is based on the video at one second after the start of the first process. This motion information contains information indicating that the motion of the worker's right hand does not fall into any of the individual motions and that the motion of the worker's left hand falls into the individual motion of grasping, as well as skeletal data extracted from the video at that time. The motion information in the last row shown in Figure 3 is based on the video at two minutes and fifty-three seconds after the start of the first process. This motion information contains information indicating that the motions of the worker's right and left hands fall into the individual motion of adjusting, as well as skeletal data extracted from the video at that time.

<Conversion section>
The conversion unit 12 shown in Fig. 2 is a function in the learning mode. The conversion unit 12 executes a conversion process for generating skeletal data for a left-handed worker based on skeletal data for a right-handed worker. The conversion process is a process for converting coordinates corresponding to each of a plurality of body parts included in the skeletal data into coordinates that are line-symmetric with respect to a reference axis described later.

The reference axis can be formed by connecting the coordinates corresponding to a number of predetermined reference body parts. For example, the nose, neck, waist, etc., which can form the central axis of the body facing the image sensor, etc., can be set as the reference body parts. In FIG. 4, when the neck and waist are set as the reference body parts, a straight line connecting coordinate a corresponding to the neck and coordinate b corresponding to the waist is formed as the reference axis s.

By converting each coordinate of the skeletal data shown in FIG. 4 into a coordinate that is linearly symmetric with respect to the reference axis s, for example, the coordinates corresponding to the right hand and the right leg are converted into coordinates near the left hand and the left leg, and the coordinates corresponding to the left hand and the left leg are converted into coordinates near the right hand and the right leg. In this way, skeletal data for a left-handed worker can be generated based on the skeletal data of a right-handed worker.

The conversion unit 12 stores the generated skeletal data for a left-handed worker in the skeletal data item of the corresponding motion information 19b, and swaps the contents of the right-hand motion item and the left-hand motion item of that motion information 19b.

Here, the conversion unit 12 may further include an adjustment unit that performs an adjustment process to adjust the skeletal data extracted from the video. The adjustment unit included in the conversion unit 12 performs the adjustment process before performing the above-mentioned conversion process. This adjustment process includes, for example, a time series complementation process for skeletal data, a normalization process for height (physique), a time series smoothing process for skeletal data, a shift process for skeletal data, and a noise addition process. Each process included in the adjustment process is described below.

The time series completion process of skeletal data is a process that completes missing data that occurs in skeletal data based on other skeletal data located before and after in time. Missing data may occur, for example, in areas that cannot be estimated because they are hidden by the worker's posture, etc.

Height (physique) normalization is a process that normalizes skeletal data based on body shape to absorb physique differences that occur between men and women, for example. As a normalization process, for example, normalized data can be generated by dividing the skeletal data by the length of the torso (for example, the length from the nose to the waist).

The time series smoothing process of the skeletal data is a process that removes noise from changes in the time axis direction of the skeletal data. As a smoothing process, for example, Gaussian filter processing can be performed on the skeletal data to generate smoothed data.

Skeletal data shift processing is a process that aligns the origin of skeletal data and standardizes the starting point to reduce variation in movement and make movement characteristics easier to recognize. For example, the entire skeletal data is translated so that the neck joint is located at the origin, and shifted skeletal data is generated.

The noise addition process is a process that virtually increases the skeletal data by adding noise to the skeletal data. The noise added to the skeletal data can be generated, for example, by randomly generating values within a range that can be used as skeletal data.

<Study Section>
The learning unit 13 shown in FIG. 2 is a function in the learning mode. The learning unit 13 generates (learns) a model for motion recognition that outputs information indicating a motion of a worker corresponding to any of the individual motions based on the motion information 19b. The motion information 19b to be learned by the model includes motion information 19b including skeletal data extracted from a video of a right-handed worker, and motion information 19b including skeletal data for a left-handed worker generated by the above-mentioned conversion process. The order in which each piece of motion information 19b is learned by the model does not matter. Note that it is preferable to perform the above-mentioned adjustment process on the skeletal data of a right-handed worker before learning it by the model.

The model trained by the training unit 13 is transmitted to the memory unit 19 and stored as a trained model 19c.

<Action Recognition Unit>
The motion recognition unit 14 is a function in the motion recognition mode. The motion recognition unit 14 inputs skeletal data extracted from a video 19a of a worker to be recognized to a trained model 19c, and recognizes the motions of the right and left hands of the worker based on information indicating the motion of the worker corresponding to any of the individual motions output from the trained model 19c. When recognizing the motion of the worker, for example, a known motion recognition method such as ST-GCN (Spatial Temporal Graph Convolutional Networks) can be used.

When the skeletal data extracted from the video 19a is input to the trained model 19c, the skeletal data may be subjected to the above-mentioned time series complementation process, height (physique) normalization process, time series smoothing process, and shift process.

[Hardware configuration]
Next, an example of the hardware configuration of the action recognition device 10 according to the present embodiment will be described with reference to FIG. 5. The action recognition device 10 includes a CPU (Central Processing Unit) 10a corresponding to a calculation device, a RAM (Random Access Memory) 10b corresponding to a storage unit 19, a ROM (Read Only Memory) 10c corresponding to the storage unit 19, a communication device 10d, an input device 10e, and a display device 10f. These components are connected to each other via a bus so as to be able to transmit and receive data to each other. Note that, although a case where the action recognition device 10 is configured by one computer will be described in the present embodiment, the action recognition device 10 may be realized using a plurality of computers.

The CPU 10a executes programs stored in the RAM 10b or ROM 10c and functions as a control unit that performs calculations and processing of data. The CPU 10a receives various input data from the input device 10e or communication device 10d, and displays the results of calculations on the input data on the display device 10f or stores them in the RAM 10b or ROM 10c.

RAM 10b is composed of, for example, a semiconductor memory element, and stores rewritable data. ROM 10c is composed of, for example, a semiconductor memory element, and stores readable but non-rewritable data.

The communication device 10d is an interface that connects the action recognition device 10 to an external device. The communication device 10d is connected to the image sensor 20 via a communication network such as a LAN (Local Area Network) or the Internet, for example, and receives video from the image sensor 20.

The input device 10e is an interface that accepts data input from a user and may include, for example, a keyboard, a mouse, and a touch panel.

The display device 10f is an interface that visually displays the results of calculations performed by the CPU 10a, and can be configured, for example, with an LCD (Liquid Crystal Display).

The program may be provided by being stored in a computer-readable storage medium such as RAM 10b or ROM 10c, or may be provided via a communication network connected by communication device 10d. In the action recognition device 10, the CPU 10a executes the program to perform the operations of the acquisition unit 11, conversion unit 12, learning unit 13, and action recognition unit 14 shown in FIG. 2. Note that these physical configurations are merely examples and do not necessarily have to be independent configurations. For example, the action recognition device 10 may be provided with an LSI (Large-Scale Integration) in which the CPU 10a is integrated with the RAM 10b and ROM 10c.

6 is a flowchart showing an example of the operation of the action recognition device 10 according to the present embodiment. This operation is performed when skeletal data for a left-handed worker is generated based on skeletal data of a right-handed worker, and a model for action recognition is trained using the generated skeletal data.

First, the acquisition unit 11 acquires from the storage unit 19 the motion information 19b generated based on the learning video 19a (step S101). This motion information 19b is generated based on a video of a right-handed worker. In addition, the skeleton data item of the motion information 19b stores skeleton data of the right-handed worker corresponding to the video.

Next, the conversion unit 12 executes a conversion process to convert each coordinate included in the skeletal data into coordinates that are line-symmetric with respect to the reference axis (step S102). The procedure for this conversion process will be described later.

Then, the learning unit 13 trains a model for action recognition that outputs information indicating the action of the worker that corresponds to one of the individual actions, based on the action information 19b that includes the skeletal data after conversion in step S102 and in which the contents of the right hand action item and the left hand action item have been swapped (step S103). Then, this operation ends.

The conversion process performed in step S102 above will be described with reference to FIG. 7.

First, the acquisition unit 11 extracts skeletal data for one body per frame from an image of a video 19a having, for example, F frames (step S201). In other words, the acquisition unit 11 extracts skeletal data for F bodies.

Then, the conversion unit 12 converts the coordinates corresponding to each of the multiple body parts included in the skeletal data of one body into coordinates that are line-symmetric with respect to the reference axis s (step S202). The converted coordinates are stored in the skeletal data item of the motion information 19b.

Next, the conversion unit 12 determines whether the coordinates of the skeletal data of all F bodies have been converted (step S203). If the determination is NO, the process proceeds to step S202, whereas if the determination is YES, the conversion process is terminated.

As described above, the motion recognition device 10 according to this embodiment can generate skeletal data for a left-handed worker by using an axis formed by connecting the coordinates corresponding to the reference parts, namely the neck and waist, among the coordinates corresponding to each of the multiple body parts included in the skeletal data of a right-handed worker as a reference axis, and converting the coordinates corresponding to each body part into coordinates that are linearly symmetrical with respect to the reference axis. Then, a learning model for motion recognition can be trained using this skeletal data for a left-handed worker. This can improve the accuracy of individually recognizing motions performed independently by the left and right parts of the worker.

The present invention is not limited to the above-described embodiment, and can be implemented in various other forms without departing from the spirit of the present invention.

For example, in the above-described embodiment, skeletal data for a left-handed worker is generated based on skeletal data for a right-handed worker, but this is not limited to the above. Skeletal data for a right-handed worker may also be generated based on skeletal data for a left-handed worker.

In the above-described embodiment, a model for motion recognition is trained using motion information 19b generated based on a video of a right-handed worker and motion information 19b including skeletal data for a left-handed worker generated by a conversion process, but this is not limited to this. A model for motion recognition may be trained using motion information 19b generated based on a video of a left-handed worker and motion information 19b including skeletal data for a right-handed worker generated by a conversion process.

Embodiments of the present invention may also be described as in the following appendices. However, the embodiments of the present invention are not limited to the forms described in the appendices below. Furthermore, the embodiments of the present invention may be in the form of a substitution or combination of the descriptions between the appendices.

[Appendix 1]
an acquisition unit (11) for acquiring motion information (19b) including an elapsed time of a task, an individual motion to be recognized, and skeletal data of a worker; a conversion unit (12) for converting the coordinates corresponding to each of a plurality of body parts included in the skeletal data into coordinates that are linearly symmetric with respect to the reference axis, the coordinates being formed by connecting coordinates corresponding to a plurality of predetermined reference parts among the coordinates corresponding to each of a plurality of body parts included in the skeletal data;
a learning unit (13) that learns a model that outputs information indicating a motion of the worker that corresponds to any one of the individual motions based on the motion information (19b) after conversion by the conversion unit (12) as a model for motion recognition;
An action recognition system (100) comprising:

[Appendix 2]
The learning unit (13) further learns the model based on the operation information (19b) acquired by the acquisition unit (11).
2. The action recognition system (100) according to claim 1.

[Appendix 3]
The acquisition unit (11) further acquires time-series information (19a) regarding the motion of the worker to be recognized, inputs skeletal data corresponding to the time-series information (19a) acquired by the acquisition unit (11) into a trained model (19c) trained by the learning unit (13), and configures a motion recognition unit (14) that recognizes the motions of the right hand and the left hand of the worker to be recognized based on information indicating the motion of the worker corresponding to any of the individual motions output from the trained model (19c).
The action recognition system (100) according to claim 2, further comprising:

[Appendix 4]
The reference site includes at least any one of the nose, the neck, and the lower back.
4. The action recognition system (100) according to any one of claims 1 to 3.

[Appendix 5]
1. A processor-implemented method, comprising:
Acquiring motion information (19b) including an elapsed time of the task, an individual motion to be recognized, and skeletal data of the worker;
a reference axis is defined as an axis formed by connecting coordinates corresponding to a plurality of predetermined reference parts among the coordinates corresponding to each of a plurality of body parts included in the skeletal data, and the coordinates corresponding to each of the body parts are transformed into coordinates that are linearly symmetric with respect to the reference axis;
learning a model that outputs information indicating a motion of the worker corresponding to any one of the individual motions based on the motion information (19b) after the conversion;
The method includes:

[Appendix 6]
Computer,
an acquisition unit (11) for acquiring motion information (19b) including the elapsed time of a task, an individual motion to be recognized, and skeletal data of a worker;
a conversion unit (12) for converting the coordinates corresponding to each of a plurality of body parts included in the skeletal data into coordinates that are line-symmetric with respect to the reference axis, the axis being formed by connecting the coordinates corresponding to a plurality of predetermined reference parts among the coordinates corresponding to each of a plurality of body parts included in the skeletal data;
a learning unit (13) that learns a model that outputs information indicating a motion of the worker that corresponds to any one of the individual motions, based on the motion information (19b) after conversion by the conversion unit (12);
A program that functions as a

10...motion recognition device, 10a...CPU, 10b...RAM, 10c...ROM, 10d...communication device, 10e...input device, 10f...display device, 11...acquisition unit, 12...conversion unit, 13...learning unit, 14...motion recognition unit, 19...storage unit, 19a...video, 19b...motion information, 19c...trained model, 20a, 20b, 20c...image sensor, 100...motion recognition system, A...worker, R...work area

Claims (4)

an acquisition unit that acquires from a storage unit motion information generated based on time-series information on a motion of a worker in a task performed by the worker, the motion information storing an elapsed time of the task performed by the worker, a motion performed with the right hand of the worker, a motion performed with the left hand of the worker , and skeletal data of the worker in association with each other;
a conversion unit which, for each elapsed time of the acquired motion information, converts the coordinates corresponding to each of the body parts into coordinates that are linearly symmetric with respect to the reference axis , using an axis formed by connecting coordinates corresponding to a plurality of reference parts that can form a predetermined central axis of the body among coordinates of a coordinate system representing the skeletal data corresponding to each of a plurality of body parts included in the skeletal data associated with the elapsed time as a reference axis, and stores the coordinates corresponding to each of the body parts into coordinates that are linearly symmetric with respect to the reference axis as the skeletal data based on the converted coordinates as the skeletal data to be stored in association with the elapsed time, and replaces the content of the motion performed with the right hand of the worker and the content of the motion performed with the left hand of the worker, which are stored in association with the elapsed time, to generate new motion information and store it in the storage unit ;
a learning unit that inputs the motion information stored in the storage unit as learning data , and learns a model that outputs information indicating a motion of the worker corresponding to either a motion performed with the right hand of the worker or a motion performed with the left hand of the worker , as a model for motion recognition;
a motion recognition unit that inputs the skeletal data generated based on time-series information on the motion of the target person in a task performed by the target person, into a trained model trained by the learning unit, and recognizes the motions of the right hand and the left hand of the target person based on information indicating the motion of the target person corresponding to either the motion performed with the right hand of the target person or the motion performed with the left hand of the target person output from the trained model;
A motion recognition system comprising: The reference site includes at least any one of the nose, the neck, and the lower back.
The action recognition system according to claim 1 . 1. A processor-implemented method, comprising:
acquiring from a storage unit motion information generated based on time-series information on motions of a worker in a task performed by the worker, the motion information storing an elapsed time of the task performed by the worker, a motion performed with the right hand of the worker, a motion performed with the left hand of the worker, and skeletal data of the worker in association with each other;
for each elapsed time of the acquired motion information, an axis formed by connecting the coordinates corresponding to a plurality of reference parts that can form a predetermined central axis of the body among the coordinates of a coordinate system representing the skeletal data corresponding to each of a plurality of body parts included in the skeletal data associated with the elapsed time is used as a reference axis, the coordinates corresponding to each of the body parts are converted to coordinates that are linearly symmetric with respect to the reference axis , the skeletal data based on the converted coordinates is used as the skeletal data to be stored in association with the elapsed time, and the content of the motion performed with the right hand of the worker and the content of the motion performed with the left hand of the worker, which are stored in association with the elapsed time, are exchanged to generate new motion information, and the storage unit stores the new motion information .
inputting the motion information stored in the storage unit as learning data, and training a model that outputs information indicating a motion of the worker corresponding to either a motion performed with the right hand of the worker or a motion performed with the left hand of the worker ;
inputting the skeletal data generated based on time-series information regarding the movements of the target person in a task performed by the target person, whose movements are to be recognized, into the trained model; and recognizing the movements of the right hand and the left hand of the target person based on information indicating the movements of the target person corresponding to either the movements performed with the right hand of the target person or the movements performed with the left hand of the target person output from the trained model;
The method includes: Computer,
an acquisition unit that acquires from a storage unit motion information generated based on time-series information on a motion of a worker in a task performed by the worker, the motion information storing an elapsed time of the task performed by the worker, a motion performed with the right hand of the worker, a motion performed with the left hand of the worker , and skeletal data of the worker in association with each other;
a conversion unit which, for each elapsed time of the acquired motion information, defines an axis formed by connecting coordinates corresponding to a plurality of reference parts that can form a predetermined central axis of the body among coordinates of a coordinate system representing the skeletal data corresponding to each of a plurality of body parts included in the skeletal data associated with the elapsed time as a reference axis, converts the coordinates corresponding to each of the body parts into coordinates that are linearly symmetric with respect to the reference axis , and stores the skeletal data based on the converted coordinates as the skeletal data to be stored in association with the elapsed time, and replaces the content of the motion performed with the right hand of the worker and the content of the motion performed with the left hand of the worker, which are stored in association with the elapsed time, to generate new motion information and store it in the storage unit ;
a learning unit that inputs the motion information stored in the storage unit as learning data and learns a model that outputs information indicating a motion of the worker corresponding to either a motion performed with the right hand of the worker or a motion performed with the left hand of the worker ;
a motion recognition unit which inputs the skeletal data generated based on time-series information on the motion of the target person in a task performed by the target person, into a trained model trained by the learning unit, and recognizes the motions of the right hand and the left hand of the target person based on information indicating the motion of the target person corresponding to either the motion performed with the right hand of the target person or the motion performed with the left hand of the target person output from the trained model;
A program that functions as a

Images