KR102494659B1 - Apparatus and method for estimating of 6d pose - Google Patents

Abstract

A 6D pose estimating device is a device for estimating an object pose using unsupervised learning, comprising: an external feature extractor for extracting external features by inputting RGB data of an image into a first deep learning model; a feature extractor including a geometric feature extractor extracting geometric features by inputting depth data of an image to a second deep learning model; a combination unit generating a feature map of the image by combining the extracted shape features and geometric features; And, an object pose estimator for estimating a 6D pose for an object corresponding to the image based on the generated feature map, wherein the second deep learning model is based on a user-defined problem (Pretext task) of the learned network Transfer learning based on weights.

Description

Apparatus and method for estimating 6D pose {APPARATUS AND METHOD FOR ESTIMATING OF 6D POSE}

The present invention relates to a 6D pose estimation apparatus and method.

In general, object pose estimation uses a deep learning model trained using label-included training data. That is, the pose of the object may be estimated by inputting the 3D data of the object to the learned deep learning model.

Good quality data is very important to train a deep learning model for 6D (x, y, z, roll, pitch, yaw) pose estimation. At this time, since random data must be generated in order to create a large data set, performance in a real environment may deteriorate. In particular, it is very difficult to create a data set to handle objects with six degrees of freedom (6DOF) using 3D data, and it takes much more time and money than 2D.

Unsupervised Learning is a type of machine learning that belongs to the category of problems that determine how data is structured. Unlike Supervised Learning or Reinforcement Learning, training data with only input values is It is a learning method to find the regularity of inputs using

Among unsupervised learning, Self-Supervised Learning is a type of unsupervised learning that uses unlabeled data to actually solve a network that has learned a user-defined pretext task. It is a learning method that transfer learning to a downstream task.

Korean Registered Patent Publication No. 1994316 (registered on June 24, 2019)

The present invention is to solve the above-mentioned problems of the prior art, a 6D pose estimating device that can solve the problem of performance degradation that occurs when estimating the pose of an object using a deep learning model learned based on arbitrary data want to provide

In addition, it is intended to provide a 6D pose estimation device capable of estimating an object pose efficiently only with unlabeled data or data containing a small amount of labels.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above technical problem, an embodiment of the present invention, in an apparatus for estimating an object pose using unsupervised learning, RGB data for an image is converted to a first deep learning model. a feature extraction unit including an external feature extraction unit that extracts external shape features by input and a geometric feature extraction unit that extracts geometric features by inputting depth data of the image to a second deep learning model; a combination unit generating a feature map of the image by combining the extracted shape features and geometric features; And, an object pose estimator for estimating a 6D pose for an object corresponding to the image based on the generated feature map, wherein the second deep learning model is based on a user-defined problem (Pretext task) of the learned network It is possible to provide a 6D pose estimating device that is transfer-learned based on weights.

Another embodiment of the present invention is a method for estimating an object pose using unsupervised learning, comprising the steps of extracting appearance features by inputting RGB data of an image to a first deep learning model; A feature extraction step including the step of extracting a geometric feature by inputting depth data of the image to a second deep learning model; generating a feature map for the image by combining the extracted shape features and geometric features; And, estimating a 6D pose for the object corresponding to the image based on the generated feature map, wherein the second deep learning model is based on a weight of a network learned by a user-defined problem (Pretext task) It is possible to provide a 6D pose estimation method based on transfer learning.

The above-described means for solving the problems is only illustrative and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, it is possible to provide a 6D pose estimating device capable of estimating the pose of an object efficiently only with unlabeled data or data containing a small amount of labels.

In addition, by using the 6D information of the estimated object, it is possible to perform complex tasks including assembly as well as Pick & Place tasks of the robot.

1 is a block diagram of a 6D pose estimation device according to an embodiment of the present invention.
2 is an exemplary diagram for explaining the configuration of a 6D pose estimation device according to an embodiment of the present invention.
3 is an exemplary diagram for explaining a tray model according to an embodiment of the present invention.
4 is an exemplary diagram for explaining a noise unit of a pretext task performer according to an embodiment of the present invention.
5 is a flowchart of a 6D pose estimation method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.

In this specification, some of the operations or functions described as being performed by a terminal or device may be performed instead by a device connected to the terminal or device. Likewise, some of the operations or functions described as performed by the device may also be performed by a terminal or device connected to the device.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a 6D pose estimation device according to an embodiment of the present invention. Referring to FIG. 1 , the 6D pose estimating device 100 may include a feature extractor 110, a combiner 120, an object pose estimator 130, and a pretext task performer 140. can

The feature extractor 110 may include an external feature extractor 111 and a geometric feature extractor 112, and the object pose estimator 130 includes a semantic segmentation module 131 and a feature point detection module. (Keypoint detection module, 132) and center voting module (Center voting module, 133), and the user-defined problem (Pretext task) performer 140 is a modeling unit 141, noise unit 142 and learning A section 143 may be included. However, the above components 110 to 140 are merely examples of components that can be controlled by the 6D pose estimating device 100 .

The 6D pose estimation apparatus 100 according to an embodiment of the present invention can estimate the pose of an object efficiently only with unlabeled data or data including a small amount of labels.

In addition, the 6D pose estimating device 100 according to an embodiment of the present invention can perform complex tasks including assembly as well as Pick & Place tasks of a robot using 6D information of an estimated object.

2 is an exemplary diagram for explaining the configuration of a 6D pose estimation device according to an embodiment of the present invention. Referring to FIG. 2 , the feature extractor 110 may include an external feature extractor 111 and a geometric feature extractor 112 .

The external feature extraction unit 111 according to an embodiment of the present invention may extract the external features by inputting the RGB data 111a of the image to the first deep learning model 111b. Here, the first deep learning model 111b may be a Convolutional Neural Network (CNN). For example, the external feature extractor 111 may receive the RGB data 111a and extract external features of the image through the CNN 111b.

The geometric feature extractor 112 may extract the geometric feature by inputting the depth data 112a of the image to the second deep learning model 112b. Here, the second deep learning model 112b may be a PointNet model. For example, the geometric feature extractor 112 may receive the depth data 112a and extract geometric features of the image through the second deep learning model 112b.

The geometric feature extractor 112 according to an embodiment of the present invention may transfer-learn the weights of the second deep learning model 112b to the weights of a separate network 143a. Here, the network 143a may be trained using unlabeled data through self-supervised learning, which is a type of unsupervised learning. For example, the network 143a may be trained by a user-defined pretext task. In this way, since the use of a user-defined task (Pretext task) as self-supervised learning can increase the understanding of 3D data itself, it is possible to improve performance for real environments when estimating 6D poses.

The geometric feature extractor 112 may extract geometric features of an image using the second deep learning model 112b obtained by transfer learning of a network learned by a user-defined pretext task.

The pretext task performer 140 according to an embodiment of the present invention may perform a pretext task based on the network 143a. For example, the user-defined task (Pretext task) performer 140 trains the network 143a using unlabeled data, that is, a user-defined user-defined task (Pretext task, input) to generate the network 143a. Iteratively learns to understand the data and extract meaningful features (outputs) from the data.

The modeling unit 141 of the pretext task performing unit 140 models a tray model including a plurality of holes and an object model insertable into at least one of the plurality of holes. can do.

3 is an exemplary diagram for explaining a tray model according to an embodiment of the present invention. Referring to FIG. 3 , a tray model 300 may include a plurality of holes 310 . The tray model 300 is an object model and may include, for example, a plurality of holes 310 into which an electric drill, a nipper, a stripper, a screwdriver, a wrench, and a spanner can be inserted.

The modeling unit 141 may model a tray model 300 including a plurality of holes 310 into which various object models may be inserted. Also, the modeling unit 141 may model a plurality of object models insertable into at least one of the plurality of holes 310 .

When the user-defined problem (Pretext task) performer 140 learns the network 143a using the tray model and the object model that have no noise in the virtual environment, the object pose estimator 130 detects the real object with noise. You may not understand the objects in the image properly.

To solve this problem, the noise unit 142 may add noise to the point cloud of the modeled tray model and object model. For example, the noise unit 142 estimates a 6D pose for an object corresponding to a real image in the object pose estimator 130 by adding noise to the point clouds of the modeled tray model and the object model. can improve comprehension.

For example, the noise unit 142 may randomly change a point cloud value based on a depth quality specification of an RGB-D camera in use. For example, the noise unit 142 may add or subtract random values from the original data within a depth standard deviation range, and the data as much as the ratio of the non-fill rate is NaN. (not a number).

4 is an exemplary diagram for explaining a noise unit of a pretext task performer according to an embodiment of the present invention. Referring to FIG. 4 , the depth quality of the RealSense ^TM D400 series camera is taken as an example.

The learning unit 143 may train the network 143a to determine a hole into which the object model is inserted by rotating the object model to which noise is added and inputting the object model to the network 143a.

For example, the learning unit 143 may input a rotated object model with added noise to the network 143a. The learning unit 143 may output a probability of correct answer for each object model when a point cloud of an object is input to the network 143a. The learning unit 143 may match an object model with a highest probability among output probabilities of correct answers to the tray model.

Also, the learning unit 143 may calculate an overlapping area between the object model and the tray model and an empty area as a loss (LOSS). For example, when the learning unit 143 matches an object model with the highest probability of correct answer to a tray model, the object model before noise is added and rotated is used to match the area where the tray model and the object model overlap, as well as the tray model and the tray model. The empty area between object models can be calculated as loss.

Referring back to FIG. 2 , the feature extractor 110 combines the external features of the image extracted through the external feature extractor 111 and the geometric features of the image extracted through the geometric feature extractor 112 into a combination unit 120. ) can be passed on.

The combination unit 120 according to an embodiment of the present invention may generate a feature map for an image by combining the extracted external features and geometric features. For example, the combining unit 120 combines the external features extracted from the external feature extraction unit 111 and the geometric features extracted from the geometric feature extraction unit 112 to obtain each coordinate (point) of an object recognized in the corresponding image. A feature map with combined features can be created.

The object pose estimator 130 according to an embodiment of the present invention may estimate a 6D pose of an object corresponding to an image based on the generated feature map. For example, the object pose estimator 130 may estimate 136 a 6D pose of the object based on a feature map having features combined for each coordinate of the object recognized in the image.

The object pose estimator 130 may calculate a 6D pose of the object based on the semantic segmentation module 131 , the feature point detection module 132 , and the center voting module 133 .

For example, the object pose estimator 130 receives the feature map, first, through the semantic segmentation module 131, can distinguish one or more objects included in the image, respectively, and then, the feature point detection module ( 132) to detect 3D keypoints for each object surface, and then detect the center point of the object through the center voting module 133 to estimate the 6D pose of the object (136 )can do.

When there are a plurality of objects on the image, the semantic segmentation module 131 according to an embodiment of the present invention may classify each object based on the feature map.

The feature point detection module 132 may detect 3D feature points on the surface of the identified object. For example, the feature point detection module 132 detects 3D feature points on the surface of the identified object, predicts a Euclidean transform offset from a visible point to a target keypoint for each detected point, and , it is possible to detect 3D feature points on the surface of the classified object through the process of voting and clustering the target feature points again.

The center voting module 133 may detect a center point of an object. For example, the center voting module 133 may expand the center point of an object from 2D to 3D.

Instance segmentation (Instance Segmentation, 134) is based on the objects detected by the semantic segmentation module 131, the feature point detection module 132, and the center voting module 133, and the 3D feature points and center points of the object. Local features can be extracted. In the instance segmentation 134, objects having a similar shape but different sizes may be distinguished from learned size information in order to predict an offset for a feature point.

The object pose estimator 130 performs least squares fit (least-squares fit) based on the features extracted through the semantic segmentation module 131, feature point detection module 132, center voting module 133, and instance segmentation 134. Fitting, 135) can be performed. For example, the object pose estimator 130 calculates pose parameters (R, t) for an object model that minimizes squared loss using M feature points detected in the camera coordinate system and points in the object coordinate system corresponding thereto, and calculates the object A 6D pose for 136 may be estimated.

Meanwhile, if the data is a 3D point cloud, the object pose estimator 130 may perform voxelization on the data, and in this case, the user-defined task performer 140 may also perform voxelization on the input data Voxelization may be additionally performed.

The 6D pose estimating device 100 according to the present invention improves the understanding of data including objects through the pretext task performing unit 140 using self-supervised learning, so that the object pose estimating unit 130 ), it is possible to more accurately estimate the 6D pose of the object included in the image.

In addition, the 6D pose estimating device 100 may utilize the 6D pose information of the estimated object to enable the robot to perform not only Pick & Place work but also complex and various tasks including assembling. For example, the 6D pose estimating apparatus 100 may determine the position and angle of an object using 6D information of the object.

5 is a flowchart of a 6D pose estimation method according to an embodiment of the present invention. The 6D pose estimation apparatus 100 shown in FIG. 1 includes steps processed time-sequentially according to the embodiment shown in FIGS. 1 to 4 . Therefore, even if the content is omitted below, it is also applied to the method of estimating the object pose in the 6D pose estimation apparatus 100 according to the embodiment shown in FIGS. 1 to 4 .

In step S510, the 6D pose estimating device 100 may input the RGB data of the image to the first deep learning model to extract external features.

In step S520, the 6D pose estimating apparatus 100 may extract geometric features by inputting depth data of the image to the second deep learning model.

In step S530, the 6D pose estimating device 100 may perform transfer learning on the second deep learning model based on the weights of the network learned by the user-defined pretext task.

In step S540, the 6D pose estimating device 100 may generate a feature map for the image by combining the extracted shape features and geometric features.

In step S550, the 6D pose estimating apparatus 100 may estimate the 6D pose of the object corresponding to the image based on the generated feature map.

In the above description, steps S510 to S550 may be further divided into additional steps or combined into fewer steps, depending on an implementation example of the present invention. Also, some steps may be omitted as needed, and the order of steps may be switched.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

100: 6D pose estimation device
110: feature extraction unit
120: coupling part
130: object pose estimation unit
140: Pretext task execution unit

Claims (20)

An apparatus for estimating an object pose using unsupervised learning,
An external feature extractor extracting external features by inputting RGB data of an image into a first deep learning model, and a geometric feature extracting geometric features by inputting depth data of the image into a second deep learning model. a feature extraction unit including an extraction unit;
a combination unit generating a feature map for the image by combining the extracted shape features and geometric features; and,
An object pose estimator for estimating a 6D pose of an object corresponding to the image based on the generated feature map
including,
The second deep learning model is transfer learned based on the weight of the network learned by a user-defined problem (Pretext task),
Further comprising a pretext task performing unit for performing the pretext task based on the network,
The user-defined problem (Pretext task) performing unit,
a modeling unit for modeling a tray model including a plurality of holes and an object model insertable into at least one of the plurality of holes;
a noise unit adding noise to the point clouds of the modeled tray model and object model; and
And a learning unit for training the network to determine a hole into which the object model is inserted by rotating the object model to which the noise is added and inputting the object model to the network.
According to claim 1,
The first deep learning model is a convolutional neural network (CNN), and the second deep learning model is a point net (PointNet) model, the 6D pose estimating device.
delete delete According to claim 1,
The learning unit calculates an overlapping area and an empty area between the object model and the tray model as a loss, 6D pose estimating device.
According to claim 1,
The geometric feature extraction unit transfer-learns the weights of the second deep learning model to the weights of the network, 6D pose estimating device.
According to claim 1,
The object pose estimation unit,
A 6D pose estimating device that calculates a pose for the object based on a semantic segmentation module, a keypoint detection module, and a center voting module.
According to claim 7,
The semantic segmentation module,
Based on the feature map, if there are a plurality of objects on the image, to distinguish each object, 6D pose estimating device.
According to claim 8,
The feature point detection module,
6D pose estimating device for detecting 3D keypoints on the surface of the separated object.
According to claim 9,
The center voting module,
To detect the center (center) point of the object, 6D pose estimating device.
In the method of estimating an object pose using unsupervised learning,
extracting appearance features by inputting RGB data of an image to a first deep learning model;
a feature extraction step comprising extracting a geometric feature by inputting depth data of the image to a second deep learning model;
generating a feature map for the image by combining the extracted shape features and geometric features; and,
Estimating a 6D pose for an object corresponding to the image based on the generated feature map
including,
The second deep learning model is transfer learned based on the weight of the network learned by a user-defined problem (Pretext task),
Further comprising performing the pretext task based on the network,
The step of performing the user-defined problem (Pretext task) is,
modeling a tray model including a plurality of holes and an object model insertable into at least one of the plurality of holes;
adding noise to point clouds of the modeled tray model and object model; and
training the network to determine a hole into which the object model is inserted by inputting the object model to the network while rotating the object model to which the noise is added;
To include, 6D pose estimation method.
According to claim 11,
Wherein the first deep learning model is a Convolutional Neural Network (CNN), and the second deep learning model is a PointNet model.
delete delete According to claim 11,
The step of learning the network is to calculate an overlapping area and an empty area between the object model and the tray model as a loss, 6D pose estimation method.
According to claim 11,
The step of extracting the geometric feature is transfer learning of the weights of the second deep learning model to the weights of the network, 6D pose estimation method.
According to claim 11,
Estimating the 6D pose for the object,
A 6D pose estimation method, wherein a pose for the object is calculated based on a semantic segmentation module, a keypoint detection module, and a center voting module.
18. The method of claim 17,
The semantic segmentation module,
Based on the feature map, if there are a plurality of objects on the image, to distinguish each object, 6D pose estimation method.
According to claim 18,
The feature point detection module,
6D pose estimation method of detecting 3D keypoints on the surface of the separated object.
According to claim 19,
The center voting module,
6D pose estimation method of detecting the center point of the object.

Images