CN111283664B - Registration system and method for robot augmented reality teaching - Google Patents

Abstract

The invention relates to a registration system for robot augmented reality teaching, which comprises the following steps: the system comprises a physical robot unit, a registration unit, a virtual robot generation unit and a computer; the physical robot unit comprises a physical robot, a physical robot controller and a robot point intermittent motion control program; the physical robot is provided with a physical robot base coordinate system; the physical robot controller is respectively connected with the physical robot and the computer; the robot point location intermittent motion control program is installed in the computer; the registration unit comprises a registration marker, a camera and a conversion calculation unit; the registration marker is arranged on the physical robot body; the camera is fixedly installed in a physical environment except the physical robot; the camera is connected with the computer, and the conversion calculation unit is arranged in the computer; the virtual robot generation unit is arranged in the computer and used for generating a virtual robot model.

Description

A registration system and method for robot augmented reality teaching

technical field

The invention relates to a registration system and method for robot augmented reality teaching, belonging to the fields of intelligent manufacturing and robot teaching.

Background technique

At present, augmented reality technology is widely used in manufacturing industry. Augmented reality registration technology accurately superimposes virtual objects and virtual information on the real world, so that observers can see a scene that combines virtual and real, and feel that virtual objects are part of the surrounding real world from the senses. Augmented reality registration technology is one of the most important technologies in augmented reality systems, and it is of great significance to the realization of augmented reality systems. Robot augmented reality teaching accurately superimposes the virtual robot model in the physical production environment. The operator uses human-computer interaction equipment to drive the movement of the virtual robot model, and plans the path of the robot in a virtual and real superimposed scene.

The augmented reality registration technology scheme used in the patent "Robot Teaching System and Method Based on RGB-D Image and Teaching Device" (public number CN201910665326.9) is: manually place an AR registration card, and the computer uses a camera to collect AR registration The image of the card, through the machine vision and augmented reality registration algorithm to detect the logo on the AR registration card, calculate the conversion matrix between the camera and the AR registration card, the computer uses the conversion matrix to set the orientation of the virtual camera in the virtual world, and the virtual robot model Placed exactly where the AR registration card is in the image. However, this technical solution needs to place the AR registration card in advance. If the base coordinates of the virtual robot and the coordinates of the AR registration card do not coincide, the coordinate conversion relationship between the two coordinate systems needs to be measured in advance. In addition, in the case of a physical robot, in order to make the base coordinate system of the virtual robot consistent with the base coordinate system of the physical robot, it is necessary to measure the transformation matrix between the AR registration card and the robot base coordinate system in advance. Poor performance, not suitable for robot remote teaching and mobile industrial robot (industrial robot placed on a mobile platform such as AGV) teaching.

Contents of the invention

In order to solve the above-mentioned technical problems, the present invention provides a registration system and method for robot augmented reality teaching. It is not necessary to install an AR registration card on a physical entity other than the robot in advance, so that augmented reality registration can be realized, and virtual robot model registration can be realized. The base coordinate system is consistent with the base coordinate system of the physical robot, which is suitable for remote augmented reality teaching of industrial robots and augmented reality teaching of mobile industrial robots.

The technical scheme adopted in the present invention is as follows:

Technical solution one:

A registration system for robot augmented reality teaching: including a physical robot unit, a registration unit, a virtual robot generation unit and a computer;

The physical robot unit includes a physical robot, a physical robot controller and a robot point intermittent motion control program; the physical robot is provided with a physical robot base coordinate system; the physical robot controller is connected with the physical robot and the computer respectively , the physical robot controller is used to control the movement of the physical robot and obtain the trajectory of the physical robot; the computer sends a point-to-point intermittent motion control program to the physical robot controller, and the physical robot controller executes the intermittent motion control of the robot A program for controlling the actuator of the physical robot to perform at least three or more zigzag movements;

The registration unit includes a registration marker, a camera and a conversion calculation unit; the registration marker is set on the physical robot body; the camera is fixedly installed in a physical environment other than the physical robot, and is aligned with the working area of the physical robot Shooting; the camera is connected to the computer, and the conversion calculation unit is set in the computer to calculate the difference between the three-dimensional coordinates of the registered marker in the camera coordinate system and the three-dimensional coordinates of the registered marker in the physical robot base coordinate system conversion relationship between

The virtual robot generation unit is set in the computer and is used to generate a virtual robot model.

Further, the registration marker is set on the actuator at the end of the physical robot, and the registration marker is an object with a characteristic shape or color.

Alternatively, the registration marker is set on the joints of the physical robot, and the registration marker is an object with a characteristic shape or color.

Further, the computer is specifically used for: at each intermittent point of the intermittent movement, the computer reads the three-dimensional coordinates of the registered marker in the physical robot base coordinate system through the robot controller; at the same time, obtains the physical working environment through the camera The image of the registered marker is identified by computer vision algorithm and the pixel coordinates of the registered marker in the pixel coordinate system are calculated.

Further, the conversion calculation unit is specifically configured to: use the imaging model of the camera and the internal reference matrix of the camera, according to the pixel coordinates of the registration marker in the pixel coordinate system and the three-dimensional coordinates of the registration marker in the physical robot base coordinate system The coordinates construct the conversion relationship between the camera coordinate system and the physical robot base coordinate system by constructing the least squares problem.

Technical solution two

A registration method for robot augmented reality teaching, which is implemented based on the registration system for robot augmented reality teaching described in Technical Solution 1. The specific steps are as follows:

Install the registration marker on the physical robot, and fix the camera;

Write a robot point-position intermittent motion control program on the computer and send it to the physical robot controller, the robot point-position intermittent motion control program at least controls the actuator of the physical robot to perform at least three or more zigzag motions;

Downloading the completed point-to-point intermittent motion control program of the robot to the physical robot controller, and the physical robot controller executes the point-to-point intermittent motion control program of the robot to control the movement of the physical robot;

The camera takes an image and sends it to the computer, and the computer calculates the pixel coordinates of the registered marker in the image, and at the same time, the computer obtains the three-dimensional coordinates of the registered marker in the physical robot base coordinate system through the physical robot controller;

The conversion calculation unit reads the pixel coordinates of the registration marker in the image and the three-dimensional coordinates of the registration marker in the physical robot base coordinate system, and calculates the conversion relationship between the camera coordinate system and the physical robot base coordinate system;

The virtual robot generation unit generates a virtual robot model consistent with the physical robot base coordinate system according to the conversion relationship between the calculated camera coordinate system and the physical robot base coordinate system, and superimposes the virtual robot model on the real scene through an augmented reality device , to complete the registration.

Further, the registration marker is set on the actuator at the end of the physical robot, and the registration marker is an object with a characteristic shape or color.

Alternatively, the registration marker is set on the joints of the physical robot, and the registration marker is an object with a characteristic shape or color.

Further, the camera takes an image and sends it to the computer, and the computer calculates the pixel coordinates of the registered marker in the image, and at the same time, the computer obtains the three-dimensional coordinates of the registered marker in the physical robot base coordinate system through the physical robot controller. In the step of coordinates: at each intermittent point of intermittent motion, the computer controls the camera to take pictures, and at the same time reads the three-dimensional coordinates of the current registered marker in the physical robot base coordinate system.

Further, the conversion calculation unit reads the pixel coordinates of the registered marker in the image and the three-dimensional coordinates of the registered marker in the physical robot base coordinate system, and calculates the conversion relationship between the camera coordinate system and the physical robot base coordinate system The specific steps are:

The conversion calculation unit utilizes the imaging model of the camera and the internal reference matrix of the camera, and constructs a least squares problem according to the pixel coordinates of the registered marker in the pixel coordinate system and the three-dimensional coordinates of the registered marker in the physical robot base coordinate system. , to calculate the conversion relationship between the camera coordinate system and the physical robot base coordinate system.

The present invention has following beneficial effect:

1. The present invention is oriented to the registration system and method for robot augmented reality teaching. It is not necessary to install an AR registration card on a physical entity other than the robot in advance to realize augmented reality registration, and to realize the basic coordinate system of the virtual robot model and the physical robot. The base coordinate system is consistent, suitable for remote augmented reality teaching of industrial robots and augmented reality teaching of mobile industrial robots;

2. Since there is no need to install the AR registration card in the physical scene, the influence of the AR registration card installation accuracy on the augmented reality registration accuracy is avoided, and the base coordinate system of the virtual robot can be consistent with the base coordinate system of the physical robot with high precision, improving Accuracy of the taught path.

Description of drawings

Fig. 1 is a schematic diagram of a registration system for robot augmented reality teaching of the present invention;

Fig. 2 is a schematic diagram of the use of a registration system for robot augmented reality teaching according to the present invention;

FIG. 3 is a schematic flowchart of a registration method for robot augmented reality teaching according to the present invention.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

Embodiment one

Please refer to Figure 1 and Figure 2, a registration system for robot augmented reality teaching: including a physical robot unit, a registration unit, a virtual robot generation unit and a computer;

The physical robot unit includes a physical robot, a physical robot controller and a robot point intermittent motion control program; the physical robot is provided with a physical robot base coordinate system; the physical robot controller is connected with the physical robot and the computer respectively , the physical robot controller is used to control the movement of the physical robot and obtain the trajectory of the physical robot; the robot point intermittent motion control program is installed in the computer, and the computer sends the point intermittent motion control program to the physical robot. The robot controller executes the robot intermittent motion control program by the physical robot controller, controls the actuator of the physical robot to perform at least three or more broken-line movements, and can control the robot to pause at the turning point of the broken-line segment;

The registration unit includes a registration marker, a camera and a conversion calculation unit; the registration marker is set on the physical robot body; the camera is fixedly installed in a physical environment other than the physical robot, and is aligned with the working area of the physical robot Shooting; the camera is connected to the computer, and the conversion calculation unit is set in the computer to calculate the difference between the three-dimensional coordinates of the registered marker in the camera coordinate system and the three-dimensional coordinates of the registered marker in the physical robot base coordinate system conversion relationship between

The virtual robot generation unit is set in the computer and is used to generate a virtual robot model.

This embodiment does not need to install the AR registration card on physical entities other than the physical robot in advance to realize the augmented reality registration, avoiding the influence of the installation accuracy of the AR registration card on the accuracy of the augmented reality registration, and realizing the basic registration of the virtual robot with high precision. The coordinate system is consistent with the base coordinate system of the physical robot, which improves the accuracy of the teaching path, and is suitable for remote augmented reality teaching of industrial robots and augmented reality teaching of mobile industrial robots.

Embodiment two

Further, the registration marker is set on the actuator at the end of the physical robot, and the coordinates of the registration marker in the robot coordinate system can be read in real time. In order to facilitate detection and calculation, the registration marker has a characteristic shape (such as spheres, cubes, etc.) or objects of color (such as red, yellow).

Alternatively, the registration marker is set on the joint of the physical robot, and the joint data of the robot can be read in real time, and the coordinates of the marker in the robot coordinate system can be obtained through the forward kinematics model of the robot. In order to facilitate detection and calculation Conveniently, the registration marker is an object with a characteristic shape or color.

Further, the computer is specifically used for: at each intermittent point of the intermittent motion, that is, the endpoint of each line segment in the intermittent motion, the computer reads the registered identifier in the physical robot base coordinate system O _w − through the robot controller. The three-dimensional coordinates P _i (x _wi , y _wi , z _wi ) under X _w Y _w Z _w ; at the same time, the image of the physical working environment is obtained through the camera, and the registration mark is recognized by the computer vision algorithm on the computer, and the registration mark is calculated The pixel coordinates Z _i (u _i , v _i ) of the object in the pixel coordinate system. Specifically refer to the intermittent motion path in Figure 2. The robot’s intermittent motion trajectory contains at least 3 or more broken line segments, so the data of at least 4 non-collinear feature points (such as 4 endpoints of a square) can be obtained, that is, the data of 4 Pi data.

Further, the conversion calculation unit is specifically used to calculate the conversion matrix R: assuming that the conversion matrix from the world coordinate system to the camera coordinate system is R, the following relationship can be obtained by using the imaging model of the camera:

Among them, A _3*4 is the internal reference matrix of the camera, and Q _i (x _ci , y _ci , z _ci ) is the three-dimensional coordinates of the registration marker in the camera coordinate system O _c -X _c Y _c Z _c .

According to the pixel coordinates Z _i (u _i , v _i ) of all feature points (4 and above), and the corresponding three-dimensional coordinates P _i (x _wi , y _wi , z _wi ), the following formula is the target

Use the singular value decomposition to find the least square rigid transposition method to solve the transformation matrix R _4*4 , and use the matrix R _4*4 to realize the transformation from the physical robot base coordinate system O _w -X _w Y _w Z _w to the camera coordinate system O _c - The transformation of X _c Y _c Z _c , R _4*4 , that is, the conversion matrix from the world coordinate system to the camera coordinate system is R.

Finally, use the transformation matrix R to set the position of the virtual camera in the virtual world, and superimpose the virtual robot model on the image captured by the camera on the position of the physical robot to complete the augmented reality registration and realize the base coordinate system of the virtual robot model and the base coordinate system of the physical robot. Consistency of the coordinate system.

Embodiment Three

Referring to Fig. 3, a registration method for robot augmented reality teaching, which is implemented based on the registration system for robot augmented reality teaching described in Embodiment 1, the specific steps are as follows:

Install the registration marker on the physical robot, and fix the camera;

Writing a point-to-point intermittent motion control program of the robot on the computer and sending it to the physical robot controller, the robot point-to-point intermittent motion control program at least controls the actuator of the physical robot to perform at least 3 times of zigzag motion;

Downloading the completed point-to-point intermittent motion control program of the robot to the physical robot controller, and the physical robot controller executes the point-to-point intermittent motion control program of the robot to control the movement of the physical robot;

The camera takes an image and sends it to the computer, and the computer calculates the pixel coordinates of the registered marker in the pixel coordinate system, and at the same time, the computer obtains the three-dimensional coordinates of the registered marker in the physical robot base coordinate system through the physical robot controller;

The conversion calculation unit reads the pixel coordinates of the registration marker in the image and the three-dimensional coordinates of the registration marker in the physical robot base coordinate system, and calculates the conversion relationship between the camera coordinate system and the physical robot base coordinate system;

The virtual robot generation unit generates a virtual robot model consistent with the physical robot base coordinate system according to the conversion relationship between the calculated camera coordinate system and the physical robot base coordinate system, and superimposes the virtual robot model on the real scene through an augmented reality device , to complete the registration.

This embodiment does not need to install the AR registration card on physical entities other than the physical robot in advance to realize the augmented reality registration, avoiding the influence of the installation accuracy of the AR registration card on the accuracy of the augmented reality registration, and realizing the basic registration of the virtual robot with high precision. The coordinate system is consistent with the base coordinate system of the physical robot, which improves the accuracy of the teaching path, and is suitable for remote augmented reality teaching of industrial robots and augmented reality teaching of mobile industrial robots.

Embodiment Four

Further, the registration marker is set on the actuator at the end of the physical robot, and the coordinates of the registration marker in the robot coordinate system can be read in real time. In order to facilitate detection and calculation, the registration marker has a characteristic shape (such as spheres, cubes, etc.) or objects of color (such as red, yellow).

Alternatively, the registration marker is set on the joint of the physical robot, and the joint data of the robot can be read in real time, and the coordinates of the marker in the robot coordinate system can be obtained through the forward kinematics model of the robot. In order to facilitate detection and calculation Conveniently, the registration marker is an object with a characteristic shape or color.

Further, the camera takes an image and sends it to the computer, and the computer calculates the pixel coordinates of the registered marker in the image, and at the same time, the computer obtains the three-dimensional coordinates of the registered marker in the physical robot base coordinate system through the physical robot controller. In the step of coordinates: at each intermittent point of intermittent movement, the computer controls the camera to shoot, recognize the registered marker and calculate the pixel coordinate Z _i (u _i , v _i ) of the registered marker in the pixel coordinate system, and read The three-dimensional coordinates _{P i (} x _{wi ,} y _wi , z _wi ) of the current registered identifier in the physical robot base coordinate system O _w -X w Y w Z _w .

Further, the conversion calculation unit reads the pixel coordinates of the registered marker in the image and the three-dimensional coordinates of the registered marker in the physical robot base coordinate system, and calculates the conversion relationship between the camera coordinate system and the physical robot base coordinate system The specific steps are:

Assuming that the transformation matrix from the world coordinate system to the camera coordinate system is R, the following relationship can be obtained by using the camera imaging model:

Among them, A _3*4 is the internal reference matrix of the camera, and Q _i (x _ci , y _ci , z _ci ) is the three-dimensional coordinates of the registration marker in the camera coordinate system O _c -X _c Y _c Z _c .

According to the pixel coordinates Z _i (u _i , v _i ) of all feature points (4 and above), and the corresponding three-dimensional coordinates P _i (x _wi , y _wi , z _wi ), the following formula is the target

Use the singular value decomposition to find the least square rigid transposition method to solve the transformation matrix R _4*4 , and use the matrix R _4*4 to realize the transformation from the physical robot base coordinate system O _w -X _w Y _w Z _w to the camera coordinate system O _c - The transformation of X _c Y _c Z _c , R _4*4 , that is, the conversion matrix from the world coordinate system to the camera coordinate system is R.

Finally, use the transformation matrix R to set the position of the virtual camera in the virtual world, and superimpose the virtual robot model on the image captured by the camera on the position of the physical robot to complete the augmented reality registration and realize the base coordinate system of the virtual robot model and the base coordinate system of the physical robot. Consistency of the coordinate system.

The above is only an embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technologies fields, all of which are equally included in the scope of patent protection of the present invention.

Claims (10)

1. The utility model provides a registration system towards robot augmented reality teaching which characterized in that:

the system comprises a physical robot unit, a registration unit, a virtual robot generation unit and a computer;

the physical robot unit comprises a physical robot, a physical robot controller and a robot point position intermittent motion control program; the physical robot is provided with a physical robot base coordinate system; the physical robot controller is respectively connected with the physical robot and the computer, and is used for controlling the physical robot to move and acquiring the motion track of the physical robot; the robot point position intermittent motion control program is installed in the computer, the physical robot controller executes the robot intermittent motion control program, and an execution mechanism of the physical robot is controlled to perform at least three or more broken line motions;

the registration unit comprises a registration identifier, a camera and a conversion calculation unit; the registration marker is arranged on the physical robot body; the camera is fixedly installed in a physical environment except the physical robot and is used for shooting aiming at a working area of the physical robot; the camera is connected with the computer, and the conversion calculation unit is arranged in the computer and used for calculating the conversion relation between the three-dimensional coordinates of the registration marker in the camera coordinate system and the three-dimensional coordinates of the registration marker in the physical robot base coordinate system;

the virtual robot generation unit is arranged in the computer and used for generating a virtual robot model.

2. The registration system for robot augmented reality teaching of claim 1, wherein: the registration marker is arranged on an actuating mechanism at the tail end of the physical robot, and the registration marker is an object with a specific shape or color.

3. The registration system for robot augmented reality teaching of claim 1, wherein: the registration marker is arranged on a joint of the physical robot, and the registration marker is an object with a specific shape or color.

4. A registration system for robot-oriented augmented reality teaching according to claim 2 or 3, wherein the computer is specifically configured to: at each intermittent point of the intermittent motion, the computer reads the three-dimensional coordinates of the registration marker in the physical robot base coordinate system through the robot controller; meanwhile, an image of a physical working environment is acquired through a camera, the registration marker is identified through a computer vision algorithm, and pixel coordinates of the registration marker in a pixel coordinate system are calculated.

5. The registration system for augmented reality teaching of robot according to claim 4, wherein the conversion calculation unit is specifically configured to: and calculating the conversion relation between the camera coordinate system and the physical robot base coordinate system by constructing a least square problem according to the pixel coordinates of the registration marker in the pixel coordinate system and the three-dimensional coordinates of the registration marker in the physical robot base coordinate system by using the imaging model of the camera and the internal reference matrix of the camera.

6. A registration method for robot augmented reality teaching is realized based on the registration system for robot augmented reality teaching of claim 1, and comprises the following specific steps:

installing a registration marker on the physical robot, and fixing a camera;

writing a robot point position intermittent motion control program on a computer and sending the robot point position intermittent motion control program to a physical robot controller, wherein the robot point position intermittent motion control program at least controls an execution mechanism of the physical robot to perform polygonal line motion more than at least three times;

downloading a written robot point location intermittent motion control program to the physical robot controller, wherein the physical robot controller executes the robot point location intermittent motion control program to control the physical robot to act;

the method comprises the steps that a camera shoots an image and sends the image to a computer, the computer calculates pixel coordinates of a registration marker in the image, and meanwhile, the computer obtains three-dimensional coordinates of the registration marker in a physical robot base coordinate system through a physical robot controller;

the conversion calculation unit reads the pixel coordinates of the registration marker in the image and the three-dimensional coordinates of the registration marker in the physical robot base coordinate system, and calculates the conversion relation between the camera coordinate system and the physical robot base coordinate system;

and the virtual robot generation unit generates a virtual robot model consistent with the physical robot base coordinate system according to the conversion relation between the calculation camera coordinate system and the physical robot base coordinate system, and superimposes the virtual robot model in a real scene through augmented reality equipment to finish registration.

7. The robot augmented reality teaching-oriented registration method according to claim 6, wherein: the registration marker is arranged on an actuating mechanism at the tail end of the physical robot, and the registration marker is an object with a specific shape or color.

8. The robot augmented reality teaching-oriented registration method according to claim 6, wherein: the registration marker is arranged on a joint of the physical robot, and the registration marker is an object with a specific shape or color.

9. The robot augmented reality teaching-oriented registration method according to claim 7 or 8, wherein the camera takes an image and sends the image to the computer, the computer calculates pixel coordinates of the registration marker in the image, and meanwhile, the computer obtains three-dimensional coordinates of the registration marker in a physical robot base coordinate system through the physical robot controller: and at each intermittent point of the intermittent motion, the computer controls the camera to shoot and simultaneously reads the three-dimensional coordinates of the current registration marker in the physical robot base coordinate system.

10. The robot augmented reality teaching-oriented registration method according to claim 9, wherein the step of reading the pixel coordinates of the registration marker in the image and the three-dimensional coordinates of the registration marker in the physical robot base coordinate system by the conversion calculation unit and calculating the conversion relationship between the camera coordinate system and the physical robot base coordinate system specifically comprises:

and the conversion calculation unit calculates the conversion relation between the camera coordinate system and the physical robot base coordinate system by constructing a least square problem according to the pixel coordinates of the registration marker in the pixel coordinate system and the three-dimensional coordinates of the registration marker in the physical robot base coordinate system by using the imaging model of the camera and the internal reference matrix of the camera.

Images