CN111483750A - Robot system control method and control device - Google Patents

Abstract

The subject of the present disclosure is to achieve a high degree of collaboration between units including robots and to fully improve the storage efficiency of the operation objects. The control method of the present disclosure includes: obtaining the approach position of the end effector to hold the operation object; obtaining the scanning position for scanning the identification of the operation object; and creating or obtaining a control sequence based on the approach position and the scanning position, and instructing the robot to execute the control sequence. The control sequence includes: (1) holding the operation object from the starting position; (2) scanning the identification of the operation object using a scanner located between the starting position and the working position; (3) when the specified conditions are met, temporarily releasing the operation object from the end effector at the holding change position, and holding it again with the end effector in the holding change manner; and (4) moving the operation object to the working position.

Description

Robot system control method and control device

technical field

The present disclosure generally relates to a robot system, and more particularly, to a control device, a control method, a logistics system, a program, and a storage medium of a robot system that operates an operation object such as an item.

Background technique

Currently, most robots (eg, machines configured to perform physical actions autonomously/independently) are widely used in many fields due to ever-increasing performance and reduced cost. For example, a robot can be used to perform various operations and tasks (tasks), such as manipulation and moving an operation object, in manufacturing, assembly, packing, transfer, and transportation. During the execution of the work, the robot can repeat the actions of the human, thereby replacing or reducing the dangerous or repetitive work performed by the human.

As a system (robot system) using such a robot, for example, Patent Document 1 proposes an automatic distribution system for automating and labor-saving from storage to delivery of articles, including a transport container storage mechanism, a temporary A storage and transportation container; and an automatic article delivery mechanism, which automatically collects the articles in the transportation container into the shipping container based on the out-of-stock information.

prior art literature

Patent Literature

Patent Document 1: Japanese Patent Laid-Open No. 2018-167950

SUMMARY OF THE INVENTION

However, regardless of technological advancement, with more robots, the sophistication needed to reproduce human-engaged jobs that perform larger and/or more complex jobs is lacking. Therefore, the automation and high performance of the robot system are not sufficient, and there are many operations that are difficult to replace manual participation. In addition, the robot system lacks granularity and fine control and flexibility in the actions performed. Therefore, technical improvements for managing various actions and/or dialogues between robots and further promoting automation and performance enhancement of robot systems are still required. Therefore, an object of the present disclosure is to provide a control device, a control method, and the like of a robot system, which can realize a high degree of cooperation among units including robots, for example, to sufficiently improve the storage efficiency of operation objects.

In order to solve the above-mentioned problems, the present invention adopts the following structures.

[1] That is, the control method of a robot system including a robot with a robot arm and an end effector according to the present disclosure includes: acquiring an approach position where the end effector holds (clamping) an operation object; A scanning position where the identification of the manipulation object is scanned; and based on the approach position and the scanning position, a control sequence is created or acquired, and the robot is instructed to execute the control sequence. In addition, the control sequence includes the following (1) to (4):

(1) Holding the operation object from the starting position;

(2) Scan the identification information of the operation object (for example, a computer-readable mark such as a barcode, a quick response (QR) code (registered trademark), etc.) with a scanner located between the start position and the work position. ;

(3) when a predetermined condition is satisfied, temporarily release the operation object from the end effector at the grip changing position, and re-grip by the end effector in a grip changing manner; and

(4) Move the operation object to the work position.

Among them, the "operation object" refers to the object operated by the operation robot provided in the robot system. For example, it includes one or more articles (commodities), containers such as bottles, containers, and boxes on which they are placed or stored. The container may be a package. It can also be packaged open, and in addition, it can be a part of the container (for example, the top) that is open. In addition, in other embodiments and examples, the "operation object" may also be a concept including a shelf, a pallet, a conveyor belt, other temporary placement places, and the like. In addition, the "control sequence" refers to the sequence of operations that are set in advance when the control for executing each work is performed in one or more units such as robots included in the robot system.

[2] In the above configuration, the control sequence may be configured to include the following (5) and (6):

(5) in the case of changing the gripping direction of the operation object by changing the gripping direction of the operation object, setting improvement of the storage efficiency of the operation object at the working position as the predetermined condition; as well as

(6) Calculate the storage efficiency at the work position before the grip change of the operation object and the storage efficiency at the work position after the grip change of the operation object.

[3] In the above-mentioned structure, the control sequence may also be configured to include the following (7) and (8):

(7) obtaining the height of the operation object; and

(8) Calculate the storage efficiency based on the height of the operation object.

[4] In the above configuration, it may be configured based on the height position (horizontal height) of the top surface of the operation object and the height position of the bottom surface of the operation object measured in the state held by the end effector (horizontal height), the height of the operation object is calculated.

[5] In the above configuration, the height of the measurement object may be measured when the scanner scans the operation object.

[6] In the above configuration, the control sequence may be configured to include (9) when the predetermined condition is satisfied, placing the operation object on the temporary placement table at the grip changing position, and moving from the end The actuator is temporarily released.

[7] In the above configuration, it may further include: acquiring photographing data indicating a pickup area including the operation object; judging the initial posture of the operation object based on the photographing data; calculating an image indicating the operation object a reliable reference for the likelihood that the initial pose is correct; and based on the reliable reference, the approach position and the scan position are obtained.

Among them, "pose" refers to the position and/or orientation of the operation object (for example, a posture including the orientation of the stopped state), and includes the parallel movement component and/or the rotation component in the grid system used by the robot system. Additionally, a "pose" may be represented by a vector, a collection of angles (eg, Euler angles, and/or roll-pitch-yaw angles), a homogeneous transformation, or a combination thereof, and the "pose" of the manipulated object These coordinate transformations and the like may contain translational components, rotational components, variations thereof, or combinations thereof.

In addition, the "reliable reference" refers to a quantitative reference for indicating the degree of agreement (degree of certainty or possibility) between the determined posture of the operator and the actual posture of the operator in the real world. In other words, the "reliable reference" is a reference indicating the accuracy of the determined posture of the operation object, or an index indicating the possibility that the determined posture matches the actual posture of the operation object. This "reliable reference" can, for example, be based on one or more visual characteristics (eg, shape, color, image, design, marking, text, etc.) of the manipulation object in the image data containing the pickup area of the manipulation object and the manipulation stored in the master data The matching results between the information related to the visual properties of the objects are quantified.

[8] In the above structure, the control sequence may be configured to include: (10) based on the comparison result of the reliable benchmark and the sufficiency threshold, according to a metric of performance and/or a metric of scanning, selectively The approaching position and the scanning position are calculated, and the measurement of the scanning is not related to whether the initial posture of the operation object is correct or not, but is related to the possibility that the identification of the operation object is not covered by the end effector.

[9] In the above configuration, when the reliability reference does not satisfy the sufficiency threshold, the approach position and the scan position may be acquired based on the scan metric, or the The metric of the scan over which the metric of the performance is based on which the proximity position and the swept position are obtained is obtained.

[10] Alternatively, in the above configuration, when the reliability criterion satisfies the sufficiency threshold, the approach position and the scanning position may be acquired based on the performance metric.

[11] In the above configuration, the control sequence may be configured to include the following (11 and (12):

(11) acquiring a first scanning position for providing identification information of the operation object to the scanner and a second scanning position for providing the alternative identification information of the operation object to the scanner;

(12) After the operation object is moved to the first scan position, if the scan result indicates a successful scan, move the operation object to the work position and ignore the second A scan position, or, if the scan result indicates a failed scan, moving the operation object to the second scan position.

[12] In addition, the present disclosure provides a non-transitory computer-readable storage medium storing processor commands for implementing a control method of a robotic system including a robot with a robotic arm and an end effector, so The processor command includes: a command for acquiring an approach position where the end effector holds an operating object; a command for acquiring a scanning position for scanning the identification information of the operating object; and based on the approaching position and the scanning position, Create or acquire a control sequence, instructing the robot to execute the commands of the control sequence. In addition, the control sequence includes the following (1) to (4):

(1) Holding the operation object from the starting position;

(2) using the scanner located between the starting position and the working position to scan the identification of the operating object;

(3) when a predetermined condition is satisfied, temporarily release the operation object from the end effector at the grip changing position, and perform gripping again with the end effector in a grip changing manner; and

(4) Move the operation object to the work position.

[13] In the above configuration, the control sequence may be configured to include the following (5) and (6):

(5) in the case of changing the gripping direction of the operation object by changing the gripping direction of the operation object, setting improvement of the storage efficiency of the operation object at the working position as the predetermined condition; as well as

(6) Calculate the storage efficiency at the work position before the grip change of the operation object and the storage efficiency at the work position after the grip change of the operation object.

[14] In the above configuration, the control sequence may be configured to include the following (7) and (8):

(7) obtaining the height of the operation object; and

(8) Calculate the storage efficiency based on the height of the operation object.

[15] In the above configuration, it may be configured based on the height position (horizontal height) of the top surface of the operation object and the height position of the bottom surface of the measurement object measured in the state held by the end effector (horizontal height), the height of the measurement object is calculated.

[16] In addition, a control device of a robot system including a robot with a robot arm and an end effector of the present disclosure is used to execute the control method according to any one of the above [1] to [11].

Description of drawings

FIG. 1 is a diagram showing an example environment in which a robot system according to an embodiment of the present disclosure operates.

2 is a block diagram showing an example of a hardware configuration of a robot system according to an embodiment of the present disclosure.

3A is a perspective view schematically showing a first posture of an operation object.

3B is a perspective view schematically showing a second posture of the operation object.

3C is a perspective view schematically showing a third posture of the operation object.

4A is a plan view showing an exemplary operation performed by the robot system according to the embodiment of the present disclosure.

4B is a front view showing an exemplary operation performed by the robot system according to the embodiment of the present disclosure.

5A is a flowchart showing an example of an operation sequence of the robot system according to the embodiment of the present disclosure.

5B is a flowchart showing an example of an operation sequence of the robot system according to the embodiment of the present disclosure.

Description of reference numerals

100...Robot system; 102...Unloading unit; 104...Transfer unit; 106...Conveying unit; 108...Stacking unit; 204...storage equipment; 206...communication equipment; 208...input-output equipment; 210...display; 212...operation equipment; 214...transfer motor; 216...sensor; sensor; 252...main data; 254...tracking data; 302...operating object; 304...first exposed surface; 306...second exposed surface; 312...first attitude; 314...second attitude; 316...third attitude; 322 ...top surface; 324...bottom surface; 326...peripheral surface; 332...identity; 334...identity position; 402, 404...job; 412, 416...scanner; 432...first access position; 434...second access position; 442...first delivery position; 444...second delivery position; 450...reservoir; 464...self-propelled trolley; ; 466... Distance measuring equipment; 468... Temporary placement table; 472, 474... Control sequence.

Detailed ways

According to the present disclosure, there are provided a robot system that highly integrates a plurality of units (for example, various robots, various devices, a control device provided integrally with them or separately, etc.), a control device thereof, and a logistics system provided with them, and a on their methods, etc. That is, the robot system of the embodiment of the present disclosure is, for example, an integrated system capable of autonomously executing one or more tasks. In addition, when the robot system according to the embodiment of the present disclosure stores the operation object in the storage container or the like, based on the shape and size of the operation object and the space volume of the storage container, the robot system includes advanced processing that can greatly improve the storage efficiency. way to operate. In addition, a control sequence is created or acquired and executed based on a reliable reference associated with the initial pose of the operation object, thereby providing a scan job in which the operation object is heightened.

The robot system according to the embodiment of the present disclosure can be configured to perform work based on manipulation (eg, physical movement and/or orientation) of the manipulation object. Specifically, the robot system can, for example, pick up an operation object from a pickup area including a start position (for example, a large box, a bottle, a container, a pallet, a storage container, a bucket, a cage, a conveyor belt, etc., which are a supply source of the operation object), Move it to the placement area including the target work position (for example, a large box, a bottle, a container, a pallet, a storage container, a bucket, a cage, a conveyor belt, etc., which are the moving destination of the operation object), and carry out various operation objects. permutation or replacement.

In addition, in the control sequence executed by the robotic system, one or more identifications (eg, barcodes or Quick Response (QR) codes (registered trademarks), etc.) to be located at one or more specific locations and/or surfaces of the operating object may be included Scan during transfer. Therefore, the robot system can perform the following various tasks: grasp and pick up the operation object, scan the mark at the appropriate position/orientation, adjust the posture, change the posture to perform grasp transformation (release the grasp, re-hold and pick up), transfer to the working position, and release the grasp , configured in the job location.

In addition, the robot system may be provided with a photographing device (eg, camera, infrared sensor/camera, radar, lidar, etc.) for recognizing the position and posture of the operation object and the surrounding environment of the operation object. Also, the robot system can calculate a reliable reference related to the posture of the manipulation object. In addition, the robot system can acquire an area indicating a pick-up area including a start position, a placement area including a work position, and a grip change position in the middle of the movement path of the operation object (for example, a suitable work table such as a temporary placement table, other robots ) and other images of the position and orientation of the operation object during transfer.

Furthermore, the robot system can perform image processing in a manner of identifying or selecting the operation objects in a predetermined order (for example, from top to bottom, from outside to inside, from inside to outside, etc.). And further, the robot system can, for example, recognize the contour of the operation object based on the color, brightness, depth/position, and/or combinations thereof, changes in these values of pixels in the pattern image of the captured data, and in addition, by grouping them, etc. , for example, the initial posture of the operation object in the pickup area can be determined from this image. In the determination of the initial pose, the robot system can calculate a reliable reference according to prescribed procedures and/or equations.

In addition, the robot system can perform the grip transformation (change the grip position of the operation object) of the manipulation object at the grip transformation position set in the middle of the path from the pick-up area including the start position to the placement area including the work position, etc., as necessary. In addition, during the movement of the operation object from the pick-up area including the start position to the placement area including the work position, the robot system can obtain the height of the operation object as required, for example, through a photographing device with a ranging function.

In addition, the robot system can execute a control sequence for performing each job according to the position, attitude, height, reliable reference, or a combination thereof of the operation object, and/or the position, attitude, or combination of the robot. For example, the above-mentioned control sequence can be created or acquired through motion planning, machine learning such as deep learning, or the like. The control sequence, for example, for arrangement transformation, grip transformation, replacement, etc. of the operation object, handles the following processing: grasping the operation object at the start position and/or any position in the middle of the movement, manipulating the operation object, placing the operation object at the target work position, and the like.

Here, a conventional robot system executes a control sequence of grasping the operation object in a pick-up area including the start position, etc., moving the operation object in the grasped state to a placement area including the work position, etc., and releasing. Therefore, in the conventional system, the grasped operation object moves in the grasped state, and is only released from the grasped state, and the space for stacking or accommodating the operation object cannot be effectively used. Therefore, manual intervention (adjustment, reproduction, replenishment, system stop, etc.) and operation input therefor are sometimes required in view of the stacking or storage efficiency of the operation objects.

On the other hand, the robot system in the present disclosure can create or acquire and execute a control sequence based on the shape information of the operation object and the stacking or storage information of the operation object, unlike the conventional one. In other words, the robot system in the present disclosure can further optimize the stacking or storage efficiency of the manipulation objects based on the shape information of the manipulation objects and the stacking or storage information of the manipulation objects. In addition, the robot system of the present disclosure can change the grasping position of the operation object to a grasping position suitable for optimizing stacking or storage efficiency of the operation object at the grasping change position located halfway from the start position to the working position.

In addition, unlike the existing system, the robot system of the present disclosure can create or acquire and execute a control sequence suitable for optimizing the stacking or storage efficiency of the operation object according to the actual height of the operation object as required. For example, even if the scanned symbols located at more than one specific position and/or on the surface of the operation object are the same operation object, they may have different shapes and sizes in practice. Therefore, on the upstream side (pre-stage) of the control sequence that is closer to the upstream than the grip change position, for example, by the photographing device (camera, distance measuring device) positioned in the vertical direction, based on the operation object known to the support position the distance information, and actually measure the height of the operation object. Then, the stacking or storage efficiency of the manipulation objects at the work position can be calculated based on the actually measured heights of the manipulation objects, and based on the results, the control sequence can be further optimized.

Moreover, the robot system in the present disclosure is different from the existing system, and can create or acquire a control sequence and execute it according to a reliable reference as needed. For example, it is possible to change the orientation of the operation object, change the grip position on the operation object, change the posture/position of the operation object, and/or change a part of the movement path according to a reliable reference.

In addition, with regard to the posture of the measurement object held in the pick-up area or the like, generally, the top surface is exposed horizontally (upward), and the side surface of the operation object is exposed vertically (laterally). Therefore, in the robot system of the present disclosure, in the master data, the operation object has an identification on the bottom surface of the operation object (ie, the side opposite to the top surface of the operation object), and has another identification on the side surface of the operation object. an identity.

In addition, when the robot system in the present disclosure processes the image of the pickup area in the recognition of the operation object, a reliable reference can be calculated as needed. In the event that the reliable reference exceeds the adequacy threshold, recognizing the sufficient certainty that the top surface of the manipulation object is exposed, the robotic system can deploy the end effector on this exposed top surface, grasping the top surface to be in front of the scanner Rotates the operand by providing the bottom surface of the operand at the specified location. On the other hand, if the reliable reference is smaller than the sufficiency threshold and it is impossible to identify whether the top surface or bottom surface of the operation object is exposed, the robot system can arrange the end effector along one side surface of the operation object and hold the side of the operation object. The surface, for example by passing between opposing sets of scanners, rotates the operand.

In this case, the operation object is scanned within the movement path of the operation object, for example, between the pickup area including the start position and the placement area including the work position, thereby improving work efficiency and work speed. At this time, the robot system in the present disclosure creates or acquires a control sequence that also cooperates with the scanner at the scanning position, so that the movement operation of the operation object and the scanning operation of the operation object can be effectively combined. Furthermore, by creating or acquiring a control sequence based on a reliable reference of the initial posture of the operation object, it is possible to further improve the efficiency, speed, and accuracy related to the scanning operation.

In addition, the robot system in the present disclosure can create or acquire a corresponding control sequence when the initial posture of the operation object is incorrect. Thereby, even in the case where the attitude determination of the operation object is wrong (for example, the result of correction error, unexpected attitude, expected outside light condition, etc. is wrongly determined), the possibility of correct and reliable scanning of the operation object can be increased. As a result, the overall throughput related to the robot system can be increased, and the labor/intervention of the operator can be further reduced.

In addition, in this specification, many specific detailed descriptions are provided in order to fully understand this disclosure, and this disclosure is not limited to these. In addition, in the embodiments of the present disclosure, the techniques described in this specification may be implemented without these specific details. Further, details of well-known specific functions, routines, and the like are not described in order to avoid unnecessarily obscuring the present disclosure. Reference in this specification to "an embodiment," "one embodiment," and the like, means that a particular feature, structure, material, or characteristic recited is included in at least one embodiment of the present disclosure. Therefore, such terminology in this specification does not necessarily refer to all the same embodiments. On the other hand, such references are not necessarily mutually exclusive. Furthermore, in one or more embodiments, the particular features, structures, materials or characteristics may be combined in any suitable manner. On this basis, it should be understood that the various embodiments shown are illustrative only and are not necessarily shown to scale.

Furthermore, descriptions of well-known structures or processes that are commonly associated with robotic systems and subsystems and that unnecessarily obscure several prominent aspects of the present disclosure are omitted for the purpose of clarifying the present disclosure. Furthermore, although this specification describes various embodiments of the present disclosure, the present disclosure as other embodiments may include structures different from those described in this section or structures having different constituent elements. Accordingly, the present disclosure may include other embodiments with additional elements or without some of the elements recited below.

Furthermore, various embodiments of the present disclosure may take the form of commands, including routines executed by a programmable computer or controller, and which may be executed by the computer or controller. It should be noted that those skilled in the art to which this disclosure pertains can understand that the techniques of this disclosure may be implemented in systems including various computers or controllers. The techniques of this disclosure can be implemented with special purpose computers or data processors programmed, configured, or constructed to execute more than one command on various computers. Thus, the terms "computer" and "controller" as used in this specification can be any data processor, and can include Internet devices and handheld devices (including palmtop computers, wearable computers, cellular or mobile telephones, multiprocessor systems, processor-based or programmable home appliances, network computers, microcomputers, etc.). Information processed by these computers and controllers can be provided to any suitable display medium, such as a liquid crystal display (LCD). Commands for performing computer- or controller-executable operations may be stored in any suitable computer-readable storage medium, including hardware, firmware, or a combination of hardware and firmware. Furthermore, these commands may be stored in any suitable memory device including, for example, a flash drive and/or other suitable media.

Also, terms such as "coupled" and "connected" in this specification may be used in derivative forms and describe structural relationships between constituent elements. It should be understood that these terms are not intended to be synonymous with each other. Specifically, in certain embodiments, "connected" may be used to indicate that two or more elements are in direct contact with each other. Unless the context clearly dictates otherwise, the term "coupled" may be used to mean that two or more elements are in direct or indirect contact with each other (with other intervening elements therebetween), or that two or more elements act or interact with each other (such as with a signal send/receive or function call correlation, etc.), or both.

[appropriate environment]

FIG. 1 is a diagram showing an example of an environment in which a robot system 100 according to an embodiment of the present disclosure operates. The robot system 100 includes one or more units such as robots configured to execute one or more tasks.

For the example shown in FIG. 1, the robotic system 100 may have an unloading unit 102, a transfer unit 104, a conveying unit 106, a stacking unit 108, or a combination thereof within a warehouse or distribution/conveying hub. Among these units, for example, a robot that operates an operation object by a robot arm called an unpacking robot, a picking robot, a picking robot, an end effector, or the like can be mentioned as the robot that operates the operation object. In addition, each unit in the robot system 100 executes a control sequence for combining a plurality of operations, such as one or more operations, to perform a plurality of operations, such as a truck, a truck, or the like, for the operation object to be stored in a warehouse. Unloading is performed, and the operation object is unloaded from the storage location, for example, the operation object is moved between containers, and the operation object is stacked on a truck, a truck, or the like for transporting the operation object. That is, the "job" here is a concept including various operations and actions for the purpose of transferring an operation object from "a certain position" to "another certain position".

More specifically, "work" includes an operation (eg, movement, orientation, attitude change, etc.) of causing the operation object 112 to move from the start position 114 of the operation object 112 to the work position 116 , and the operation of the operation object 112 set at the start position 114 to the work position 116 . The grip transformation of the operation object 112 at the grip transformation position 118 in the middle of the movement path of the operation object 112 , the scanning of the operation object 112 for acquiring the identification information of the operation object 112 , and the like.

In addition, for example, the unloading unit 102 can be configured to move the operation object 112 from a certain position within a vehicle (eg, a truck) to a certain position on a conveyor belt. Further, the transfer unit 104 can be configured to transfer the operation object 112 from a certain position (for example, a pickup area including a start position) to another position (for example, a placement area including a work position on the conveying unit 106 ), and in this movement In the middle of the path, the operation object 112 is grasped and transformed. Further, the conveying unit 106 can transfer the operation object 112 from the area related to the transfer unit 104 to the area related to the stacking unit 108 . Further, the stocking unit 108 can transfer the operation object 112 from the transfer unit 104 to a storage position (for example, a predetermined position of a shelf such as a shelf in a warehouse), for example, by moving a pallet or the like on which the operation object 112 is placed.

In addition, in the description herein, an example in which the robot system 100 is applied to a conveying center is described, however, it is understood that the robot system 100 is used to perform manufacturing, assembly, packaging, health care, and/or other types of automated operations. It can also be configured to execute jobs in other environments/other purposes. It will be understood that the robotic system 100 may include other units such as manipulators, service robots, modular robots, etc. not shown in FIG. 1 . For example, the robotic system 100 may include, for example, an unloading unit from pallets for transferring objects 112 from cages or pallets to conveyors or other pallets, a container switching unit for transferring objects 112 between containers, A packing unit for packing the operation objects 112, an arrangement and transformation unit for grouping according to the characteristics of the operation objects 112, and a pick-up unit for performing various operations on the operation objects 112 according to the characteristics of the operation objects 112 (for example, arrangement transformation, grouping and/or transfer) , a pallet for accommodating the operation object 112, a self-propelled trolley unit (eg, an automatic pallet truck, an unmanned pallet truck, etc.) for moving a rack, or a combination thereof.

[suitable system]

FIG. 2 is a block diagram showing an example of the hardware configuration of the robot system 100 according to the embodiment of the present disclosure. The robot system 100 includes, for example, one or more processors 202, one or more storage devices 204, one or more communication devices 206, one or more input-output devices 208, one or more operation devices 212, one or more transfer motors 214, one or more sensors 216, or any of these. Combination of electronic or electrical equipment, etc. These electronic or electrical devices are coupled to each other by wired and/or wireless connections.

近场通信(NFC)等)、物联网(IoT)协议(例如，NB-IoT，LTE-M等)和/或其他基于无线通信协议的无线连接。Additionally, the robotic system 100 may be, for example, a system bus, a Peripheral Component Interconnect (PCI) bus or a PCI-express bus, a Hyper Transport or Industry Standard Configuration (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus ( USB), IIC (I2C) bus, or a bus such as the Institute of Electrical and Electronics Engineers (IEEE) Standard 1394 bus (also known as "FireWire"). Additionally, the robotic system 100 may include, for example, bridges, adapters, controllers, or other signal-related devices to provide wired connections between electronic or electrical devices. Additionally, the wireless connection may be, for example, a cellular communication protocol (eg, 3G, 4G, LTE, 5G, etc.), a wireless local area network (LAN) protocol (eg, a faithful wireless communication environment (Wireless Fidelity (WIFI)), peer-to-peer or device-to-device Device communication protocols (for example,

Near Field Communication (NFC), etc.), Internet of Things (IoT) protocols (eg, NB-IoT, LTE-M, etc.) and/or other wireless connectivity based on wireless communication protocols.

Processor 202 may include a data processor (eg, central processing unit (CPU), special purpose computer, and/or board) configured to execute commands (eg, software commands) stored in storage device 204 (eg, computer memory) upload server). The processor 202 may execute program commands to control/interact with other devices, thereby causing the robotic system 100 to perform control sequences including various actions, jobs, and/or operations.

Storage device 204 may include a non-transitory computer-readable storage medium having program commands (eg, software) stored thereon. Storage device 204 may enumerate, for example, volatile memory (eg, cache and/or random access memory (RAM)) and/or non-volatile memory (eg, flash memory and/or disk drives), portable memory drives, and/or or cloud storage devices, etc. The storage device 204 may also be used to further store processing results and/or specified data/thresholds, and provide access, such as may store master data 252 that includes information about the operational objects 112 .

The master data 252, as information related to the operation object 112, may include, for example, size, shape, mass, center of gravity, position of the center of mass, templates related to postures and contours, model data for recognizing different postures, and inventory management units ( SKU: Stock Keeping Unit), color scheme, image, identification information, logo, expected position of the operating object, expected sensor measurements (eg, force, torque, pressure, physical quantities related to contact reference values), or a combination thereof .

Additionally, storage device 204 may store, for example, tracking data 254 of object 112 . The tracking data 254 may include a log of the object being scanned or manipulated, photographed data (eg, photographs, point clouds, real-time video, etc.), the position and/or pose of the manipulation object 112 at more than one location.

Communication device 206 may include, for example, circuitry configured to communicate with external or remote devices over a network, receivers, transmitters, conditioners/demodulators (modems), signal detectors, signal encoders/decoders, connector ports , network card, etc. Additionally, the communication device 206 may be configured to send, receive and/or process electrical signals in accordance with one or more communication protocols (eg, Internet Protocol (IP), wireless communication protocols, etc.). For example, robotic system 100 may use communication device 206 to exchange information between units or with external systems or external devices for appropriate purposes such as reporting, data collection, analysis, and troubleshooting.

The input-output device 208 acts as a user interface device configured to input information and instructions from the operator, communicate information with the operator, prompts, etc., and may include, for example, a keyboard, mouse, touch screen, microphone, user interface (UI) sensors (eg, a camera for receiving motion commands), input devices such as wearable input devices, and output devices such as display 210, speakers, haptic circuits, haptic feedback devices. Additionally, the robotic system 100 may use the input-output device 208 to communicate with the operator when performing actions, tasks, operations, or combinations thereof.

The robotic system 100 may include physical or structural components (eg, robotic manipulators, robotic arms, etc., hereinafter, simply referred to as “constructive components”) connected, for example, by chains or joints to perform operations including, for example, movement or rotation of the object 112 , etc. Displaced control sequence. Such physical or structural components and links or joints may be configured to operate end effectors (eg, grippers, hands, etc.) that perform one or more tasks (eg, grip, rotate, weld, assemble, etc.) on the robotic system 100 . Additionally, the robotic system 100 may include a motion device 212 (eg, motor, actuator, wire, artificial muscle, electrical active polymer, etc.), and the transfer motor 214 configured to transfer the unit from a certain position to another position.

Additionally, the robotic system 100 may include a sensor 216 configured to acquire information for performing a job of manipulating the construction member and/or the transfer unit. Sensors 216 are devices configured to detect or measure one or more physical properties of robotic system 100 (eg, the state, condition, location, etc. of one or more structural components, links or joints, etc.) and/or properties of the surrounding environment, and may include, for example, accelerometers , gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

Further, the sensors 216 may include one or more photographing devices 222 (eg, visible and/or infrared cameras, two- and/or three-dimensional imaging cameras, lidar or radar isometry devices, etc.) configured to detect the surrounding environment. The camera device 222 may generate digital images and/or displays of the inspection environment, such as point clouds, in order to obtain visual information, eg, for automated inspection, robot guidance, and other robotic applications.

In addition, the robot system 100 can process digital images, point clouds, ranging data, etc. via the processor 202, for example, to identify the operation object 112 in FIG. 1 , the start position 114 in FIG. 1 , the work position 116 in FIG. Reliable reference for the grip transition position 118 between the work positions 116 , the posture of the operation object 112 , the posture of the operation object at the start position 114 , etc., the reliable reference for the height of the operation object 112 , or a combination thereof.

Furthermore, in order to operate the operation object 112 , the robot system 100 converts a designated area (for example, a pickup area in a truck or on a conveyor belt, a placement area on the conveyor belt for arranging the operation object 112 , and a grip conversion for the operation object 112 ). The image of the area for arranging the operation object 112 in the container, the area on the pallet for stacking the operation object 112, etc.) is acquired and analyzed by various units, and the operation object 112, its start position 114, The working position 116, the grip changing position 118, and the like. In addition, the imaging device 222 includes, for example, one or more cameras configured to generate images of a pickup area, a drop area, and an area set therebetween for grasping and transforming the operation object 112 .

In addition, the imaging device 222 may include one or more distance measuring devices such as lidar or radar configured to measure the distance from the operation object 112 supported at a predetermined position on the upstream side (previous stage) upstream of the grip change position 118 . The robot system 100 can determine the start position 114 , the work position 116 , the grip transformation position 118 , the relative posture, the actual height of the operation object 112 , a reliable reference, and the like based on the acquired image and/or distance measurement data.

In addition, in order to scan the operation object 112 , the imaging device 222 may be configured to be, for example, between the start position 114 and the operation position 116 (preferably in the previous stage of the grip change position 118 ) during the conveyance or movement of the operation object. one or more scanners 412, 416 (eg, barcode scanners, QR code scanners (registered trademark), etc.) that scan the identification information of the operation object 112 (eg, the logo 332 of FIG. 3A and/or FIG. 3C described later), etc., Refer to FIG. 4A and FIG. 4B described later). Also, the robotic system 100 can create or acquire control sequences for providing one or more portions of the manipulation object 112 for the one or more scanners 412 .

Further, the sensor 216 may include, for example, a position sensor 224 (eg, a position encoder, a potentiometer, etc.) configured to detect the position of a structural member, a chain, or a joint. The position sensor 224 is used to track the position and/or orientation of the structural component, chain or joint during execution of the work.

Additionally, sensors 216 may include, for example, contact sensors 226 (eg, pressure sensors, force sensors, strain gauges, piezoresistive/piezoelectric sensors, capacitive sensors, elastic resistance sensors, other tactile sensors, etc.). The touch sensor 226 can measure the characteristic corresponding to the grip of the end effector of the manipulation object 112 . Thereby, the contact sensor 226 can output a contact reference indicating a quantitative measurement value (eg, measured force, moment, position, etc.) corresponding to the degree of contact between the end effector and the operation object 112 . The "contact reference" may include, for example, a read value of one or more forces or moments related to the force applied by the end effector to the manipulation object 112 .

[Judgment of reliable reference for initial posture]

FIGS. 3A , 3B, and 3C are perspective views schematically illustrating a first posture 312 , a second posture 314 , and a third posture 316 as examples of various postures (positions and orientations) of the operation object 302 , respectively. The robotic system 100 may, for example, process two-dimensional images, three-dimensional images, point clouds, and/or other photographed data from the photographing device 222 in order to recognize the pose of the manipulation object 302 . In addition, the robot system 100 may analyze the photographing data of one or more photographing devices 222 toward the pickup area, for example, in order to recognize the initial posture of the manipulation object 302 .

In order to recognize the posture of the operation object 302, the robot system 100 firstly analyzes the pattern image of the operation object 302 in the photographed data based on the prescribed recognition mechanism, recognition rules, and/or templates related to the attitude and outline, and recognizes the outline of the operation object 302. (eg surrounding edges or surfaces), or group them. More specifically, the robot system 100, for example, based on a template of the outline and posture of the master data 252, can identify the entire outline of the manipulation object with the color, brightness, depth/position, and/or combinations thereof, and changes in their values. The grouping of contours corresponding to the pattern of (eg, whether it is the same value, whether it varies in a known scale/pattern, etc.).

If the contours of manipulated objects 302 are grouped, robotic system 100 is able to identify, for example, one or more surfaces, edges, and/or points and poses of manipulated objects 302 in the grid or coordinate system used by robotic system 100 .

In addition, the robotic system 100 can identify one or more exposed surfaces (eg, the first exposed surface 304 , the second exposed surface 306 , etc.) of the manipulation object 302 within the captured data. Furthermore, the robot system 100 determines the shape of the outline of the operation object 302 and one or more dimensions (eg, length, width, etc.) based on, for example, the outline of the operation object 302 and imaging data related to calibration, or mapping data related to the imaging device 222 . , and/or height), the determined size is compared with the corresponding data in the master data 252, and the operation object 302 can be identified. Furthermore, the robot system 100 can identify which of the top surface 322 , the bottom surface 324 , and the outer peripheral surface 326 the exposed surface is based on the length, width, and height of the operation object 302 that recognizes the size of the exposed surface.

Additionally, the robotic system 100, for example, compares one or more indicia (eg, text, numbers, shapes, visual images, indicia, or combinations thereof) displayed on one or more exposed surfaces with one or more specified images within the master data 252 , the operation object 302 can be identified. In this case, the master data 252 may contain, for example, one or more images of the trade name, logo, design/image, or a combination thereof on the packaging surface of the manipulation object 302 . In addition, the robot system 100 compares a part of the photographed data (for example, a part within the outline of the operation object 302 ) with the master data 252 to recognize the operation object 302 , and also recognizes the operation based on a predetermined image pattern unique to the surface. The pose (especially the orientation) of the object 302 .

3A shows that the first exposed surface 304 (eg, the exposed surface facing upward) is the top surface 322 of the operation object 302 , and the second exposed surface 306 (eg, the exposed surface generally facing the source of the captured data) is the operation object 302 the first posture 312 in the case of one of the peripheral surfaces 326 .

In the identification of exposed surfaces, the robotic system 100 processes the captured data of FIG. 3A and can use a prescribed calibration or mapping function to determine the dimensions (eg, number of pixels) of the first exposed surface 304 and/or the second exposed surface 306 Values are mapped to real-world dimensions. Additionally, the robotic system 100 can compare the mapped size to the known/expected size of the manipulative object 302 within the master data 252, and based on the result, identify the manipulative object 302. Furthermore, the robotic system 100 can identify whether the first exposed surface 304 is the top surface 322 or the bottom after determining that a pair of intersecting edges of the boundary of the first exposed surface 304 match the length and width of the recognized operation object 302 . Surface 324. Likewise, the robot system 100 can recognize the second exposed surface 306 as the outer peripheral surface 326 after specifying that one edge of the second exposed surface 306 matches the height of the recognized operation object 302 .

Additionally, the robotic system 100 can process the captured data of FIG. 3A and identify one or more markers unique to the surface of the manipulation object 302 . In this case, the master data 252 may include one or more images and/or other visual characteristics (eg, color, size, size, etc.) of the surface and/or characteristic indicia of the manipulation object 302 described above. As shown in FIG. 3A, the manipulation object 302 has an "A" on the top surface 322, so the robotic system 100 can identify the manipulation object 302 as the manipulation object stored in the master data 252, and thus the first exposed surface 304 as the manipulation Top surface 322 of object 302 .

In addition, the master data 252 may contain the identification 332 as identification information of the operation object 302 . More specifically, the master data 252 may contain an image and/or encoded message of the identification 332 of the manipulation object 302, the location 334 of the identification 332 for a set of surfaces and/or edges, one or more visual characteristics thereof, or the like. The combination. As shown in FIG. 3A , the robotic system 100 can identify the second exposed surface 306 as the peripheral surface 326 based on the presence of the markers 332 , and/or their matching positions with the locations 334 of the markers 332 .

In addition, FIG. 3B shows a second posture 314 in which the operation object 302 is rotated 90 degrees around the vertical axis in the direction B of FIG. 3A . For example, the reference point "α" of the operation object 302 may be the lower left corner of FIG. 3A and the lower right corner of FIG. 3B . As a result, the top surface 322 of the operation object 302 is recognized as a different orientation in the shooting data compared to the first gesture 312 , and/or the outer peripheral surface 326 of the operation object 302 with the identification 332 cannot be visually recognized.

The robotic system 100 may recognize various gestures of the manipulation object 302 based on the particular orientation of the identification 332 of one or more visual features. For example, the size that matches the known length of the operation object 302 extends horizontally in the shooting data, the size that matches the known height of the operation object 302 extends vertically in the shooting data, and/or the size that matches the known height of the operation object 302 extends vertically in the shooting data. The first posture 312 and/or the third posture 316 can be determined in the case that the dimension matching the width extends along the depth axis in the shooting data. Likewise, for the robotic system 100, a dimension that matches the width extends horizontally in the shot data, a dimension that matches the height extends vertically in the shot data, and/or a dimension that matches the length extends along the depth in the shot data With the shaft extended, the second posture 314 can be determined.

In addition, the robot system 100 can determine that the operation object 302 is in the first posture 312 or the second posture 314 based on, for example, the orientation of visual markers such as "A" shown in FIGS. 3A and 3B . Further, the robotic system 100, for example, in the case where the identification 332 of the manipulation object 302 is visually recognized with the marking "A" (ie, on a different surface), based on the visual recognition marking to be visually recognized in the combination of the surfaces, It can be determined that the operation object 302 is in the first posture 312 .

Furthermore, FIG. 3C shows a third posture 316 in which the operation object 302 is rotated 180 degrees around the horizontal axis along the direction C of FIG. 3A . For example, the reference point "α" of the operation object 302 is the lower left front corner of FIG. 3A and the upper left rear corner of FIG. 3C . Therefore, compared to the first gesture 312, the first exposed surface 304 is the bottom surface 324 of the operation object, and in addition, both the top surface 322 with the logo 332 and the outer peripheral surface 326 of the operation object 302 cannot be visually recognized.

As described above, the robotic system 100 can recognize, based on the size determined from the image data, that the manipulation object 302 is in the first pose 312 or the third pose 316, where the marking (eg, "A") of the top surface 322 can be seen Next, it can be recognized that the operation object 302 is in the first gesture 312 . In addition, the robotic system 100 can recognize that the manipulation object 302 is in the third pose 316 if the indicia on the bottom surface (eg, an example of the manipulation object's identification 332 ) can be seen.

In addition, when the posture of the operation object 302 is determined, the situation in the real world may affect the accuracy of the determination. For example, due to light conditions, reflections and/or shadows, etc., sometimes reduce the visibility of surface markings. Furthermore, depending on the actual orientation of the operation object 302 , the exposure or visual recognition angle of one or more surfaces may be reduced, so that any mark on the surface may not be recognized. Therefore, the robot system 100 can calculate a reliable reference related to the determined posture of the operation object 302 .

In addition, the robot system 100 can calculate a reliable reference based on the deterministic interval (interval) associated with the dimension measurement in the image in the captured data. In this case, as the distance between the operation object 302 and the imaging source (for example, the imaging device 222 ) decreases, and/or as the measured edge of the operation object 302 becomes orthogonal to the direction radiated from the imaging source The deterministic spacing can be increased away from the direction parallel to the radial direction. In turn, the robotic system 100 may calculate a reliable benchmark based on, for example, the degree of match between the markings or designs in the captured data and known markings/designs in the master data 252 . Furthermore, the robot system 100 can measure the degree of overlap or deviation between at least a part of the captured data and a predetermined mark/image.

In this case, the robot system 100 measures the maximum repetition and/or minimum deviation according to the mechanism-dependent minimum mean square error (MMSE), etc., and can identify the manipulation object 302 and/or its orientation, and further, based on the obtained repetition/deviation degree, a reliable benchmark can be calculated. In addition, the robot system 100 can calculate the movement path of the operation object 302 in the control sequence based on the obtained reliable reference. In other words, the robot system 100 can appropriately move the operation object 302 based on the obtained reliable reference.

[system operation]

FIG. 4A is a plan view showing an exemplary job 402 performed by the robotic system 100 according to one embodiment of the present disclosure. As described above, job 402 is an example of a control sequence performed by robotic system 100 (eg, performed by the unit or the like shown in FIG. 1 ). As shown in FIG. 4A , for example, the job 402 may include: moving the operation object 112 from the pick area containing the start position 114 to the drop area containing the work position 116 via the grip transformation position 118 ; In the process, the operation object 112 is scanned; and the operation object 112 is grip-transformed (changing the grip position) at the grip-transformation position 118 . Accordingly, the robot system 100 can update the tracking data 254 at any time by adding the scanned operation object 112 to the tracking data 254 of the operation object 112 , removing the operation object 112 from the tracking data 254 , and/or evaluating the operation object 112 .

In addition, the robotic system 100 may include, in order to identify and/or determine the starting position 114, a method for photographing the pick-up area (more specifically, for example, the area designated for pallets or totes used for component deployment, and/or the receipt of conveyor belts). The scanner 412 (an example of the imaging device 222 ), such as a 3D vision scanner 412 (an example of the imaging device 222 ), which is directed toward the pickup region, can acquire imaging data of the specified region. Also, the robot system 100 may perform computer image processing (visual field processing) of photographed data, for example, in order to recognize various manipulation objects 112 located in a designated area through the processor 202 .

In addition, the robot system 100 selects the operation object 112 for executing the operation 402 from the recognized operation object 112, for example, based on a predetermined selection criterion, and/or selection rule, and/or a template related to a posture and an outline. The operation object 112 may further process the photographed data in order to determine the start position 114 and/or the initial posture.

In order to identify and/or determine the work position 116 and the grip change position 118, the robot system 100 may include a placement area and other specified areas for imaging (more specifically, for example, a pallet or a large box, and/or The other scanner 416 (an example of the imaging device 222 ) facing the area designated by the transmission side area of the conveyor belt, etc., can acquire imaging data of the designated area. In addition, the robot system 100 may perform computer image processing (field of view processing) of the captured data, for example, in order to recognize the work position 116 for arranging the operation object 112 , the grip transformation position 118 , and/or the posture of the operation object 112 by the processor 202 . In addition, the robotic system 100 may identify and select the work position 116 and the grip transformation position 118 (based on or not based on the photographing results) based on prescribed criteria or rules for stacking and/or arranging the plurality of manipulation objects 112 .

Among them, the scanner 416 can be configured to face the horizontal direction to scan the mark existing on the surface of the operation object 112 which is adjacent to it (for example, at a height corresponding to the height of the corresponding scanner(s)) and faces vertically . Furthermore, the scanner 416 may be arranged in the vertical direction in order to scan the marks existing on the surface of the operation object 112 which are vertically and horizontally oriented. Also, the scanners 416 may be disposed opposite to each other to scan both sides of the operation object 112 located between the scanners 416 .

In addition, the robotic system 100 may position the object 112 at the provided location based on the position and/or scanning direction of the scanner 416, and/or in such a manner that the scanner 416 can scan more than one surface/portion of the object 112. The operation object 112 operates. Furthermore, the robot system 100 may include, for example, the imaging device 222 (see FIG. 4B ) configured to scan with the scanner 416 and measure the height position of the bottom surface 324 of the operation object 112 whose supporting position is known.

In this way, using the identified start position 114 , grip transition position 118 , and/or work position 116 , the robotic system 100 can operate one or more structural components of each cell (eg, the robotic arm 414 and/or the end effector) in order to perform the job 402 . device). Thus, the robotic system 100 can, for example, by the processor 202 create or obtain a control sequence corresponding to one or more actions performed by the corresponding unit for performing the job 402 .

For example, the control sequence associated with the transfer unit 104 may include: deploying the end effector at an approaching location (eg, a location/location for configuring an end effector for holding the manipulation object 112 ); grasping the manipulation object 112 ; placing the manipulation object 112 Lift; move the operation object 112 from the start position 114 to the provided position/pose for the scanning operation; perform a grip transformation (change the grip position) on the operation object 112 at the grip transformation position 118; move the operation object 112 from the start position 114 according to the It is necessary to move to the working position 116 via the grip change position 118 ; lower the operation object 112 ; and release the grip of the operation object 112 .

Additionally, the robotic system 100 may determine a sequence of commands and/or settings for operating the robotic arm 414 and/or one or more operational devices 212 of the end effector, creating or acquiring control sequences. In this case, the robotic system 100 , eg, using the processor 202 , may calculate commands and/or settings for manipulating the end effector and the motion device 212 of the robotic arm 414 to place the end effector in proximity around the starting position 114 . Position, hold the operation object 112 by the end effector, place the end effector in the scanning position, hold the proximity position around the transformation position 118, place the end effector in the proximity position around the work position 116, and execute the operation object 112 from the end release on the device. Thereby, the robot system 100 can operate the operation equipment 212 according to the control sequence determined by the command and/or setting, and can execute the operation for completing the work 402 .

In addition, the robotic system 100 can create or acquire control sequences based on reliable fiducials related to the pose of the manipulated object 112 . In this case, the robot system 100 can place the end effector at various positions for picking up in order to hold or cover different surfaces, for example, according to a reliable reference related to the posture, and calculate various provided positions/provided positions related to the operation object 112 . gestures, or combine them.

As an example, the operation object 112 is the operation object 302 in the first posture 312 of FIG. 3A (in this case, the top surface 322 of the operation object 302 is exposed upward as a whole), and the posture-related reliability benchmark is high (ie, If the degree of certainty exceeds the sufficiency threshold, the probability that the determined pose is correct is high), the robotic system 100 can create or acquire the first control sequence 422 including the first approaching position 432 and the first providing position 442 . At this point, for example, with sufficient certainty that the top surface 322 of the manipulation object 302 is facing upward (ie, the bottom surface 324 with the object identification 332 of FIG. 3C is facing downward), the robotic system 100 may calculate the A first control sequence 422 of a first access location 432 placed on the top surface 322 of the manipulation object 302.

As a result, the robot system 100 can hold the manipulation object 112 by the end effector contacting/covering the top surface 322 of the manipulation object 302 so that the bottom surface 324 of the manipulation object 302 is exposed. Additionally, the robotic system 100 may calculate a first control sequence 422 that includes a first control sequence 422 for causing the manipulation object 112 to be the first providing location 442 that scans the upward facing scanner 416 located on the bottom surface 324 for the marking 332 .

On the other hand, in situations where the pose-related reliable benchmark is low (ie, the degree of certainty is less than the sufficiency threshold, the probability that the determined pose is correct is low), the robotic system 100 may create or acquire a second A second control sequence 424 (ie, different from the first control sequence 422 ) is proximate to location 434 and one or more second provision locations 444 . At this time, the robotic system 100 may, for example, measure the size of the manipulation object 112 and compare it with the master data 252, and determine that the manipulation object 302 is the first posture 312 of FIG. 3A or the third posture 316 of FIG. the level exceeds the specified threshold).

However, in the robot system 100 , it may be difficult to image and process the marks printed on the surface of the operation object 112 , and as a result, the determined attitude-related reliability criterion is smaller than the sufficiency threshold. In other words, the robotic system 100 is sometimes not able to adequately determine whether the upwardly facing exposed surface of the manipulation object 302 is its top surface 322 (eg, first pose 312 ) or its bottom surface 324 (eg, third pose 316 ).

In this case, due to the low reliability fiducial (low degree of certainty), the robotic system 100 may calculate the inclusion for abutment with one peripheral surface 326 of the manipulated object 302 of FIG. 3A (eg, relative to the top surface of the manipulated object 302 ). 322 and/or the bottom surface 324 are oriented and/or facing in a parallel direction) a second control sequence 424 that configures the second access position 434 of the end effector.

As a result, the robot system 100 can hold the manipulation object 112 by the end effector which contacts/covers one outer peripheral surface 326 of the manipulation object 302 and exposes both the top surface 322 and the bottom surface 324 of the manipulation object 302 . Additionally, robotic system 100 may provide or place top surface 322 and bottom surface 324 of object 302 simultaneously or sequentially in front of scanner 416 (eg, within and/or facing the scanning field). With the object 112 in the scanning position, the robotic system 100 may use the scanner 416 (eg, at least the scanner 416 toward the top surface 322 and bottom surface 324 of the object 302 ), while simultaneously and/or continuously scanning the provided , obtain the identification 332 (multiple) of the operation object 302 on the surface.

In addition, the second control sequence 424 includes a second supply position 444 (which may be multiple) for making the initially downward facing surface (the bottom surface 324 of the manipulation object 302 ) horizontal and disposed directly above the upward facing scanner 416, And/or make the initially upward facing surface (top surface 322 of the manipulation object) vertical and placed directly in front of the horizontally facing scanner 416 . The second control sequence 424 includes the action of reorienting/rotating (eg, the action shown by the dashed hollow circle) in order to provide the two provided positions/poses, whereby, using the orthogonally oriented scanner 416, scanning the top surface 322 and Bottom surface 324 both. Further, the robotic system 100 may, for example, continuously provide the top surface 322 of the object 302 to the scanner facing upward for scanning, and then rotate the object 302 by 90 degrees to provide the bottom surface 324 of the object 302 to the horizontally facing scanner for scanning. Scanner 416. At this time, in the case of failure to read the identification 332 of the operation object 302, the action of reorienting/rotating may be conditional, so that the robot system 100 implements the corresponding command.

Alternatively, as an example, the robotic system 100 may create or acquire a control sequence (not shown) for grasping/covering a peripheral surface 326 along the width of the manipulated object 302 with a low reliability benchmark. In this case, the robotic system 100 moves the object 302 between the horizontally opposed pair of scanners 416, providing a peripheral surface 326 of the object 302 along its length, eg, as shown in FIG. 3A, capable of scanning between the peripheral surfaces 326 Logo 332 on one. Further, details related to the control sequence based on the reliable reference will be described later with reference to FIG. 5A and FIG. 5B below.

In addition, the robot system 100 can be based on the two-dimensional or three-dimensional shape of the operation object 112 (hereinafter, the "operation object 302" is replaced by "the operation object 112") held by the end effector and the storage container 450 ( For example, the relevant information of the operation object 112 in a large box, a bucket, etc.), the control sequence is obtained again.

As an example, the robot system 100 grasps the size of the operation object 112 in any of the above-described first control sequence and second control sequence, for example. In addition, the other operation objects 112 already stored in the storage container 450 placed at the work position 116 and their dimensions are known, so the robot system 100 can obtain the spatial information of the vacant volume in the storage container 450 . The robot system 100 can calculate the spatial shape parameters of the operation object 112 when various postures of the operation object 112 held by the end effector change in two or three dimensions. Therefore, by comparing these spatial shape parameters with the spatial information in the storage container 450 , it is possible to optimally select a mode or plan for storing the operation object 112 in the storage container 450 with a higher packing density.

In this case, when the end effector accesses the storage container 450 , the robot system 100 may consider whether there is interference between the end effector and the operation object 112 already stored in the storage container 450 . In addition, compared with directly storing the grasped operation object 112 in the storage container 450 at the current orientation, changing the posture of the operation object 112 when the filling rate of the operation object 112 to the storage container 450 is high, the robot system 100 can create A control sequence including an operation to transform the grip of the operation object 112 in an optimally stored posture is acquired.

FIG. 4B is a front view showing an exemplary job 404 performed by the robotic system 100 according to one embodiment of the present disclosure. In this example, a plurality of operation objects 112 are mixed and placed on a pallet 464, and the pallet 464 starts to include in a state of being mounted on a self-propelled cart 462 such as an AGV (Automated Guided Vehicle), for example. Pickup area handling at location 114. In addition, in FIG. 4B , a state in which a plurality of operation objects 112 of the same shape are mixed and placed in an orderly manner is shown. However, it should be noted that a plurality of operation objects 112 of different sizes and shapes are randomly stacked on the pallet according to the actual unloading situation. More on board 464.

The pick-up area in which the pallet 464 is conveyed is imaged by the scanner 412, and the operation object 112 is selected in the same manner as in the description of FIG. 4A. The selected manipulation object 112 passes through the end effector provided at the front end of the robot arm 414 of the transfer unit 104 , in this example, grasps the top surface 322 of the manipulation object 112 , and scans with the scanner 416 to acquire the marker 332 . For example, the robot system 100 can compare the information of the identifier 332 of the manipulation object 112 with the master data 252 to grasp the information including the size of the manipulation object 112 .

On the other hand, even the operation objects 112 with the same identification 332 may actually have different sizes (especially heights). Therefore, when scanning the operation object 112, the robot system 100 measures the distance to the bottom surface 324 of the operation object 112 by the distance measuring device 466 (an example of the imaging device 222) installed on or near the ground of the work space, for example. At this time, when the movement of the operation object 112 is temporarily stopped during scanning, the distance to the bottom surface 324 of the operation object 112 may be measured during the temporary stop. In addition, FIG. 4B shows the case where the distance measuring device 466 performs the measurement immediately after the operation object 112 is unloaded (destacked) from the pallet 464. However, as long as the timing of the measurement is compared with the control sequence The conversion position 118 may be performed at the upstream position, and is not particularly limited.

In this example, the robot system 100 can grasp the height position (holding level) of the top surface 322 of the operation object 112 at the time of measurement by a control sequence or appropriate position measurement. Thereby, the height 112h of the operation object 112 can be obtained by acquiring the measured value of the distance to the bottom surface 324 of the operation object 112 . That is, the robot system 100 can acquire the measurement data of the bottom surface 324 of the operation object 112 by the distance measurement device 466, and calculate the height 112h based on the acquired measurement data and the height position (holding level) of the top surface 322 of the operation object 112. When the height 112h is different from the value stored in the master data 252 of the operation object 112 , the robot system 100 may replace the master data 252 or add and update the master data 252 .

In this way, after determining the actual size of the operation object 112 , the robot system 100 can calculate the spatial shape parameters of the posture when the operation object 112 is grasped from various directions. Then, the robot system 100 compares these spatial shape parameters with the spatial information in the storage container 450 placed at the working position 116, and optimally selects a plan or mode for storing the operation object 112 in the storage container 450 with a higher packing density .

At this time, when the end effector accesses the storage container 450 , the robot system 100 calculates whether interference occurs between the end effector and the operation object 112 already stored in the storage container 450 , and if interference may occur, this mode can be eliminated. In addition, the control sequence including the operation of changing the grip of the manipulation object 112 can be recreated, thereby changing the posture of the manipulation object 112 compared to directly storing the gripped manipulation object 112 in the storage container 450 with the current orientation at that moment. In this way, when the filling rate of the storage container 450 by the operation object 112 is high, the robot system 100 can change the control sequence 472 up to this point (equivalent to the first control sequence 422 or the second control sequence 424 in FIG. 4A and FIG. 4B ), Form an optimal storage posture.

Conversely, when the posture of the currently grasped manipulation object 112 is optimal in terms of storage efficiency, the robot system 100 does not change the control sequence 472 and stores the grasped manipulation object 112 at the work position 116 for storage of the conveying unit 106 . In the storage container 450, such as a bucket, mounted on the conveyor.

In addition, when the manipulation object 112 is grasped and transformed, the robot system 100 operates the manipulation object 112 according to the recalculated control sequence 474 . For example, the scanned operation object 112 is moved to the surrounding area of the grip transformation position 118, the end effector is directed in a predetermined direction, and the operation object 112 is set to the temporary placement posture, and the operation object 112 is loaded and placed on the temporary placement table 468 in this state, and the grip is released. The temporary placing table is not particularly limited. For example, the operation object 112 can be supported and held in a tilted state from the standpoint of placing the operation object 112 in such a way that at least both sides can be exposed, particularly in view of ease of gripping and stability during gripping. A pedestal is listed as a preferred example. The robot system 100 can change the grip of the operation object 112 by changing the orientation of the end effector and gripping a surface different from the surface gripped before the temporary placement.

The robot system 100 stores the manipulation object 112 whose grip has been converted into a storage container 450 such as a bucket placed on a storage conveyor or the like of the conveying unit 106 at the work position 116 . At this time, the end effector may not be directly and temporarily positioned, for example, it may be operated in a forward/backward/leftward/leftward/upward-downward-moving manner relative to the target position. In addition, a plurality of or multi-unit end effectors may be provided, and each end effector may be used for control according to the relationship with the size of the operation object 112 .

In addition, in the above, in order to perform actions related to the job 402, the robot system 100 can track the current position of the operation object 112 (eg, the coordinate set corresponding to the grid used by the robot system 100), and/or the current posture. For example, the robotic system 100 may track the current position/pose based on data from the position sensor 224 of FIG. 2 via, eg, the processor 202 . The robotic system 100 can configure one or more portions (eg, chains, joints) of the robotic arm 414 based on data from the position sensors 224 . The robot system 100 can further calculate the position/posture of the end effector based on the position and orientation of the robot arm 414 , thereby calculating the current position of the operation object 112 held by the end effector. In addition, the robotic system 100 may be based on the processing of readings from other sensors (eg, force readings or acceleration readings), operating commands/settings executed, and/or related timing, in a speculative mechanism. , or a combination of them to track the current location.

[Operational flow (control sequence based on reliable benchmarks)]

5A is a flowchart of a method 500 showing an example of a flow of operations of the robot system 100 according to an embodiment of the present disclosure. In order to perform the job 402 of FIG. 4A according to a reliable reference related to the determination of the initial pose of the manipulated object 112, the method 500 includes the steps of obtaining/computing a control sequence based on the reliable reference and executing it. Additionally, method 500 may be implemented by one or more processors 202 based on executing commands stored in one or more storage devices 204 .

In block 501 , the robotic system 100 may identify the scanning area of the one or more camera devices 222 of FIG. 2 . For example, the robotic system 100, through, for example, one or more processors 202, can identify the space scanned by one or more imaging devices 222, such as scanners 412, 416 in FIGS. 4A and 4B. Based on the orientation of the scanner 416, the robotic system 100 identifies areas of scanning that are oriented in opposite and/or orthogonal directions. As shown in FIGS. 4A and 4B , the scanners 416 may be disposed on both sides in the horizontal direction, or both sides in the vertical direction, opposite sides of each other, and/or facing each other. In addition, the scanners 416 may also be configured with one facing up or down, the other facing horizontally, etc. perpendicular to each other.

The robotic system 100 may, for example, identify the scanning area from the master data 252 . The master data 252 contains the camera device 222 and/or grid locations, coordinates, and/or other indicia representing the corresponding scan field. The master data 252 may be pre-determined based on the layout and/or physical configuration of the camera 222, the capabilities of the camera 222, environmental factors (eg, light conditions, and/or obstructions/configurations), or a combination thereof. Additionally, the robotic system 100 may perform a calibration procedure in order to identify the scanning area. For example, the robot system 100 uses the transfer unit 104 to configure known markers or codes in the position set, and can determine whether the corresponding imaging device 222 correctly scans the known markers. The robotic system 100 can identify the scan area based on the location of the known marker as the correct scan result.

In block 502, the robotic system 100 can scan the designated area. The robotic system 100 may use one or more imaging devices 222 (eg, scanners 412 of FIGS. 4A and 4B and/or other area scanners), for example, through commands/prompts sent by the processor 202, to generate pick areas and/or drop areas Capture data (eg, acquired digital images and/or point clouds) for more than one designated area. Capture data can be communicated by capture device 222 with one or more processors 202 . Thus, for subsequent processing, one or more processors 202 may receive representations of pick regions (eg, containing operands 112 before job execution), hold transform areas, and/or drop areas (eg, containing operands 112 after job execution) shooting data.

In block 504 , the robotic system 100 can identify the manipulation object 112 and associated locations (eg, starting position 114 of FIG. 1 , and/or work position 116 of FIG. 1 ), and/or the initial pose of the manipulation object 112 . In order to identify the contours (eg, surrounding edges and/or surfaces) of the manipulation object 112 , the robotic system 100 may analyze the captured data, eg, by the processor 202 based on pattern recognition mechanisms and/or recognition rules. The robotic system 100 may further identify groups of contours and/or surfaces of the manipulation objects 112 based on, for example, prescribed recognition mechanisms, identification rules, and/or pose, contour-related templates corresponding to the various manipulation objects 112 .

The robotic system 100 can, for example, identify the operation corresponding to the color, brightness, depth/position, and/or a combination of the patterns (eg, whether the same value, whether it changes at a known scale/pattern) of the entire outline of the operation object 112 . Grouping of outlines of objects 112 . In addition, for example, the robotic system 100 may recognize groupings of contours and/or surfaces of the manipulation object 112 based on templates, images, or combinations of prescribed shapes/poses specified in the master data 252 .

From the manipulation objects 112 identified in the pickup area, the robotic system 100 may select an manipulation object as the manipulation object 112 (eg, according to a prescribed sequence or set of rules, and/or a template of the manipulation object's outline). The robotic system 100 may select the manipulation object 112 based on, for example, a point cloud representing distances/positions corresponding to known positions of the scanner 412 . In addition, the robot system 100 can select, for example, an operation object 112 located at a corner/edge and having two or more surfaces exposed/displayed in the photographed result. Also, the robot system 100 can select the operation object 112 according to a predetermined pattern or sequence (for example, from left to right, from nearest to farthest, etc., relative to the reference position).

For the selected manipulation object 112, the robot system 100 may further process the photographed data in order to determine the start position 114 and/or the initial posture. For example, the robot system 100 maps the position of the operation object 112 in the photographed data (for example, a predetermined reference point related to the determined posture) to the position in the grid used by the robot system 100, so that the start position 114 can be determined, according to The specified calibration map map location.

The robot system 100 may process the photographed data of the placement area to determine the vacant space between the operation objects 112 . The robot system 100 can determine the vacant space by mapping the outline of the operation object 112 based on a predetermined calibration map that maps the position of the image to the actual position and/or coordinates used by the system. The robot system 100 may determine the vacant space as the space between the contours (or even the surfaces of the operation objects 112 ) of the operation objects 112 belonging to different groups. Then, the robot system 100 measures one or more dimensions of the vacant space, compares the measured dimensions with one or more dimensions of the operation object 112 (for example, the dimensions stored in the master data 252 ), and determines the appropriateness for the operation object 112 vacant space. Additionally, the robotic system 100 may select an appropriate/empty space as the work position 116 according to a prescribed pattern (eg, relative to a reference position, left to right, closest to furthest, bottom to top, etc.).

The robot system 100 may determine the work position 116 on the basis of not processing or processing the photographed data. For example, the robot system 100 may arrange the operation object 112 in the arrangement area according to a predetermined control sequence and position without imaging the area. In addition, for example, the robot system 100 may process the photographed data in order to perform a plurality of operations (for example, moving the related objects of the operation objects 112 located in the common layer/column of the stack, and the like, and the plurality of operation objects 112 ).

In block 522 , for example, the robotic system 100 may determine an initial pose (eg, an estimation of the pose in which the manipulation object 112 in the pickup area stopped) based on processing of photographed data (eg, from the scanner 412 ). The robot system 100 can determine the initial posture of the operation object 112 by comparing the outline of the operation object 112 with the outline of the predetermined pose template of the master data 252 (for example, by comparing pixel values). In the prescribed gesture template, for example, a configuration in which the contour of the involved operation object 112 may be different in the direction corresponding to the expected operation object 112 may be included. The robotic system 100 can identify the selected manipulator 112 and a set of previously associated contours of the manipulator 112 (eg, the first exposed surface 304 of FIGS. 3A and/or 3C, and/or the second exposed surface 306 of FIG. 3A ). etc., the edge of the exposed surface). The robot system 100 can determine the initial pose by selecting a pose template corresponding to the measurement with the least difference between the compared contours of the manipulation objects 112 .

For a further example, the robotic system 100 may determine the initial pose of the manipulation object 112 based on the physical size of the manipulation object 112 . The robot system 100 can estimate the physical size of the manipulation object 112 based on the size of the exposed surface acquired through the photographing data. The robotic system 100 determines lengths and/or angles respectively associated with the contours of the manipulated objects 112 in the captured data, and can then map the measured lengths using a calibration map, conversion table or process, prescribed equations, or a combination thereof Or convert to real world length or standard length. The robotic system 100 may use the measured dimensions in order to identify the manipulation object 112 and/or exposed surface(s) corresponding to the physical dimensions.

The robotic system 100 may compare the estimated physical size to a known set of dimensions (eg, height, length, and/or width) of the manipulative object 112 and its surface within the master data 252 to identify the manipulative object 112 and/or expose surface (multiple). The robotic system 100 may use the matched set of dimensions to identify the exposed surface(s) and corresponding poses. For example, the robotic system 100 may identify the exposed surface as the top surface 322 of the manipulation object 302 of FIG. 3A or the manipulation object 302 of FIG. 3B if the size of the exposed surface matches the length and width associated with the intended manipulation object 112 bottom surface 324 (eg, a pair of side surfaces). Based on the orientation of the exposed surface, the robotic system 100 may determine the initial pose of the manipulation object 112 (eg, the first pose 312 or the third pose 316 of the manipulation object 302 with the exposed surface facing up).

For example, the robotic system 100 may determine the initial pose of the manipulation object 112 based on one or more surfaces of the manipulation object 112, and/or one or more marked visual images thereof. The robotic system 100 may compare the pixel values of the connected set of contours to a prescribed marker-based pose template of the master data 252 . The marker-based gesture template may include, for example, one or more unique markers that anticipate variously oriented manipulation objects 112 . The robotic system 100 can determine the initial pose of the manipulated object 112 by selecting the surface, the surface orientation, and/or the corresponding one of the poses that are the measurements with the least difference in relation to the compared images.

In block 524 , the robotic system 100 may calculate a reliable reference relative to the initial pose of the manipulated object 112 . The robotic system 100 may compute a reliable reference as part of the process of determining the initial pose. For example, the reliable reference may correspond to a reference for the difference between the contour of the operation object 112 and the contour of the template selected above. Additionally, for example, the reliable reference may correspond to the above-described estimated physical dimension and/or angle-related tolerance levels. In addition, for example, the reliable reference may correspond to a reference for the difference between a visible marker within the shot data and the image of the template described above.

In block 506, the robotic system 100 may calculate a control sequence (eg, the first control sequence 422 of FIG. 4A, the second control sequence 424 of ) and the control sequence 474 of the grip transformation operation of the operation object 112 shown in FIG. 4B .

For example, the robotic system 100 may create or acquire control sequences by computing a sequence of commands or settings associated with operating the robotic arm 414 and/or the end effector 212 of FIGS. 4A and 4B , or a combination thereof. For some jobs, the robotic system 100 calculates control sequences and setpoints for manipulating the robotic arm 414 and/or the end effector to move the manipulated object 112 from the start position 114 to the work position 116 via the grip transition position 118 as needed. Robotic system 100 may implement control sequence mechanisms (eg, processes, functions, equations, algorithms, computer-generated/readable models, or combinations thereof) that are configured to compute paths of movement within space.

For example, the robotic system 100 moves the manipulated object 112 from the start position 114 via the grip transformation position 118 through one or more provided poses/positions (eg, one or more corresponding scan positions for the end effector) to the work position 116 as needed. , A* algorithm, D* algorithm, and/or other grid-based searches may be used to calculate movement paths through space. The control sequence mechanism uses further processing, functions, or equations, and/or mapping tables, to convert the movement path into a sequence of commands or settings related to operating the device 212, or a combination thereof. When using the control sequence mechanism, the robotic system 100 may manipulate the robotic arm 414 and/or the end effector to calculate a control sequence along the calculated movement path for the manipulated object 112 .

The robotic system 100 may selectively create or acquire control sequences based on reliable benchmarks. The robotic system 100 may calculate, based on a reliable reference, an approach location (eg, the first approach location 432 of FIG. 4A and/or the second approach location 434 of FIG. 4A ), one or more scan locations (eg, the first provided location of FIG. 4 ) 442 and/or the second providing position 444 of FIG. 4), or a control sequence of a combination thereof. For example, the robotic system 100 may calculate the proximity position and/or scan position relative to a metric (eg, a metric of performance, and/or a metric of scanning) based on a comparison of a reliable baseline to an adequacy threshold. A scan position is a position in which the end effector is configured in a manner that provides one or more surfaces of the manipulated object 112 in front of one or more corresponding object scanners 416 that scan the one or more identifiers 332 of the manipulated object 112 (ie, its scanning field).

In block 532, the robotic system 100 may, for example, by the processor 202, calculate the set of available proximity locations. The available access positions may correspond to positions that are open or unoccupied about the starting position 114 in which the end effector is fully deployed. In addition, the robot system 100 may place an end effector at a selected approach position in order to contact and hold the operation object 112 without interfering with other operation objects 112 .

For example, the robotic system 100 may calculate the set of available proximity positions by calculating the separation distance between the contour of the manipulation object 112 and the contours of the adjacent manipulation objects 112 . The robotic system 100 may compare the separation distance to a prescribed set of distances corresponding to the physical size/shape of the end effector, and/or various orientations. The robotic system can identify each of the available approach positions where the corresponding separation distance exceeds a prescribed set of distances corresponding to the size of the end effector.

In determination block 534, the robotic system 100 may compare the reliability benchmark to one or more adequacy thresholds to determine if they are satisfied. In the event that the reliable benchmark shown in block 536 satisfies the adequacy threshold (eg, where the reliable benchmark exceeds the desired adequacy threshold), the robotic system 100 may calculate a control sequence (eg, the first control sequence) based on the metric of performance. 422). In the event that the reliable benchmark satisfies the sufficiency threshold, the robotic system 100 may infer that the initial posture is appropriate, regardless of the possibility of scanning the corresponding scan of at least one identifier 332 of the manipulation object 112 and/or the possibility that the initial posture is incorrect metric to calculate the control sequence.

As an example, the robotic system 100 may calculate candidate solutions in block 542 . The candidate solutions may each be an example of a control sequence corresponding to a unique combination of an available approach position and a scan position (eg, a corresponding provided position/orientation in relation to the operation object 112 ). The robotic system 100 rotates the position(s) 334 of the marker 332 within the master data 252 or the corresponding model/pose so that the position 334 of the marker 332 can be calculated from the initial pose. The robotic system 100 may remove available access locations that are covered by the end effector over the location 334 of the marker 332 (eg, placed directly above, in front of, and/or within a threshold distance).

The robotic system 100 may compute candidate solutions for each of the remaining available proximity locations within the set (eg, the results of the computation of block 532). For each candidate solution, the robot system 100 may further calculate a unique scanning position according to the available approach positions. The robotic system 100 may calculate the scan position based on the rotation and/or movement of the model of the manipulated object 112 such that the surface corresponding to the position 334 of the marker 332 is within the scan field and faces the corresponding scanner 416 . The robotic system 100 may rotate and/or move the model according to prescribed procedures, equations, functions, and the like.

In block 544, the robotic system 100 may calculate a measure of performance associated with each candidate solution. The robotic system 100 may calculate a measure of performance corresponding to the throughput (rate) associated with completing the job 402 . For example, a measure of performance may relate to the distance traveled by the operator 112 related to the candidate solution, the estimated time to travel, the number of changes to commands and/or settings related to the operating equipment 212, the completion rate (ie, complementary to the amount of lost pieces) or their combination. The robotic system 100 may use one or more measured or known data (eg, set/command-related acceleration/velocity, and/or grip surface and/or motion-related piece loss ratios), as well as prescribed computational procedures, Equations, functions, etc., calculate the values corresponding to the candidate control sequences.

In block 546, the robotic system 100 may select the candidate solution with the metric of maximum performance (ie, with the corresponding approach position) as the control sequence. For example, the robotic system 100 selects the highest completion rate, the shortest travel distance, the fewest number of command and/or setting changes, the fastest travel duration, or a combination thereof among the set of candidate solutions as a control sequence. alternate program. Thus, the robotic system 100 may select the available approach locations within the set corresponding to the highest performance metric as the approach locations.

If the comparison is made, the robot system 100 can calculate candidate solutions according to different criteria when the reliable benchmark does not satisfy the sufficiency threshold (eg, the reliable benchmark is smaller than the required sufficiency threshold). As represented by block 538, the robotic system 100 may calculate a control sequence (eg, the second control sequence 424) based on the scanned metrics. The metric for scanning, regardless of whether the initial posture of the at least one marker 332 of the manipulation object 112 is correct, is a value corresponding to the possibility that can be scanned without being covered by the end effector (for example, the value of two elements, or score/percentage for non-two elements).

For example, the robotic system 100 may prioritize scan metrics over performance metrics (eg, initially satisfy and/or assign greater importance) if the reliable benchmark does not meet the adequacy threshold. As such, the robotic system 100 may calculate one or more scan locations included in front of the one or more scanners 416 for providing the identification 332 of the at least one uncovered manipulated object 112 (ie, within the scan field and/or toward the corresponding scan instrument) control sequence.

5B is a flowchart showing an example of a sequence of actions of a robotic system according to an embodiment of the present disclosure, showing scan-based metrics for selectively calculating a control sequence (eg, for one or more positions of an end effector) ) flow chart 538.

In this example, calculating a control sequence based on the metric of the scan, as indicated by block 552 , may include calculating the set of positions of the exposed markers 332 . The robotic system 100 may calculate the set of positions of the exposed markers 332 for the initial pose of the manipulated object 112 (eg, the positions 334 of the markers 332 directly scannable by the end effector in the holding position). The robotic system 100 may calculate the position 334 of the exposed marker 332 associated with each of the available proximity positions. Assuming the initial posture is correct, at the corresponding approach position, the position 334 of the exposed marker 332 corresponds to the position 334 of the marker 332 of the manipulation object 112 that is not covered by the end effector.

As described in block 542 , the master data 252 may include a computer model or template documenting the locations 334 of the expected identifications 332 associated with each operand 112 (eg, offsets relative to the edges and/or images of one or more operands 112 ). displacement measurement). The robotic system 100 can calculate the set of positions of the exposed markers 332 based on rotating and/or moving a prescribed model/template within the master data 252 in a manner that matches the initial pose. The robotic system 100 may remove the proximity position that is covered by the end effector over the position 334 of the marker 332 (eg, placed directly above, in front of, and/or within a threshold distance). In other words, robotic system 100 may remove available proximity locations that are directly above, in front of, and/or within a threshold distance of location 334 of marker 332 .

In block 554 , the robotic system 100 may calculate the set of locations 334 of the alternative identifiers 332 . The robotic system 100 may calculate a set of locations 334 of the alternate signatures 332 for an alternate pose to the original pose. For each of the available proximity positions, the robotic system 100 may calculate an alternate pose, and for each alternate pose, the robotic system 100 may calculate the position of the alternate marker 332 . Thus, assuming that the initial posture is incorrect, at the corresponding approach position, the position of the alternative marker 332 may correspond to the position 334 of the marker 332 of the operation object 112 that remains uncovered by the end effector. Regarding the position 334 of the exposed marker 332, as described above, the robotic system 100 can calculate the position 334 of the replacement marker 332 based on the rotation and/or movement of the specified model/template within the master data 252 based on the replacement pose.

In block 556, the robotic system 100 may calculate the exposure likelihood associated with each approached position, each alternate pose, each manipulative object 112 identification 332, or a combination thereof. Exposure likelihood represents the likelihood that the identification of one or more manipulation objects 112 remains exposed and remains scannable by the end effector holding the manipulation object 112 from the corresponding approach position. The exposure possibility indicates two scenarios, a scene in which the initial posture is correct and a scene in which the initial posture is incorrect. In other words, the possibility of exposure may indicate the possibility that the identification of one or more operation objects 112 remains exposed and remains in a scannable state even when the initial posture is incorrect.

For example, robotic system 100 may calculate exposure likelihood as condition-related certainty as a specific instance of a probability value corresponding to a particular condition (eg, proximity location, alternate pose, identity of manipulated object 112, or a combination thereof). The robotic system 100 may calculate exposure likelihood based on combining (eg, by appending and/or multiplying) the certainty associated with the condition with the certainty/likelihood that the particular condition is true (eg, a value close to a reliable benchmark) sex. The robotic system 100 may calculate a likelihood of exposure based on additional certainty associated with each of the markers that are expected to be exposed when multiple markers are exposed at the considered proximity positions and/or the considered poses.

The robotic system 100 may calculate the exposure likelihood by combining the position of the exposed marker and the certainty value of the position of the surrogate marker for the potential respective poses, etc., relative to the considered approach position. For example, the robotic system 100 may calculate the likelihood of exposure using the position of the exposed marker and the relative certainty of the location of the alternate marker with inverse signs (eg, positive and negative). The robotic system 100 calculates the exposure likelihood by adding the magnitude of the two deterministic values, and/or adding the deterministic values with a sign. The overall size may represent the overall likelihood that one or more markers 332 of the operand 112 will remain scannable, and the signed/vector likelihood may represent the likelihood that one or more markers 332 of the operand 112 will remain in a scannable state if the initial pose is incorrect. Possibility to remain scannable. Thus, regardless of whether the initial posture is correct or not, the approach position is ideal when the overall size is large due to the similar possibility of the identification 332 representing the scannable manipulation object 112, with the possibility of a sign/vector close to zero.

In block 558, the robotic system 100 may select an approach location. The robotic system 100 as an approach position, at the set of exposed markers 332 (eg, a set of estimated positions of the markers 332 of the manipulator 112 assuming the initial pose is correct) and a set of alternative markers 332 (eg, assuming the initial pose is incorrect) Of the one or more sets of estimated positions of the identification 332 of the manipulation object 112), an available proximity position that includes the position 334 of the identification 332 uncovered may be selected. In other words, the robot system 100 can select the proximity position of the remaining at least one marker 332 of the exposed scannable manipulation object 112 regardless of the correctness of the initial posture. The robotic system 100 may select an available approach location with the largest overall size, and/or a signed/vector likelihood close to zero, etc. that matches the target conditions, and/or corresponds to the closest exposure likelihood.

The robotic system 100 may calculate the likelihood of scanning based on the exposure likelihood (eg, the likelihood that the identification 332 of the exposed manipulation object 112 can be successfully scanned). For example, the robotic system 100 may combine the likelihood of exposure with an evaluation value associated with the identification 332 of the corresponding exposed manipulation object 112 (eg, tracked percentage of successful scans, physical size, and/or type of identification 332). The robotic system 100 may select the available approach location corresponding to the highest scanning probability as the approach location.

The robotic system 100 compares the set of exposed indicia 332 to the set of alternate indicia 332, and can determine whether the set of exposed indicia 332 and the set of substitute indicia 332 contain the location of the opposite surface of the manipulated object 112 (eg, the first set of indicia 332). between pose 312 and third pose 316). Thus, the robotic system 100 may select an available approach location corresponding to a third surface (eg, one peripheral surface 326 of the manipulation object 302 ) that is orthogonal to the two opposing surfaces.

In block 560, the robotic system 100 may create or acquire a control sequence as a candidate based on the selected approach position in the event that the reliable reference does not satisfy the sufficiency threshold or the like. The robotic system 100 may compute a control sequence as a candidate that includes placing the marker 332 of the manipulation object 112 in one or more of the set of exposed markers 332 and the set of substituted markers 332 to provide position/orientation correspondence. more than one scan position associated with the end effector. In other words, the robot system 100 can calculate the control sequence of the candidates that can scan the operation object 112 regardless of whether the initial posture is correct or not.

The robotic system 100 may create or acquire control sequences that address candidates for the location 334 of the marker 332 in both the set of exposed markers 332 and the set of substituted markers 332 . For example, the robotic system 100 may calculate a candidate control sequence for the location of the identification 332 of the likelihood of existence of opposing and/or orthogonal surfaces. Therefore, the robot system 100 can cope with the reverse orientation based on the original orientation (eg, the contour of the manipulation object 112 is located in the reverse orientation based on the visual confirmation of the same position/angle), and/or other rotation orientations. As an illustrative example, referring again to FIGS. 3A and 3C , when the gripping position corresponds to one outer peripheral surface 326 of the operation object 302 , the robot system 100 can calculate the candidate corresponding to both the first posture 312 and the third posture 316 . control sequence.

To account for poses where there are multiple possibilities (eg, a misestimation of the original pose), the robotic system 100 may calculate the cost of placing the identity 332 of the manipulated object 112 on both the set of exposed tokens 332 and the set of surrogate tokens 332 . Scan posture. As represented by block 562, the robotic system 100 may calculate a set of candidate poses associated with the manipulation object 112 by scanning the field or scanning the field. If the proximity location is selected, the robotic system 100 rotates and/or moves the model of the location 334 of the marker 332 in a manner that configures the location 334 of the marker 332 within the scanning field, whereby, as described in block 542, can be calculated as Alternate scan positions.

In block 564, the robotic system 100 may map the set of exposed indicia 332 and the set of alternate indicia 332 to candidate scan locations, respectively. The robotic system 100 may map the set of exposed markers 332 based on rotating the model of the location 335 of the markers 332 using the initial pose as a starting point. The robotic system 100 may map a set of alternate markers 332 based on rotating the model of the location 334 of the marker 332, using one surrogate pose (eg, a reverse-oriented pose) as a starting point.

When mapping the location 334 of the indicia 332, in block 568, the robotic system 100 may associate the position 334 and/or the orientation of the indicia 332 of the manipulated object 112 in both the set of exposed indicia 332 and the set of alternate indicia 332 with the Scan fields for comparison. In determination block 570 , the robotic system 100 may determine whether the candidate pose provides the identifier 332 of the manipulation object 112 in both the set of exposed markers 332 and the set of substitute markers 332 to the scanner simultaneously.

In block 572, the robotic system 100 may identify candidate gestures that simultaneously provide the identifiers 332 of the manipulated objects 112 in both the set of exposed identifiers 332 and the set of alternative identifiers 332 to different scanners/scanning fields as scanning gestures. . For example, in a state where the positions of the operation objects 112 in the set of marks 332 of the exposed operation objects 112 and the set of substitute marks 332 are located on opposite surfaces, when the holding position corresponds to one outer peripheral surface 326 of the operation object 112, The robot system 100 can recognize the scanning posture of placing the operation object 112 between the opposite/opposite pair of scanners in a state in which the two side surfaces of the operation object 112 face one of the scanners respectively.

In block 574, in the event that neither of the candidate poses provides the identifier 332 of the manipulator 112 in both the set of identifiers 332 of the exposed manipulator 112 and the set of substitute identifiers 332, the robotic system 100 may The set of identities 332 of the exposed operand 112 and the set of surrogate identities 332 are calculated to provide a plurality of scan positions (eg, a first scan position and a second scan position) of the identity 332 of at least one operand 112, respectively. For example, a first scan position may provide an object scanner with the positions 334 of one or more identifications 332 within the set of identifications 332 of the exposed manipulation object 112, and the second scan position may provide an alternate set of identifications 332 of manipulation objects 112 The location 334 of one or more markers 332 within is provided to a scanner. The second scan position may be associated with pivoting the end effector from the first scan position, translating the end effector in parallel, or a combination thereof.

In addition, referring again to the example shown in FIGS. 4A and 4B , the second control sequence 424 may be orthogonal to two opposing surfaces (eg, associated with the first pose 312 and the third pose 316 ) as described above. The second approach position 434 corresponding to the three surfaces (eg, one outer peripheral surface 326 of the operation object 112 ) corresponds to. Thus, the first scan position may correspond to positioning the surface (eg, estimated to be the bottom surface 324 of the manipulated object 112 ) to which the initial gesture (eg, the first gesture 312 ) corresponds, over the upward facing scanner 416 and facing the scanner. A first one of the second providing locations 444 . The second scan position corresponds to a second position that rotates the operation object 112 by 90 degrees, eg, a second supply position 444 that rotates substantially counterclockwise in the overall movement direction from the start position 114 to the work position 116 . Thus, the second scan position corresponds to the second providing position 444 that sets the surface (eg, the bottom surface 324 of the determination manipulation object 112 ) corresponding to the alternative gesture (eg, the third gesture 316 ) in a horizontal orientation The scanner 416 is positioned in front of the scanner 416 and in a vertical orientation towards the scanner 416 .

Based on the resulting set of scan poses and/or scan positions, the robotic system 100 may create or acquire candidate control sequences. Using one or more of the mechanisms described above (eg, the A* mechanism), the robotic system 100 places the end effector at a selected proximity position, touches and holds the operand 112 accordingly, and calculates to lift and move the operand 112 to a position where it has been reached. Candidates for the set of identified scan poses and/or scan positions. For example, when recognizing the scan pose, the robotic system 100 may calculate alternatives and determine the scan pose relative to the manipulation object 112 within the scan field or through the scan field. In the case where the robot system 100 cannot recognize the scanning posture, the robot system 100 calculates a candidate solution and can move/orient the end effector in a manner of continuously passing through a set of a plurality of scanning positions, thereby, according to the plurality of provided positions/orientations, The operation object 112 is continuously moved/rotated.

In block 576, the robotic system 100 may recreate or update the scan likelihood associated with each control sequence of the candidate. The robotic system 100 may be based on the various possibilities and/or priorities described above in relation to the combo box 544 (eg, proximity location, scan location, scanner 416 utilized, identification 332 considered exposed, associated errors, and/or loss rate). , or their combination related likelihoods and/or scores) to update the scan likelihood, but instead of a measure of performance, the scan likelihood is updated for the measure of the scan.

In block 578, the robotic system 100 may create or acquire a control sequence based on the selection of alternatives based on the scanning likelihood. The robot system 100 may select the candidate with the highest scanning possibility among the candidate schemes as the control sequence. For example, the robotic system 100 may select one or more scan areas (ie, in front of the one or more scanners 416 ) during movement of the manipulation object 112 , eg, for scanning the space between the start position 114 and the work position 116 , and the configuration exposes The at least one position 334 of the identification 332 of the , and the at least one position 334 of the alternative identification 332 are the candidates with the highest probability.

Within a small difference value (eg, a specified threshold), where more than two candidate solutions correspond to scan probabilities, the robotic system 100 may calculate and evaluate a metric of performance corresponding to the corresponding candidate solutions (eg, such as Blocks 544 and 546). The robotic system 100 can select the candidate that is closest to the target condition as the control sequence.

The robotic system 100 may depart from the illustrated exemplary flow. For example, the robotic system 100 may select an approach location as described above. Based on the selected approach position, the robotic system 100 may implement a prescribed set of actions to grasp, lift, reorient, move horizontally, return down and release, or a combination thereof, on the manipulation object 112 . During or after the specified set of actions, the robotic system 100 may (eg, by looping back to block 502 ) again photograph or scan the pickup area, redetermining the original pose and reliable fiducials (eg, through blocks 522 and 524 ).

If returning to FIG. 5A, in block 508, the robotic system 100 may begin implementing the resulting control sequence. The robotic system 100 sends commands and/or settings of control sequences to other devices (eg, corresponding operating devices 212 and/or other processors) to implement operations based on operating one or more processors 202 in a manner to perform operations 402, 404. control sequence. Thus, the robot system 100 operates the operating equipment 212 according to the sequence of commands or settings, or a combination thereof, thereby executing the control sequence. For example, the robotic system 100 may operate the operating device 212, deploy the end effector in an approached position about the starting position 114, contact and hold the manipulation object 112, or perform a combination thereof.

In block 582, the robotic system 100 can move the end effector to the scan position, thereby moving the manipulated object 112 to the provided position/orientation. For example, after the manipulation object 112 is lifted from the starting position 114 , or as the robotic system 100 moves the end effector, a scanning pose relative to the manipulation object 112 can be established. Additionally, the robotic system 100 can move the end effector to the first scan position.

In block 584 , the robotic system 100 may operate the scanner 416 to scan the manipulation object 112 . For example, one or more processors 202 may send commands to scanner 416 to perform a scan, and/or send a query to scanner 416 to receive scan status and/or scan values. In block 585 or the like, where the control sequence includes a scan pose, the robotic system 100 may execute a control sequence to move the manipulated object 112 in the scan pose throughout the scan area in a direction orthogonal to the scan area's orientation. During the movement of the object 112, the scanner 416 may scan (simultaneously and/or consecutively) a plurality of surfaces associated with a plurality of possible locations 334 of the identification 332 of the object 112.

In determination block 586, the robotic system 100 evaluates the scan results (eg, status and/or scan values) to determine whether the manipulation object 112 was scanned. For example, the robotic system 100 may evaluate the scan results after executing the control sequence to the first scan position. In block 588 or the like, in the event that the indicated scan result is that the scan of the manipulation object 112 was successful (eg, the status corresponds to the detection of a valid code/identification, and/or the scan value matches the identified/expected manipulation object 112), the robotic system 100 can move the operation object 112 to the work position 116 . Based on the successful scanning, the robot system 100 may directly move the operation object 112 to the working position 116 regardless of the subsequent scanning position (eg, the second scanning position).

If the scan result indicates that the scan of the operating object 112 was unsuccessful, the robotic system 100 determines in determination block 590 whether the current scan position is the last of the control sequence. Without being the final control sequence, the robotic system 100 may move the manipulated object 112 to the next provided position/orientation in the manner shown in the loop returning to block 582 .

In the event that the current scan position is the last of the control sequence, the robotic system 100, as indicated by block 592, may perform one or more improvement operations. The robotic system 100 may stop and/or cancel the control sequence in the event that the scan results associated with all scan positions of the control sequence indicate a scan failure. The robotic system 100 may generate status/messages to alert the operator of errors. The robot system 100 may configure the operation object 112 in a designated area (ie, a position different from the start position 114 and the work position 116 ) in relation to the operation object 112 that failed to scan.

Based on the successful completion of the jobs 402 , 404 (ie, the successful scanning of the operation object 112 and placement in the operation position 116 ) or the implementation of the improvement operation, the robot system 100 can proceed to the next operation/operation object 112 movement. The robot system 100 may scan the designated area again, as shown in the loop returning to block 502 , and may select the next operation object 112 using the existing photographing data, as shown in the loop returning to block 504 .

By scanning the operand 112 within a space (eg, a location between the start location 114 and the job location 116 ), the efficiency and speed associated with performing the jobs 402 , 404 may be improved. By calculating a control sequence that includes the scan position and cooperates with the scanner 416 , the robotic system 100 can effectively combine the job for moving the manipulation object 112 with the job for scanning the manipulation object 112 . Furthermore, by creating or acquiring control sequences based on a reliable reference of the initial pose, the efficiency, speed, and accuracy associated with scanning operations can be further improved. As described above, the robotic system 100 can create or acquire control sequences for alternate orientations corresponding to scenarios in which the initial pose was incorrect. Therefore, the robot system 100 can increase the probability of correct/successful scanning of the manipulation object 112 even in the event of a posture determination error due to calibration errors, unexpected postures, errors due to unexpected light conditions, and the like. Increasing the likelihood of a correct scan can increase the overall throughput associated with the robotic system 100 and reduce operator effort/intervention.

[Summarize]

The above detailed description of the embodiments of the present disclosure is not intended to be exhaustive or to limit the present disclosure to the precise manners disclosed above. While specific embodiments of the present disclosure are described above for illustrative purposes, various equivalent modifications are possible within the scope of the present disclosure, as those skilled in the art will recognize. For example, although flows or blocks are provided in a given order, by alternative implementations, a routine with steps may be performed in a different order or a system with blocks may be employed, in addition, some flows or blocks may be deleted, moved, added, Subdivide, combine and/or morph to provide alternatives or sub-combinations. These processes or blocks, respectively, can be implemented in a variety of different ways. Additionally, although processes or blocks are performed consecutively at the points in time shown, these processes or blocks may alternatively be performed or performed in parallel, or at different times. Furthermore, any specific numbers shown herein are examples only, and different values or ranges may be employed in alternative implementations.

These and other changes can be made to the present disclosure in light of the above-described detailed description. Specific embodiments of the disclosure and the assumed best mode are described in the detailed description, but the disclosure can be practiced in a variety of ways, regardless of how the above description is presented in detail in the text. The details of the present system may vary significantly in specific embodiments thereof, while still being encompassed by the techniques disclosed herein. As noted above, the use of a particular term in describing a particular feature or aspect of the present disclosure should not be construed to imply that the term is redefined herein to be limited to any particular feature, feature, or aspect of the present disclosure to which the term is associated . Accordingly, the invention is not to be limited except as defined by the appended claims. In general, the terms used in the appended claims should not be construed to limit the disclosure to the specific embodiments disclosed herein, unless portions of the above Detailed Description explicitly state such terms.

While certain aspects of the invention have been presented in the form of the specific claims appended hereto, the applicants of the present application contemplate various aspects of the invention by way of any number of claims. Accordingly, the applicant of the present application reserves the right to make further claims and to make such additional claims after filing this application in this or a continuation application.

Claims (16)

1. A control method of a robot system including a robot with a robot arm and an end effector, the control method of the robot system comprising:

acquiring an approach position at which the end effector grips an operation object;

acquiring a scanning position for scanning the identifier of the operation object; and

creating or acquiring a control sequence based on the approach position and the scanning position, instructing the robot to execute the control sequence,

the control sequence includes the following (1) to (4):

(1) holding the operation object from a start position;

(2) scanning identification information of the operation object with a scanner located between the start position and the working position;

(3) when a predetermined condition is satisfied, temporarily releasing the operation object from the end effector at a grip conversion position, and again gripping the operation object with the end effector so as to convert the grip; and

(4) and moving the operation object to a working position.

2. The control method of a robot system according to claim 1,

the control sequence includes the following (5) and (6):

(5) setting, as the predetermined condition, an improvement in efficiency of storing the operation object at the working position when the direction in which the operation object is gripped by the end effector is changed by changing the gripping direction of the operation object; and

(6) calculating storage efficiency at the work position before the gripping conversion of the operation object and storage efficiency at the work position after the gripping conversion of the operation object.

3. The control method of a robot system according to claim 2,

the control sequence includes the following (7) and (8):

(7) acquiring the height of the operation object; and

(8) calculating the storage efficiency based on the height of the operation object.

4. The control method of a robot system according to claim 3,

calculating the height of the operation object from the height position of the top surface of the operation object and the height position of the bottom surface of the operation object measured in the end effector gripping state.

5. The control method of a robot system according to claim 3 or 4, wherein,

the height of the measurement object is measured while the operation object is scanned by the scanner.

6. The control method of a robot system according to any one of claims 1 to 5, wherein,

the control sequence includes: (9) when the predetermined condition is satisfied, the operation object is placed on the temporary placement table at the grip conversion position and is temporarily released from the end effector.

7. The control method of a robot system according to any one of claims 1 to 6, wherein,

the control method of the robot system further includes:

acquiring shooting data representing a pickup area including the operation object;

determining an initial posture of the operation object based on the shot data;

calculating a reliable reference representing a likelihood that an initial pose of the operational object is correct; and

acquiring the approach position and the scan position based on the reliable reference.

8. The control method of a robot system according to claim 7,

the control sequence includes: (10) selectively calculating the approach location and the scan location in terms of a measure of performance and/or a measure of scanning based on a result of comparing the reliable reference to a sufficiency threshold,

the metric of the scan is independent of whether the initial pose of the manipulator is correct or incorrect, and is related to the likelihood that the identity of the manipulator is not covered by the end effector.

9. The control method of a robot system according to claim 7,

in the event that the confidence measure does not meet the sufficiency threshold, the approach location and the scan location are acquired based on a metric of the scan, or the approach location and the scan location are acquired based on a metric of the scan with priority over a metric of the performance.

10. The control method of a robot system according to claim 7,

acquiring the approach location and the scan location based on a metric of the performance if the reliable reference satisfies the sufficiency threshold.

11. The control method of a robot system according to any one of claims 1 to 10,

the control sequence includes the following (11) and (12):

(11) acquiring a first scanning position for providing identification information of the operation object to the scanner and a second scanning position for providing alternative identification information of the operation object to the scanner;

(12) until the operation object is moved to the first scanning position, in a case where the scanning result indicates a successful scan, the operation object is moved to the job position and the second scanning position is disregarded, or, in a case where the scanning result indicates a failed scan, the operation object is moved to the second scanning position.

12. A non-transitory computer readable storage medium storing processor commands for implementing a method of controlling a robotic system including a robot with a robot arm and an end effector,

the processor command includes:

acquiring a command for the end effector to grip an approach position of an operation object;

acquiring a command of a scanning position for scanning the identification information of the operation object; and

creating or retrieving a control sequence based on the approach position and the scanning position, instructing the robot to execute commands of the control sequence,

the control sequence includes the following (1) to (4):

(1) holding the operation object from a start position;

(2) scanning an identification of the operation object with a scanner located between the start position and the job position;

(3) when a predetermined condition is satisfied, temporarily releasing the operation object from the end effector at a grip conversion position, and again gripping the operation object with the end effector so as to convert the grip; and

(4) and moving the operation object to a working position.

13. The non-transitory computer-readable storage medium of claim 12,

the control sequence includes the following (5) and (6):

(5) setting, as the predetermined condition, an improvement in efficiency of storing the operation object at the working position when the direction in which the operation object is gripped by the end effector is changed by changing the gripping direction of the operation object; and

(6) calculating storage efficiency at the work position before the gripping conversion of the operation object and storage efficiency at the work position after the gripping conversion of the operation object.

14. The non-transitory computer-readable storage medium of claim 13,

the control sequence includes the following (7) and (8):

(7) acquiring the height of the operation object; and

(8) calculating the storage efficiency based on the height of the operation object.

15. The non-transitory computer-readable storage medium of claim 14,

the height of the operation object is calculated from the height position of the top surface of the operation object and the height position of the bottom surface of the operation object measured in a state where the end effector is held.

16. A control device for a robot system comprising a robot having a robot arm and an end effector, and performing the control method according to any one of claims 1 to 11.

Images