CN116197887B - Image data processing method, device, electronic equipment and storage medium for generating capture auxiliary images - Google Patents

Abstract

The application discloses a method, a device, electronic equipment and a storage medium for generating a grabbing auxiliary image. The method for generating the grabbing auxiliary image comprises the following steps: acquiring image data comprising one or more items to be grabbed; outputting the image data and an operable control to form an interactive interface, wherein the control is operable by a user to select a grabbing auxiliary image and display the selected grabbing auxiliary image to the user; responding to the operation of the control by the user, acquiring grabbing auxiliary data corresponding to the grabbing auxiliary image selected by the user; generating a grabbing auxiliary layer based on the acquired grabbing auxiliary data; the capture assistance layer is combined with image data comprising one or more items to be captured to generate a user selected capture assistance image. The invention enables a user to intuitively determine various parameters in the process of the robot executing grabbing, and further determines how to adjust the parameters of the robot so that the robot can operate according to the needs of the user.

Description

Image data processing method, device, electronic equipment and storage for generating grabbing auxiliary images storage medium

Technical field

The present application relates to the field of automatic control and program control B25J of robotic arms or grippers, and more specifically, to image data processing methods, devices, electronic equipment and storage media for generating grasping auxiliary images.

Background technique

Robots have basic characteristics such as perception, decision-making, and execution. They can assist or even replace humans in completing dangerous, heavy, and complex tasks, improve work efficiency and quality, serve human life, and expand or extend the scope of human activities and capabilities. With the development of industrial automation and computer technology, robots have begun to enter the stage of mass production and practical application. In industrial scenarios, industrial robots have been widely used and can replace humans in performing some repetitive or dangerous tasks. Traditional industrial robot design focuses on the design and manufacturing of robot hardware, and the robot itself is not "intelligent" enough. When using robots at industrial sites, technicians need to plan in advance the hardware equipment, assembly lines, material locations, and robot task paths for the entire industrial site. For example, if items are to be sorted and transported, on-site workers need to first sort and transport different items. Various types of items are sorted out and neatly placed in material frames with uniform specifications. Before using the robot to operate, you need to determine the assembly line, material frames, transportation positions, etc., and set the robot settings based on the determined information. Fixed motion path, fixed grabbing position, fixed rotation angle, fixed fixture, etc.

As an improvement to the above-mentioned traditional robot technology, intelligent program-controlled robots based on robot vision have been developed. However, the current "intelligence" is still relatively simple. The main implementation method is to obtain task-related image data through visual acquisition devices such as cameras, based on The image data obtains 3D point cloud information, and then the robot's operations are planned based on the point cloud information, including movement speed, movement trajectory and other information, thereby controlling the robot to perform tasks. However, existing robot control solutions are not effective when encountering complex tasks. For example, in shopping malls, logistics and other scenarios, processing many stacked items requires a robotic arm to use visual equipment to locate and identify the location of objects in a scattered and disordered scene, and use suction cups, clamps or Other bionic devices pick up objects, and the picked objects are placed in corresponding positions according to certain rules through operations such as robotic arm movement and trajectory planning. In such an industrial scenario, there are many difficulties in using robots to perform grasping. For example, there are too many items to be grasped in the scene and the light is uneven, resulting in poor point cloud quality of some items, affecting the grabbing effect; types of items There are many, and they are not arranged neatly and in various directions. As a result, when grabbing each item, the grabbing point is different, making it difficult to determine the grabbing position of the clamp; items are stacked, and it is easy for other items to fly when grabbing an item. Case. In such an industrial scenario, there are many factors that affect the difficulty of grabbing items, and the traditional grabbing and sorting method is not effective enough. In addition, when the grabbing algorithm is designed to be more complex, it also brings problems to on-site workers. More obstacles arise. When a problem occurs, it is difficult for on-site staff to figure out why the problem occurs and how to adjust to solve the problem. It is often necessary for the robot provider to send experts to assist.

Contents of the invention

In view of the above problems, the present invention is proposed in order to overcome the above problems or at least partially solve the above problems. Specifically, the present invention proposes a method of visually displaying parameters and image data associated with the grasping control method of the present invention to the user, so that the user can intuitively determine the operation principle of the robot without understanding the The robot performs various parameters in the grabbing process, understands why the robot performs tasks in a certain way, and then determines how to adjust the robot's parameters so that the robot can operate according to its own needs.

All solutions disclosed in the claims and description of this application have one or more of the above innovations, and accordingly, can solve one or more of the above technical problems. Specifically, this application provides a method, device, electronic device and storage medium for generating a capture auxiliary image.

The method for generating capture auxiliary images according to the embodiment of the present application includes:

Obtain image data including one or more items to be grabbed;

Output the image data and operable controls to form an interactive interface, the controls being operable by a user to select a capture auxiliary image and display the selected capture auxiliary image to the user;

In response to the user's operation on the control, obtaining capture auxiliary data corresponding to the capture auxiliary image selected by the user;

Generate a crawling auxiliary layer based on the obtained crawling auxiliary data;

The grasp assist layer is combined with image data including one or more items to be grasped to generate a user selected grasp assist image.

In some embodiments, the image data and the operable controls are within the same interactive interface.

In some embodiments, the image data and operable controls are within different interactive interfaces.

In some embodiments, the different interactive interfaces switch in response to user operations.

In some embodiments, the grasping auxiliary data includes: a numerical value associated with a user-selected grasping auxiliary image and a mask of the graspable area of the item to be grasped.

In some embodiments, combining the grasping auxiliary layer with image data including one or more items to be grasped includes: adjusting the color, transparency and/or contrast of the grasping auxiliary layer , and then combine the adjusted grabbing auxiliary layer with image data including one or more items to be grabbed.

The device for generating capture auxiliary images according to the embodiment of the present application includes:

An image data acquisition module, used to acquire image data including one or more items to be grabbed;

An interactive interface display module, configured to output the image data and operable controls to form an interactive interface. The controls can be operated by the user to select the capture auxiliary image and display the selected capture auxiliary image to the user;

An auxiliary data acquisition module, configured to acquire auxiliary grasping data corresponding to the auxiliary grasping image selected by the user in response to the user's operation on the control;

An auxiliary layer generation module, configured to generate an auxiliary crawling layer based on the acquired auxiliary crawling data;

An auxiliary image generation module is configured to combine the grasping auxiliary layer with image data including one or more items to be grasped to generate a user-selected grasping auxiliary image.

In some embodiments, the image data and the operable controls are within the same interactive interface.

In some embodiments, the image data and operable controls are within different interactive interfaces.

In some embodiments, the different interactive interfaces switch in response to user operations.

In some embodiments, the grasping auxiliary data includes: a numerical value associated with a user-selected grasping auxiliary image and a mask of the graspable area of the item to be grasped.

In some embodiments, the auxiliary image generation module is also configured to: after adjusting the color, transparency and/or contrast of the grabbing auxiliary layer, combine the adjusted grabbing auxiliary layer with one or more A combination of image data of the items to be grabbed.

The electronic device of the embodiment of the present application includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, any one of the above embodiments is implemented. Method to generate crawling auxiliary images.

The computer-readable storage medium of the embodiment of the present application has a computer program stored thereon, and when the computer program is executed by the processor, the method for generating a grabbing auxiliary image in any of the above embodiments is implemented.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Description of the drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the description of the embodiments in conjunction with the following drawings, in which:

Figure 1 is a schematic diagram of mask preprocessing in some embodiments of the present application;

Figure 2 is a schematic flow chart of a grabbing parameter visualization method according to some embodiments of the present application;

Figures 3a and 3b are schematic diagrams of the visual menu and the visual image displayed to the user after the selected height and suction cup size are visualized in some embodiments of the present application;

Figure 4 is a schematic structural diagram of a grabbing parameter visualization device according to certain embodiments of the present application;

Figure 5 is a schematic structural diagram of an electronic device according to certain embodiments of the present application.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a thorough understanding of the disclosure, and to fully convey the scope of the disclosure to those skilled in the art.

In the description of specific embodiments, it should be understood that the terms "center", "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", The orientations or positional relationships indicated by "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention. and simplified description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be construed as a limitation of the present invention.

Furthermore, the terms “first”, “second”, “third”, etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, features defined by "first," "second," etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

The present invention can be used in industrial robot control scenarios based on visual recognition. A typical industrial robot control scenario based on visual recognition includes devices for collecting images, hardware for production lines and control devices such as production line PLCs, robot components for performing tasks, and operating systems for controlling these devices. or software. Devices used to collect images can include 2D or 3D smart/non-intelligent industrial cameras. Depending on the functions and application scenarios, they can include area scan cameras, line scan cameras, black and white cameras, color cameras, CCD cameras, CMOS cameras, and analog cameras. Cameras, digital cameras, visible light cameras, infrared cameras, UV cameras, etc.; production lines can include packaging production lines, sorting production lines, logistics production lines, processing production lines and other production lines that require the use of robots; robot components used to perform tasks in industrial scenarios can It is a bionic robot, such as a humanoid robot or a dog-shaped robot, or it can be a traditional industrial robot, such as a mechanical arm, etc.; industrial robots can be operating robots, program-controlled robots, teaching and reproduction robots, CNC robots, and sensory control type robots, adaptive control robots, learning control robots, intelligent robots, etc.; according to the working principle, the robot arm can be a spherical manipulator, a multi-joint manipulator, a rectangular coordinate manipulator, a cylindrical coordinate manipulator, a polar coordinate manipulator, etc. Depending on the function of the robotic arm, you can use a grabbing robotic arm, a palletizing robotic arm, a welding robotic arm, or an industrial robotic arm; the end of the robotic arm can be equipped with an end effector, and the end effector can use a robot fixture according to the needs of the task. Robot gripper, robot tool quick changer, robot collision sensor, robot rotary connector, robot pressure tool, compliance device, robot spray gun, robot burr cleaning tool, robot arc welding gun, robot electric welding gun, etc.; robot fixtures can be various General-purpose fixtures refer to fixtures that have standardized structures and have a wide range of applications, such as three-jaw chucks and four-jaw chucks for lathes, flat-nose pliers and indexing heads for milling machines, etc. For another example, according to the clamping power source used by the clamp, the clamp can be divided into manual clamping clamp, pneumatic clamping clamp, hydraulic clamping clamp, gas-hydraulic linkage clamping clamp, electromagnetic clamp, vacuum clamp, etc., or other items that can be picked up of bionic devices. Devices used to collect images, hardware used in production lines and control devices such as production line PLCs, robot components used to perform tasks, and operating systems or software used to control these devices can be based on TCP protocols, HTTP protocols, GRPC protocol (Google Remote Procedure Call Protocol, Google Remote Procedure Call Protocol) communicates to transmit various control instructions or commands. The operating system or software can be set in any electronic device. Typical such electronic devices include industrial computers, personal computers, laptops, tablets, mobile phones, etc. Electronic devices can communicate with other devices or systems through wired or wireless methods. communication. In addition, grabbing in the present invention refers to any grabbing action that can control an object to change the position of the object in a broad sense, and is not limited to grabbing objects in a narrow sense by "grabbing". In other words, The grasping of objects by methods such as sucking, lifting, and tightening also belongs to the scope of grasping of the present invention. The items to be grabbed in the present invention can be cartons, cartons, plastic soft packages (including but not limited to snack packaging, milk Tetra Pak pillow packaging, milk plastic packaging, etc.), cosmeceutical bottles, cosmeceuticals, and/or other Regular toys and other items, etc., these items to be grabbed can be placed on the ground, pallets, conveyor belts and/or material baskets.

In actual industrial scenarios, on-site workers are generally asked to set various parameters of the robot for specific grabbing tasks. However, on-site workers are not familiar with the principles of grabbing, so when a problem is discovered, they are not clear about the problem. Neither is it clear how to modify the settings to resolve the issue. For example, when grabbing multiple stacked items to be grabbed, the items flew out of the frame. The on-site staff determined that the reason was that the clamp passed over the items on the upper layer and directly grabbed the items on the lower layer, but they It is not clear why the robot considers the lower items to have a higher priority, nor is it clear how to set the weight to change the robot's grabbing order. In order to solve this problem, the inventor developed a set of methods to visually display the graphics and parameters involved in the grabbing process to the on-site workers for operation according to the needs of the on-site workers. This is also one of the key points of the present invention. one.

Figure 2 shows a schematic flowchart of a method for visualizing graphics and parameters in a grabbing process according to an embodiment of the present invention. As shown in Figure 2, the method includes:

Step S400, obtain image data including one or more items to be grabbed;

Step S410, output the image data and operable controls to form an interactive interface, and the controls can be operated by the user to select the capture auxiliary image and display the selected capture auxiliary image to the user;

Step S420, in response to the user's operation on the control, obtain the capture auxiliary data corresponding to the capture auxiliary image selected by the user;

Step S430: Generate a crawling auxiliary layer based on the obtained crawling auxiliary data;

Step S440: Combine the grabbing auxiliary layer with image data including one or more items to be grabbed to generate a grabbing auxiliary image selected by the user.

For step S400, the present invention can be applied to industrial scenarios that include one or more items to be grabbed, using a clamp to grab all the items to be grabbed in sequence, and placing the rows of grabbed items in a specific position. The type and acquisition method of image data are not limited in this embodiment. As an example, the acquired image data can include point clouds or RGB color images. Point cloud information can be obtained through a 3D industrial camera. A 3D industrial camera is generally equipped with two lenses to capture the group of items to be grabbed from different angles. After processing, the three-dimensional image of the object can be displayed. The group of items to be grasped is placed under the visual sensor, and the two lenses shoot at the same time. Based on the relative posture parameters of the two images obtained, a general binocular stereo vision algorithm is used to calculate the X of each point of the glass to be coated. , Y, Z coordinate values and the coordinate orientation of each point, and then converted into point cloud data of the item group to be grabbed. During specific implementation, laser detectors, visible light detectors such as LEDs, infrared detectors, radar detectors and other components may also be used to generate point clouds. The present invention does not limit the specific implementation manner.

The point cloud data obtained through the above method is three-dimensional data. In order to filter out the data corresponding to the dimensions that have less impact on the crawling, reduce the amount of data processing, thereby speeding up the data processing and improving efficiency, the obtained three-dimensional to-be-grabbed data can be Take the point cloud data of the item group and map it onto a two-dimensional plane through forward projection.

As an example, a depth map corresponding to the orthographic projection can also be generated. A two-dimensional color map corresponding to the three-dimensional item area and a depth map corresponding to the two-dimensional color map may be acquired along a depth direction perpendicular to the item. Among them, the two-dimensional color image corresponds to the image of the plane area perpendicular to the preset depth direction; each pixel point in the depth image corresponding to the two-dimensional color image corresponds to each pixel point in the two-dimensional color image, and each The value of a pixel is the depth value of the pixel. .

For step S410, the captured pictures and controls can be output to the display and displayed to the user. The interaction between the user and the robot can be operated by touch operation, voice operation, or traditional operating equipment, such as mouse, keyboard, etc., which is not limited by the present invention. The interactive interface is a channel for information exchange between people and computer systems. Users input information and operate the computer system through the interactive interface, and the computer provides information to the user through the interactive interface for reading, analysis and judgment. Each interactive interface contains the The interactive interface provides an information display interface and controls that can be operated by users. The controls used to control visualization can be displayed as a whole on an interactive interface with the image, or can be divided into two interfaces with the image, and provide an interface to the control interface on the image interface, and provide an interface to the image interface on the space interface. When the user operates the interface, he or she will be redirected to the control interface or image interface. As shown in Figure 3a, the control interface allows you to select operations related to visualization, including: operations to turn on visualization, operations to display the outline of overlaid objects, and operations to visualize attributes. The visual attributes may include any parameters output in any of the aforementioned embodiments. The optional visual attributes in Figure 3a include: ALL, display by posture and height, display by suction cup size, display by pressing overlap degree, display by transparency degree, Displayed by pose and orientation.

For step S420, the user can select the value of interest according to his own needs. For example, when the user finds that the robot does not grab in the order he expected, he can select the ALL control to display the grabbing priority value of each item to be grabbed to determine the difference between the actual grabbing order and the expected grabbing order. Then select specific visualization properties individually to determine which properties affect the crawl order. When the user selects a visualization option, the system will find and call the corresponding data. As a better implementation, in response to the user's selection, the system will obtain the parameters selected by the user and the mask of the grabbable area to use as auxiliary data. For example, when the user selects "Display by sucker size", the system will At the same time, the mask of the grabbable area and the value of the suction cup size are called; similarly, when the user selects "Display by pose height", the mask of the grabbable area and the mask height characteristic value are called.

A feasible implementation method for determining the graspable area and generating a mask may be to: first, after acquiring image data including one or more items to be grasped, process the image data to identify the objects in the image. For each pixel, for example, for a 256*256 image, 256*256=65536 pixels should be identified; then based on the characteristics of each of these pixels, all pixels included in the entire image Pixels are classified. Among them, the characteristics of pixels mainly refer to the RGB values of pixels. In actual application scenarios, in order to facilitate the classification of features, the RGB color image can also be processed into a grayscale image, and the grayscale value is used for classification. Classification. For the classification of pixels, you can determine in advance which categories the pixels need to be divided into. For example, the RGB image obtained includes a lot of beverage cans, food boxes, and material frames. Therefore, if the purpose is to generate beverage cans, food boxes, etc. As well as the mask of the material frame, the predetermined categories can be beverage cans, food boxes and material frames. We can set a logo for these three different categories. The logo can be a number, such as 1 for beverage cans, 2 for food boxes, and 3 for material boxes. It can also be a color, such as red for beverage cans and blue for food boxes. , the material frame is green. After classification and processing, the final image will be marked with 1 or red to identify beverage cans, 2 or blue to identify food boxes, and 3 or green to identify the material frame. What is to be generated in this embodiment is a mask of the grabbable area of the item, so only the grabbable area is classified, for example, blue. The blue area in the image processed in this way is the area of the item to be grabbed. Mask of the grabber area; then create an image output channel for each classification. The function of this channel is to extract all classification-related features in the input image as output. For example, after we create an image output channel for the grabbable area class, and input the collected RGB color image into the channel, we can obtain an image that extracts the features of the grabbable area from the output of the channel. Finally, we combine the characteristic image of the grabbable area obtained through processing with the original RGB image to generate synthetic image data marked with a grabbable area mask.

Masks generated in this way are sometimes inappropriate. For example, the size and shape of some masks are inconvenient for subsequent processing; for another example, although masks are generated in some areas, the fixture cannot be executed at the position of the mask. Grab. An inappropriate mask will have a great impact on subsequent processing, so the generated mask needs to be preprocessed before being used in other steps. As shown in Figure 1, preprocessing of the mask may include: 1. Expanding the mask to fill defects such as missing and irregular mask images. For example, for each pixel on the mask, a certain number of points around the point, such as 8-25 points, can be set to the same color as the point. This step is equivalent to filling in the surroundings of each pixel, so if the item mask is missing, this operation will fill in all the missing parts. After this process, the item mask will become complete and there will be no missing parts. At the same time, The mask as a whole will also become slightly "fat" due to expansion. Appropriate expansion will help further subsequent image processing operations; 2. Determine whether the area of the mask meets the predetermined conditions. If not, remove the mask. . First of all, a smaller mask area is likely to be wrong. Because of the continuity of image data, a grabbable area usually includes a large number of pixels with similar characteristics. The mask area formed by discrete small pixels may not be The real graspable area; secondly, when the robot's end actuator, that is, the clamp, performs a grasping task, its landing position needs to have a certain area. If the area of the graspable area is too small, the clamp cannot settle in this area at all. , it is impossible to grab items, so a mask that is too small is meaningless. The aforementioned predetermined conditions can be set according to conditions such as the size of the fixture and the size of the noise. Its value can be a certain size, the number of pixels included, or a ratio. For example, the predetermined condition can be set to 0.1%, that is, when the ratio of the mask area to the overall image area is less than 0.1%, the mask is considered unusable and is removed from the image; 3. Determine whether the number of point clouds in the mask is less than the preset The minimum number of point clouds. The number of point clouds reflects the quality of camera collection. If the number of point clouds in a certain captureable area is too small, it means that the shooting of this area is not accurate enough. Point clouds can be used to control the gripper to perform grasping. Too small a number may affect the control process of the gripper. Therefore, you can set the minimum number of point clouds that should be included in a certain mask area, for example: 10. When the number of point clouds covered in a certain grabby area is less than 10, the mask will be changed from Eliminate or randomly add point clouds to grabbable areas from the image data until 10 are reached. The mask height refers to the height of the mask of the grabable area of an item to be grabbed, or it can be the Z coordinate value. The mask height reflects the height of the graspable surface of the object. Since there are multiple items to be grabbed and they are stacked together, grabbing the upper items first can prevent the upper items from being carried away due to the lower items being suppressed. Secondly, it can avoid knocking down high-level items and affecting the grabbing of low-level items, and items located on high levels are obviously easier to grab than items located on low levels. The height of the mask can be obtained through a depth map or a point cloud at the location of the mask. In one embodiment, a point cloud including one or more items to be grabbed can be obtained first. The point cloud is in a preset coordinate system. For the data set of points, in order to conveniently calculate the height value, the camera can be used to shoot directly above the object to be grabbed. Then based on the mask area, the point cloud included in the mask area is obtained. Calculate the pose key points of the graspable area represented by the mask and the depth values of the pose key points. The three-dimensional pose information of the item object is used to describe the pose of the object to be grasped in the three-dimensional world. Pose key points refer to pose points that can reflect the three-dimensional position characteristics of the graspable area. It can be calculated by:

First, the three-dimensional position coordinates of each data point in the mask area are obtained, and based on the preset operation results corresponding to the three-dimensional position coordinates of each data point, the position information of the pose key points of the graspable area corresponding to the mask is determined. For example, assume that the point cloud in the mask area contains 100 data points, obtain the three-dimensional position coordinates of the 100 data points respectively, calculate the average of the three-dimensional position coordinates of the 100 data points, and use the data points corresponding to the average as the mask The pose key points of the graspable area corresponding to the membrane area. Of course, in addition to finding the average value, the above-mentioned preset calculation method can also be used to find the center of gravity, the maximum value or the minimum value, etc., and the present invention is not limited to this. Then, find the direction with the smallest change and the direction with the largest change among the 100 data points. Among them, the direction with the smallest amount of change is regarded as the Z-axis direction (that is, the depth direction consistent with the camera shooting direction), the direction with the largest amount of change is regarded as the X-axis direction, and the Y-axis direction is determined through the right-hand coordinate system to determine the key points of the pose. The three-dimensional state information of the point's position information reflects the directional characteristics of the pose key points in the three-dimensional space.

Finally, calculate the pose key points and the depth values of the pose key points in the grabbable area of the item corresponding to each mask area. Among them, the depth value of the pose key point is the coordinate value of the graspable area of the object corresponding to the depth coordinate axis, where the depth coordinate axis is set according to the camera shooting direction, the direction of gravity, or the direction of the vertical line of the plane where the grabable area is located. Certainly. Correspondingly, the depth value is used to reflect the position of the graspable area on the depth coordinate axis. During specific implementation, the origin and direction of the depth coordinate axis can be flexibly set by those skilled in the art. The present invention does not limit the way in which the origin of the depth coordinate axis is set. For example, when the depth coordinate axis is set according to the camera's shooting direction, the origin of the depth coordinate axis can be the position of the camera, and the direction of the depth coordinate axis is the direction from the camera to the object. Therefore, the depth of the mask of each grabbable area The value corresponds to the inverse of the distance from the grabbable area to the camera, that is, the further away from the camera, the lower the depth value of the mask, and the depth value is used as the mask height feature value.

The gripper size refers to the size of the gripper configured for an item to be grasped. Since the grabbable area of an object is on the surface of the object, using a clamp to grab the item essentially means controlling the clamp to perform a grabbing operation in the grabbable area. Therefore, the size of the clamp can also be counted as a mask for the grabbable area of the item. Characteristics. The impact of the size of the clamp on grasping is mainly reflected in whether the clamp may accidentally touch items that do not correspond to the clamp. For example, if a large-sized suction cup is used, compared with using a small-sized suction cup, the large-sized suction cup is more likely to collide with other objects during the grabbing process, causing the suction cup to shake. Or the object's position changes, which may lead to grasping failure. In actual industrial scenarios, what kind of clamps are used for each system may be predetermined. That is to say, the size of the clamps may have been determined before actual grabbing. Therefore, the size of the clamps in this embodiment can be based on the configuration. To obtain the fixture, as well as the pre-established and saved mapping relationship between the fixture and its size.

For step S430, the data called in step S420 are combined to generate a visualization layer that can be viewed by the user, assuming that the user has selected "display by pose height" and "display by suction cup size" and the grabbing auxiliary data also includes grabbizable Take the mask of the area as an example: when the user selects "Display by pose height", the mask of each item to be grabbed in the original image is called, as well as the mask height feature value of each item to be grabbed, and Generate a layer that places the mask height feature value next to the corresponding mask; when the user selects "Display by sucker size", the mask of each item to be grabbed in the original image is called, as well as the mask of each item to be grabbed. Get the suction cup size feature value of the item and generate a layer that places the suction cup size feature value next to the corresponding mask;

For step S440, the capture auxiliary layer generated in step S430 is synthesized with the original captured image data, and displayed to the user in a visual manner. You can process the layer generated in step S430, adjust the color, transparency, contrast and other attributes of the layer, and then sequentially combine all the pixels in the auxiliary image layer with the layer from left to right and from top to bottom. All pixels in the original image data are combined together to generate synthesized image data. As shown in Figure 3b, the synthesized image shows the image of each item to be grasped, as well as the mask of the graspable area covering the item to be grasped, and the user-selected "position" displayed next to the mask. "Attitude Height" value or "Suction Cup Size" value.

In addition, it should be noted that although each embodiment of the present invention has a specific combination of features, further combinations and cross-combinations of these features between embodiments are also feasible.

Figure 4 shows an image data processing device according to yet another embodiment of the present invention. The device includes:

The image data acquisition module 800 is used to acquire image data including one or more items to be grabbed, that is, to implement step S400;

The interactive interface display module 810 is used to output the image data and operable controls to form an interactive interface. The controls can be operated by the user to select the capture auxiliary image and display the selected capture auxiliary image to the user, that is, for Implement step S410;

The auxiliary data acquisition module 820 is used to obtain the grabbing auxiliary data corresponding to the grabbing auxiliary image selected by the user in response to the user's operation on the control, that is, to implement step S420;

The auxiliary layer generation module 830 is used to generate a grabbing auxiliary layer based on the obtained grabbing auxiliary data, that is, to implement step S430;

The auxiliary image generation module 840 is used to combine the grabbing auxiliary layer with image data including one or more items to be grabbed to generate a grabbing auxiliary image selected by the user, that is, to implement step S440.

It should be understood that in the above device embodiment shown in Figure 4, only the main functions of the modules are described. All functions of each module correspond to the corresponding steps in the method embodiment. The working principles of each module can also be referred to the method embodiment. Description of the corresponding steps in . For example, in the above embodiment, the auxiliary image generation module 840 is used to implement the method of step S440, indicating that the content used to describe and explain step S440 is also the content used to describe and explain the function of the auxiliary image generation module 840. In addition, although the corresponding relationship between the functions of the functional modules and the methods is limited in the above embodiments, those skilled in the art can understand that the functions of the functional modules are not limited to the above corresponding relationships, that is, specific functional modules can also implement other method steps. or part of a method step. For example, the above embodiment describes that the auxiliary image generation module 840 is used to implement the method of step S440. However, according to the needs of the actual situation, the auxiliary image generation module 840 can also be used to implement the method of steps S400, S410, S420 or S430. part.

This application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method of any of the above embodiments is implemented. It should be pointed out that the computer program stored in the computer-readable storage medium in the embodiment of the present application can be executed by the processor of the electronic device. In addition, the computer-readable storage medium can be a storage medium built in the electronic device, or can be a storage medium that can The storage medium is plugged into the electronic device in a pluggable manner. Therefore, the computer-readable storage medium in the embodiment of the present application has high flexibility and reliability.

Figure 5 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic device may be a control system/electronic system configured in a car, a mobile terminal (for example, a smart mobile phone, etc.), a personal computer (PC, for example) , desktop computer or notebook computer, etc.), tablet computer, server, etc. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

As shown in Figure 5, the electronic device may include: a processor (processor) 1202, a communications interface (Communications Interface) 1204, a memory (memory) 1206, and a communication bus 1208.

in:

The processor 1202, the communication interface 1204, and the memory 1206 complete communication with each other through the communication bus 1208.

The communication interface 1204 is used to communicate with network elements of other devices such as clients or other servers.

The processor 1202 is configured to execute the program 1210, and specifically can execute the relevant steps in the above method embodiment.

Specifically, program 1210 may include program code including computer operating instructions.

The processor 1202 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the electronic device may be the same type of processor, such as one or more CPUs; or they may be different types of processors, such as one or more CPUs and one or more ASICs.

Memory 1206 is used to store programs 1210. The memory 1206 may include high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

Program 1210 may be downloaded and installed from the network through communication interface 1204, and/or installed from removable media. When the program is executed by the processor 1202, the processor 1202 can be caused to perform various operations in the above method embodiment.

In the description of this specification, reference to the description of the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" is intended to be in conjunction with the described implementation. A specific feature, structure, material, or characteristic described in a manner or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments, or portions of code that include one or more executable instructions for implementing the specified logical functions or steps of the process. , and the scope of the preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in a substantially simultaneous manner or in the reverse order, depending on the functionality involved, which shall It should be understood by those skilled in the technical field to which the embodiments of this application belong.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, may be considered a sequenced list of executable instructions for implementing the logical functions, and may be embodied in any computer-readable medium, For use by, or in combination with, instruction execution systems, devices or equipment (such as computer-based systems, systems including processing modules, or other systems that can fetch instructions from and execute instructions from the instruction execution system, device or equipment) or equipment. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wires (electronic device), portable computer disk cartridges (magnetic device), random access memory (RAM), Read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, and subsequently edited, interpreted, or otherwise suitable as necessary. process to obtain the program electronically and then store it in computer memory.

The processor can be a Central Processing Unit (CPU), other general-purpose processors, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), or off-the-shelf programmable processors. Gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

It should be understood that various parts of the embodiments of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented in hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a logic gate circuit with a logic gate circuit for implementing a logic function on a data signal. Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

Those of ordinary skill in the art can understand that all or part of the steps involved in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The program can be stored in a computer-readable storage medium. When executed, one of the steps of the method embodiment or a combination thereof is included.

In addition, each functional unit in various embodiments of the present application can be integrated into a processing module, each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

The storage media mentioned above can be read-only memory, magnetic disks or optical disks, etc.

Although the embodiments of the present application have been shown and described above, it can be understood that the above-mentioned embodiments are illustrative and cannot be understood as limitations of the present application. Those of ordinary skill in the art can make modifications to the above-mentioned embodiments within the scope of the present application. The embodiments are subject to changes, modifications, substitutions and variations.

Claims (12)

1. A method for generating grabbing auxiliary images, characterized by including: Obtain image data including one or more items to be grabbed; Output the image data and operable controls to form an interactive interface, the controls being operable by a user to select a capture auxiliary image and display the selected capture auxiliary image to the user; In response to the user's operation on the control, obtaining capture auxiliary data corresponding to the capture auxiliary image selected by the user; Generate a crawling auxiliary layer based on the obtained crawling auxiliary data; Combining the grasping auxiliary layer with image data including one or more items to be grasped to generate a user-selected grasping auxiliary image; The grasping auxiliary data includes: a numerical value associated with the grasping auxiliary image selected by the user and a mask of the graspable area of the item to be grasped. 2. The method for generating grasping auxiliary images according to claim 1, characterized in that the image data and the operable controls are in the same interactive interface. 3. The method for generating grasping auxiliary images according to claim 1, characterized in that the image data and the operable controls are in different interactive interfaces. 4. The method for generating grabbing auxiliary images according to claim 3, wherein the different interactive interfaces are switched in response to user operations. 5. The method for generating a grabbing auxiliary image according to any one of claims 1 to 4, wherein the grabbing auxiliary layer is combined with image data including one or more items to be grabbed. Combining includes: after adjusting the color, transparency and/or contrast of the grabbing auxiliary layer, combining the adjusted grabbing auxiliary layer with image data including one or more items to be grabbed. 6. A device for generating capture auxiliary images, characterized by comprising: An image data acquisition module, used to acquire image data including one or more items to be grabbed; An interactive interface display module, configured to output the image data and operable controls to form an interactive interface. The controls can be operated by the user to select the capture auxiliary image and display the selected capture auxiliary image to the user; An auxiliary data acquisition module, configured to acquire auxiliary grasping data corresponding to the auxiliary grasping image selected by the user in response to the user's operation on the control; An auxiliary layer generation module, configured to generate an auxiliary crawling layer based on the acquired auxiliary crawling data; An auxiliary image generation module, configured to combine the grabbing auxiliary layer with image data including one or more items to be grabbed to generate a user-selected grabbing auxiliary image; The grasping auxiliary data includes: a numerical value associated with the grasping auxiliary image selected by the user and a mask of the graspable area of the item to be grasped. 7. The device for generating grasping auxiliary images according to claim 6, wherein the image data and the operable controls are in the same interactive interface. 8. The device for generating grasping auxiliary images according to claim 6, wherein the image data and the operable controls are in different interactive interfaces. 9. The device for generating grasping auxiliary images according to claim 8, wherein the different interactive interfaces are switched in response to user operations. 10. The device for generating an auxiliary grabbing image according to any one of claims 6 to 9, characterized in that the auxiliary image generation module is also used to: adjust the color, transparency and transparency of the auxiliary grabbing layer. After/or contrasting, the adjusted grasping auxiliary layer is combined with image data including one or more items to be grasped. 11. An electronic device, characterized in that it includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the claims are realized. The method for generating a grabbing auxiliary image according to any one of 1 to 5. 12. A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the method for generating a grabbing auxiliary image according to any one of claims 1 to 5 is implemented. .