CN114445745A - Method, system, apparatus and storage medium for identifying stacked articles - Google Patents

Abstract

The present disclosure provides a method for identifying stacked items. In the method, the items in the image with the stacked items are marked to obtain marked areas corresponding to the items in the image respectively; the marked areas are rotated so that each marked area is used as a reference area at least once; the marked areas are screened and processed to obtain Corresponding to each reference area and detecting the middle area with the highest confidence; the items marked by the reference area and the corresponding middle area are identified as stacked items. This method can effectively improve the detection rate of stacked items missed due to line detection failure.

Description

Method, system, device and storage medium for identifying stacked items

technical field

The present disclosure belongs to the technical field of image processing in computer vision, and in particular, relates to a method, system, device and storage medium for recognizing stacked objects.

Background technique

There is no shortage of stacking items on the shelves. Since the stacked items are usually the same as the products placed at the bottom, which objectively causes invalid card surface occupation, the relevant manufacturers will actively ignore the stacked items when calculating the brand card surface ratio of the product.

In the prior art, overlapping identification is usually implemented based on line detection. A row in the prior art refers to each level of a shelf.

SUMMARY OF THE INVENTION

An aspect of the present disclosure provides a method for identifying stacked items. The method for identifying stacked items includes the following steps:

acquiring an image with stacked items, identifying and marking the items in the image, to obtain marked areas corresponding to the items in the image respectively;

Rotating the marked regions such that each of the marked regions is used as a reference region at least once;

Screening and processing the marked areas to obtain an intermediate area corresponding to each of the reference areas and having the highest detection confidence; identifying the items marked by the reference area and the corresponding intermediate areas as stacked items.

According to an embodiment of the present disclosure, the step of identifying the items marked in the reference area and the corresponding intermediate area as stacked items further includes:

Re-screening the marked area to obtain a target area corresponding to each of the intermediate areas and with the highest detection confidence;

Comparing the detection confidence level to obtain a comparison result of the detection confidence level of the reference area corresponding to the intermediate area and the target area respectively;

Identifying the comparison result to identify the items marked by the reference area and the corresponding target area as stacked items; wherein the detection confidence of the reference area is not lower than the detection confidence of the target area .

According to an embodiment of the present disclosure, before the step of rotating the marked area, the step further includes:

The marked regions are sorted so that the marked regions are rotated sequentially.

According to an embodiment of the present disclosure, the value range of the second threshold in the screening process is 0.6˜1.

According to an embodiment of the present disclosure, two or more marked areas identified as stacked items are combined into one marked area.

Another aspect of the present disclosure provides an identification system for stacked items. The identification system for stacked items is used to implement the aforementioned method for identifying stacked items. The identification system includes:

an item marking module, used for acquiring an image with stacked items, identifying and marking the items in the image, so as to obtain marked areas corresponding to the items in the image respectively;

an area rotation module, configured to rotate the marked areas so that each of the marked areas is used as a reference area at least once;

a screening and processing module for screening and processing the marked regions to obtain an intermediate region corresponding to each of the reference regions and having the highest detection confidence;

The stacking identification module is configured to identify the items marked by the reference area and the corresponding intermediate area as stacked items.

Yet another aspect of the present disclosure also provides an identification device for stacked items. The identification device includes:

a memory and a processor, the memory having instructions stored therein, the memory and the processor being interconnected by wires;

The processor invokes the instructions in the memory to implement the method for recognizing stacked objects in an embodiment of the present disclosure.

A final aspect of the present disclosure provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, implements the method for recognizing stacked objects in an embodiment of the present disclosure.

Compared with the prior art, the present disclosure has the following advantages and positive effects due to the adoption of the above technical solutions:

1) The method for recognizing stacked items in an embodiment of the present disclosure is to mark items in an image with stacked items to obtain marked areas corresponding to the items in the image respectively; and rotate the marked areas so that each marked area It is used as a reference area at least once; the marked area is screened and processed to obtain an intermediate area corresponding to each reference area and the detection confidence is the highest; the items marked by the reference area and the corresponding intermediate area are identified as stacked items. Compared with the prior art, the detection success rate of stacked objects is further improved.

2) In the method for identifying stacked items in an embodiment of the present disclosure, two or more marked areas identified as stacked items are combined into one marked area. Will not be double-counted for stacking items.

3) In the method for recognizing stacked objects in an embodiment of the present disclosure, in order to facilitate the rotation of the marked regions, the marked regions are sorted, so as to rotate the marked regions in sequence to avoid repetition or omission, thereby improving the recognition rate of object stacking.

Description of drawings

FIG. 1 is a flowchart of a method for identifying stacked objects in an embodiment of the disclosure;

FIG. 2 is a block diagram of an identification system for stacked objects in an embodiment of the disclosure;

FIG. 3 is a schematic diagram of a device for identifying stacked objects in an embodiment of the disclosure;

FIG. 4 is a schematic diagram of a computer-readable storage medium in an embodiment of the disclosure.

Detailed ways

The method, system, device and storage medium for recognizing stacked articles proposed in the present disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments. The advantages and features of the present disclosure will become apparent from the following description and claims.

Example 1

Current stack recognition is based on row detection, where "row" in the prior art refers to each level area on the shelf. Stacking occurs when multiple products are in the same row with a certain degree of overlap in the vertical direction of the shelf level.

In the specific implementation process, the existing detection scheme has failures in line detection. Generally, in order to clearly photograph large-scale packaged products in warehouse supermarkets or objects with excessive line spacing such as roll paper, the camera frame may be lost on the shelf. Some features, such as laminates, columns, etc., make the detection model in the existing detection scheme unable to effectively detect the features of the row, and eventually lead to frequent failure of row detection.

In view of the above problems, the present embodiment provides a method for identifying stacked items, please refer to FIG. 1 , and the method for identifying stacked items includes the following steps:

Step S100: acquiring an image with stacked items;

Step S200: marking the items in the image to obtain marked areas corresponding to the items in the image respectively;

Step S300: Rotate the marked areas so that each marked area is used as a reference area at least once;

Step S400: Screening and processing the marked area to obtain an intermediate area corresponding to each reference area and having the highest detection confidence;

Step S500: Identify the items marked in the reference area and the corresponding intermediate area as stacked items.

Specifically, in step S100, a live image of a scene of stacking items captured in real time can be obtained, where the on-site image is an image of the storage of items on the shelf, including the shelf and the items on the shelf.

In step S200, the items in the image are marked to obtain marked regions corresponding to the items in the image respectively. In practical applications, images with stacked items obtained on site can be input into a preset item recognition model for item recognition, and a bounding box (ie, a marked area) surrounding a single item can be generated.

The item recognition model may be a MASK R-CNN model, and the MASK R-CNN model includes a ResNet-101 network, an RPN network, an ROI Align layer, and a classification network. The recognition principle of the MASK R-CNN model is as follows:

Input the image into the preset MASK R-CNN model, extract the image features through the ResNet-101 network, and obtain the first feature map; input the first feature map into the RPN network, and perform the first feature map through the preset selective search algorithm. The prediction frame is extracted to obtain the prediction frame corresponding to the first feature map; the first feature map and the prediction frame are input into the ROI Align layer for prediction, and the second feature map including the prediction frame is obtained; The connection layer performs classification processing to obtain the predicted probability that the predicted frame contains the item; based on the predicted probability that the predicted frame contains the item, the image of the item marked by the marked area is obtained.

In practical applications, different item recognition models can be selected according to application scenarios, which is not limited to the MASK R-CNN model in this embodiment, and can also be an item recognition model based on the YOLO series.

In addition, a static image detector can be used to label the items in the image and generate bounding boxes for each item. The still image detector is DeepID-Net or CRAFT still image detector.

In step S300, the marked areas of the article are rotated so that each marked area is used as a reference area at least once. The rotation here refers to taking each marked area as a reference area in turn for screening.

In order to facilitate the rotation of the marked areas, the marked areas can be sorted, so as to rotate the marked areas in sequence, to avoid repetition or omission, thereby improving the recognition rate of subsequent article stacking.

In step S400, the marked area of the processed item is screened to obtain an intermediate area corresponding to each reference area and having the highest detection confidence.

In order to more clearly describe the content of the solution provided by the present disclosure, the present disclosure provides a unified definition in each embodiment, and defines the extending direction of the stacking of items on the shelf as the first direction, and the second direction is defined as the same as the first direction Opposite Direction.

For the concept of "direction" involved in various embodiments of the present disclosure, more specific orientation nouns are usually used in practical applications. For example, in the present disclosure, the indicated shelves are usually conventional shelves placed on the ground of supermarkets, convenience stores, and the like. Since each layer of the shelf is parallel to each other, and in order to make the items placed more stable, each layer of the shelf is also parallel to the ground. It is easy to understand that the floors of supermarkets, convenience stores, etc. that are put into commercial use can often be considered to be in a horizontal state, so those skilled in the art can regard each layer of a shelf as a horizontal state in the present disclosure. Relative to the ground where the shelves are located, the vertical upward direction, that is, the vertical positive direction, is consistent with the extending direction of the stacking of items. Therefore, in each embodiment of the present disclosure, the first direction defined above can be specifically expressed as The vertical positive direction, while the second direction can be specifically expressed as the vertical negative direction.

The screening process in the present disclosure refers to selecting other marked regions whose distance from the reference region in the first direction is not greater than the first threshold value, as screening candidate regions; and continuing to select the projection overlap degree on the plane perpendicular to the first direction with the reference region. The screening candidate regions that are not less than the second threshold and have similar shapes are used as the screening pending regions; the marked regions with the highest detection confidence are selected from the screening pending regions to obtain the intermediate regions. Similar shape refers to an area in which the difference in aspect ratio and the difference in area size between the marked areas are not greater than the third threshold and the fourth threshold, respectively.

The above-mentioned first threshold, second threshold, third threshold and fourth threshold may be set according to actual needs. Combined with the actual use effect, when faced with conventional supermarket shelves, when the value range of the second threshold in the present disclosure is set to 0.6 to 1, it can bring more accurate detection results, and is suitable for most supermarket shelves in the market. The image obtained by shooting can also reduce the processing intensity of the computer and improve the recognition efficiency. When the value of the third threshold and/or the fourth threshold is set larger, it can be considered that the shapes of the selected marked regions tend to be more consistent.

Based on the definition of "direction", in this embodiment, the marked boxes containing items may be sorted in the vertical direction to construct a connected graph of marked areas; the marked areas are rotated in the connected graph so that at least one marked area is used as a reference area. When the labeled regions are rotated in the connectivity graph, each labeled region can also be used as a reference region at least once.

Step S400 is described below by taking the item bounding box a as an example:

For each box a (reference area), along the vertical positive direction, find the bounding box of the item whose distance from the box a is less than the first threshold, as a screening candidate area. When the bounding box is a rectangle, the distance is the distance between the upper top edge of the lower rectangle and the lower bottom edge of the upper rectangle. The first threshold can be adjusted according to the height of different items, for example, according to the size of the current bounding box (only the height of the box is used to represent the size of the box) multiplied by a coefficient reasonably set according to the actual data. For example, the line spacing of toothpaste will be smaller than the line spacing of roll paper.

From the screening candidate area, the marked items whose overlapping degree in the horizontal direction and the marked item is greater than the second threshold and are similar in shape are selected as the screening pending area. In the screening undetermined area, the marked area with the highest detection confidence is selected to obtain the intermediate area.

That is, from the above-mentioned screening candidate regions, an item bounding box with a horizontal overlap with the frame a greater than the second threshold and a similar shape is selected as the screening pending region. Select the marked area with the highest detection confidence in the undetermined area to be screened to obtain the intermediate area a_i.

It should be pointed out that the overlap in the horizontal direction here can be understood as the overlap of projections on the horizontal plane. If all items are projected on the same horizontal plane (that is, projected in the vertical direction), in general, there will be a large horizontal overlap only when the target item, that is, a, is in a state of being listed up and down. Therefore, by setting a second threshold, items with a high degree of overlap can be filtered out. After analysis, in the embodiment of the present disclosure, setting the second threshold to 0.7 can obtain an ideal screening result.

The degree of overlap in this disclosure is used to measure the degree of overlap between two or more objects. Generally, it can be considered that the value of the overlap degree is between 0 and 1, where 0 means no overlap at all, and 1 means complete overlap.

In step S500, the items marked by the reference area and the corresponding intermediate area are identified as stacked items.

Specifically, the marked area is re-screened to obtain the target area corresponding to each intermediate area and the detection confidence is the highest; the detection confidence is compared to obtain the comparison of the detection confidence of the reference area and the target area corresponding to the intermediate area respectively. Result: Identify the comparison result to identify the items marked by the reference area and the corresponding target area as stacked items; wherein, the detection confidence of the reference area is not lower than the detection confidence of the target area.

The re-screening process here refers to selecting other marked areas whose distance from the middle area in the second direction is not greater than the first threshold value, as the re-screening candidate area; continue to select the projection overlap degree on the plane perpendicular to the second direction with the middle area The re-screening candidate regions that are not less than the second threshold and have similar shapes are used as the re-screening pending regions; the marked regions with the highest detection confidence are selected from the re-screening pending regions to obtain the target regions.

The shape similarity is consistent with the shape similarity requirement in the screening process in the above step S400.

In practical applications, starting from the middle area a_i, along the vertical negative direction, select other marked areas with a distance not greater than the first threshold as the candidate area for re-screening; continue to select the projection on the plane in the vertical negative direction with the middle area The re-screening candidate regions with a degree of overlap not less than the second threshold and similar shapes are used as re-screening pending regions; the marked regions with the highest detection confidence are selected from the re-screening pending regions to obtain target regions a_j.

The vertical negative direction and the vertical positive direction in the present disclosure are relative concepts, that is, the vertical positive direction is in step S400, then this step must be opposite to the vertical direction in step S400, that is, the vertical negative direction.

After the above-mentioned multiple steps of screening and exclusion, the detection confidence is continued to be verified. At this time, if the detection confidence is not lower than a_j, it can be considered that a to a_i belong to the same group of stacked items.

After the stacked items are identified, the two or more marked areas identified as stacked items are combined into one marked area, so that when calculating the proportion of the brand's card surface, it will not be repeated due to the stacking of items Calculation, the stacked items can be automatically ignored, so as to improve the calculation speed and improve the accuracy of the card face ratio.

Based on the above description, it can be seen that the method for identifying stacked objects provided by the present disclosure can effectively reduce the problem of missed detection due to failure of line detection.

Embodiment 2

This embodiment provides an identification system for stacked items, please refer to FIG. 2 , the identification system for stacked items includes:

The item marking module 110 is used for acquiring an image with stacked items, identifying and marking items in the image, so as to obtain marked areas corresponding to the items in the image respectively;

an area rotation module 120, configured to rotate marked areas, so that each marked area is used as a reference area at least once;

The screening processing module 130 is used for screening and processing the marked region to obtain the intermediate region corresponding to each reference region and having the highest detection confidence;

The stacking identification module 140 is configured to identify the items marked in the reference area and the corresponding intermediate area as stacked items.

The identification system for stacked items further includes a re-screening processing unit and a comparing unit. Among them, the re-screening processing unit is used to re-screen and process the marked area to obtain the target area corresponding to each intermediate area and the highest detection confidence; compare the detection confidence level to obtain the detection of the reference area and the target area corresponding to the intermediate area respectively The comparison result of the confidence level; identify the comparison result to identify the item marked between the reference area and the corresponding target area as a stacked item; wherein, the detection confidence level of the reference area is not lower than the detection confidence level of the target area Spend.

The comparison unit is used for comparing the detection confidence level, so as to obtain a comparison result of the detection confidence level of the reference area and the target area corresponding to the intermediate area respectively.

In addition, the stack identification module 140 can also be used to identify the comparison result obtained by the comparison unit, so as to identify the items marked by the reference area and the corresponding target area as stacked items. It should be noted that, under the condition that the comparison result is that the detection confidence of the reference area is not lower than the detection confidence of the target area, the stacking identification module 140 identifies the stacked items referred to in this paragraph.

Compared with the prior art, the identification system for stacked items provided in this embodiment can obtain the effective features of stacked items more quickly by screening the item spacing and detection confidence, thereby improving the system's ability to identify stacking items. In addition, through the steps of rotating and sorting the marked area, the corresponding identification operations are carried out in an orderly manner, which avoids the problems of duplication and omission of identification, thereby reducing the occupation ratio of computing resources.

Embodiment 3

This embodiment provides an identification device for stacked articles. Referring to FIG. 3 , the identification device 500 of the stacked items may vary greatly due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (eg, one or more processors) ) and memory 520, one or more storage media 530 (eg, one or more mass storage devices) storing applications 533 or data 532. Among them, the memory 520 and the storage medium 530 may be short-term storage or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown), and each module may include a series of instruction operations in the identification device 500 for stacked items.

Further, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the identification device 500 of the stacked items.

The identification device 500 of stacked items may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and/or, one or more operating systems 531, Such as Windows Serve, Vista and so on.

Those skilled in the art can understand that the structure of the identification device for stacked objects shown in FIG. 3 does not constitute a limitation on the identification device for stacked objects, and may include more or less components than those shown in the figure, or combine some components , or a different component arrangement.

Another embodiment of the present disclosure also provides a computer-readable storage medium.

FIG. 4 depicts a program storage product 600 for implementing the identification methods disclosed in the preceding embodiments, in accordance with an embodiment. The computer-readable storage medium may be a non-volatile computer-readable storage medium, and the computer-readable storage medium may also be a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions are executed on the computer, the computer executes the steps of the method for recognizing stacked objects in the first embodiment.

The method for identifying the stacked items, if implemented in the form of program instructions and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present disclosure can be embodied in the form of software, in essence or in part that contributes to the prior art, or all or part of the technical solutions. The computer software is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

Those skilled in the art can clearly understand that, for the convenience and brevity of the description, for the identification content of the specific execution of the system and device described above, reference may be made to the corresponding process in the foregoing method embodiments.

The embodiments of the present disclosure are described above in detail with reference to the accompanying drawings, but the present disclosure is not limited to the above-mentioned embodiments. Even if various changes are made to the present disclosure, if such changes fall within the scope of the claims of the present disclosure and their equivalents, they still fall within the protection scope of the present disclosure.

Claims (10)

1. an identification method of stacked articles, is characterized in that, comprises: Get an image with stacked items; marking the items in the image to obtain marked areas respectively corresponding to the items in the image; Rotating the marked regions such that at least one of the marked regions is used as a reference region; Screening and processing the marked regions to obtain an intermediate region corresponding to each of the reference regions and having the highest detection confidence; The items marked by the reference area and the corresponding intermediate area are identified as stacked items. 2 . The method for recognizing stacked articles according to claim 1 , wherein the step of recognizing the articles marked by the reference area and the corresponding intermediate area as stacked articles further comprises: 3 . Re-screening the marked area to obtain a target area corresponding to each of the intermediate areas and with the highest detection confidence; Comparing the detection confidence level to obtain a comparison result of the detection confidence level of the reference area corresponding to the intermediate area and the target area respectively; Identifying the comparison result to identify the items marked by the reference area and the corresponding target area as stacked items; wherein the detection confidence of the reference area is not lower than the detection confidence of the target area . 3. The method for recognizing stacked articles according to claim 1, wherein the step of rotating the marked areas so that at least one marked area is used as a reference area further comprises: ordering the labeled regions along a first direction to construct a connected graph of the labeled regions; The marked regions are rotated in the connectivity graph such that at least one of the marked regions is used as a reference region. 4. The method for recognizing stacked articles according to claim 3, wherein the marked regions are rotated in the connectivity graph so that each of the marked regions is used as a reference region at least once. 5 . The method for recognizing stacked articles according to claim 1 , wherein the value range of the second threshold in the screening process is 0.6˜1. 6 . 6. The method for recognizing stacked objects according to claim 5, wherein the second threshold is 0.7. 7. The method for identifying stacked articles according to claim 1 or 2, wherein two or more marked areas identified as stacked articles are combined into one marked area. 8. An identification system for stacked articles, for implementing the steps of the method for identifying stacked articles according to any one of claims 1-7, wherein the identification system comprises: an item marking module, used for acquiring an image with stacked items, identifying and marking the items in the image, so as to obtain marked areas corresponding to the items in the image respectively; an area rotation module, configured to rotate the marked areas so that at least one of the marked areas is used as a reference area; a screening and processing module for screening and processing the marked regions to obtain an intermediate region corresponding to each of the reference regions and having the highest detection confidence; The stacking identification module is configured to identify the items marked by the reference area and the corresponding intermediate area as stacked items. 9. An identification device for stacked articles, characterized in that, comprising: a memory and a processor, the memory having instructions stored therein, the memory and the processor being interconnected by wires; The processor invokes the instructions in the memory to implement the method for recognizing stacked objects according to any one of claims 1-7. 10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the computer program according to any one of claims 1-7 is implemented. How to identify stacked items.

Images