CN116109992A - Goods shelf commodity identification method and system - Google Patents

Abstract

The invention discloses a method and a system for identifying goods on a goods shelf, and relates to the technical field of image identification. The method for identifying the goods on the goods shelf comprises the following steps: acquiring a shelf image, positioning a shelf to be detected and framing the area of the shelf to be detected; coarse detection is carried out on the framed area through global detection, commodities are classified according to rows, and images are segmented according to classification results to obtain commodity images to be detected; performing character recognition on the commodity image to be detected to preliminarily determine commodity category; image feature comparison and color comparison are carried out on the commodity image to be detected; obtaining a final commodity category through information fusion; judging the commodity category according to the template category information and the category information to be detected, and outputting a commodity identification result. The method and the device not only solve the problem that the efficiency of manually managing the goods on the goods shelf is low and errors are easy to occur, but also overcome the defect that the error rate of image identification by a single method in the existing goods shelf goods identification method is high.

Description

A method and system for identifying goods on a shelf

technical field

The invention relates to the technical field of image recognition, in particular to a method and system for recognizing goods on a shelf.

Background technique

The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

Shelf display is the main commodity display method in shopping malls and supermarkets. Reasonable commodity placement can attract customers, facilitate shopping, and improve store performance. At the same time, it is also possible to know the supply and demand level of the commodity according to the stock and quantity changes of the corresponding commodity on the shelf every day, so as to realize intelligent purchase management.

At present, shelf commodity auditing, that is, shelf placement and inspection, is mainly done manually. On the one hand, the tally staff needs to determine the quantity of goods, and on the other hand, they need to scan and place the goods one by one. Due to the wide variety of commodities and the tight arrangement of the shelves, it is easy for the products to be placed incorrectly or missing. In addition, as time goes by, the goods in the shelves are gradually sold out, and timely replenishment is required. Manual inspections are slow and inaccurate, and it is time-consuming and labor-intensive, and it is difficult to accurately find problems.

Contents of the invention

In view of the deficiencies in the existing technology, the purpose of the present invention is to provide a method and system for identifying goods on the shelf, which can identify and analyze the goods on the shelf in real time according to the pre-designed placement based on the image analysis technology, with real-time and flexible , convenience, low resource usage, and strong scalability.

In order to achieve the above object, the present invention is achieved through the following technical solutions:

The first aspect of the present invention provides a method for identifying goods on a shelf, including the following steps:

Collect shelf images, locate the shelf to be tested and frame the area of the shelf to be tested;

Roughly detect the framed area through global detection to obtain the information of the required goods;

Classify the products by rows according to the information of the required products, and segment the images according to the classification results to obtain images of the products to be detected;

Carry out text recognition on the image of the commodity to be detected to initially determine the category of the commodity;

Carry out image feature comparison and color comparison on the image of the product to be tested;

Information fusion of text recognition results and image feature comparison and color comparison information to obtain the final product category;

The product category is judged according to the template category information and the category information to be detected, and the product identification result is output.

Further, the shelf to be tested is positioned and the area of the shelf to be tested is framed by means of back-code positioning and perspective transformation processing.

Further, a rough detection is performed on the framed area through the global detection, and the specific steps for obtaining the information of the desired commodity include:

Use the YOLOv5 network to perform global rough detection on the picture, and obtain the rough detection result;

According to the position of the return code, the average height of the product center, and the number of shelf rows, the rough detection results are filtered and supplemented to obtain the position information and size information of the required product in the picture.

Further, the specific steps of performing text recognition on the image of the commodity to be detected to initially determine the category of the commodity are as follows:

Use the pre-trained model to extract text from the segmented image of the product to be detected;

Query according to the extracted text information and keyword database to determine the commodity category.

Further, the specific steps for comparing the image features of the product images to be tested are as follows:

Extract pattern features from the image of the product to be detected;

The extracted features are subjected to repeated feature filtering to obtain comparison information.

Further, the specific steps for color comparison of the images of the products to be tested are as follows:

Taking advantage of the insensitivity of HSV color space to light, the original RGB color space is converted and compressed into HSV color space as color features, and the consistency between color features and main colors is compared, and the comparison information is output.

Further, the specific steps for obtaining the final commodity category are as follows:

Based on the text recognition results, when the features are similar, the pattern features and color comparison information are added to get the final result.

The second aspect of the present invention provides an identification system for goods on shelves, including:

The positioning module is configured to collect shelf images, locate the shelf to be tested and frame the area of the shelf to be tested;

The detection module is configured to roughly detect the framed area through global detection to obtain the information of the required commodity; classify the commodity by row according to the information of the required commodity, and segment the image according to the classification result to obtain the image of the commodity to be detected;

The identification module is configured to perform text recognition on the image of the product to be detected to initially determine the product category; perform image feature comparison and color comparison on the product image to be tested; perform information fusion of the text recognition result and the image feature comparison and color comparison result, The final product category is obtained, and the product category is judged according to the template category information and the category information to be detected, and the product identification result is output.

The third aspect of the present invention provides a medium on which a program is stored, and when the program is executed by a processor, the steps in the method for identifying goods on shelves as described in the first aspect of the present invention are implemented.

The fourth aspect of the present invention provides a device, including a memory, a processor, and a program stored on the memory and operable on the processor, when the processor executes the program, the method described in the first aspect of the present invention is implemented Steps in a method for identifying a shelf item.

The above one or more technical solutions have the following beneficial effects:

The invention discloses a method for identifying goods on a shelf. Based on image analysis technology, the goods on the shelf are identified and analyzed in real time according to the pre-designed placement situation, which not only solves the problem of low efficiency and error-prone manual management of goods on the shelf, The text recognition results are further confirmed by comparing the additional image features and color information, which overcomes the defect that the single method of the existing shelf commodity recognition methods has a high error rate for image recognition.

The present invention can aim at the template category information and the category information to be detected for the obtained product category recognition results, and further judge with correct placement, reverse placement, out of stock, and extra placement as the final category, and take the result with the least number of errors in each row as the final result output, greatly reducing the existing tasks of manual tallying, and providing data support for managers to analyze sales records. The invention can complete the image acquisition process only by ordinary cameras, and has the characteristics of real-time, flexibility, convenience, low resource occupation rate, easy realization, strong scalability and the like.

Advantages of additional aspects of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, and the schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention.

FIG. 1 is a flowchart of a method for identifying goods on shelves in Embodiment 1 of the present invention;

Fig. 2 is the sample diagram of back shape code in the embodiment of the present invention one;

FIG. 3 is a schematic diagram of the identification and judgment process in Embodiment 1 of the present invention;

Fig. 4 is a schematic diagram of image feature comparison and color comparison judgment process in Embodiment 1 of the present invention;

FIG. 5 is a schematic diagram of an ini description file in Embodiment 1 of the present invention.

Detailed ways

It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

Embodiment one:

Embodiment 1 of the present invention provides a method for identifying goods on a shelf. Taking beverage goods as an example, as shown in FIG. 1 , the method includes the following steps:

Image preprocessing:

S1: Collect the shelf image, locate the shelf to be tested and frame the area of the shelf to be tested through zigzag positioning and perspective transformation processing.

S1.1: Return shape code detection performs two binarization processing on the shelf image to be detected, and performs closed interval graphic detection on the binarized black and white image. Using the coincidence degree of the center point of the closed interval, the inclusion relationship of each closed interval (satisfied with the number of layers of the zigzag code), the area ratio of each closed interval, etc., are used as judgment conditions to filter and filter out wrong judgments, so as to obtain the position information of the four zigzag codes in the image. An example of back bar code is shown in Figure 2. In this embodiment, four back-shaped codes are pasted on the upper left corner and upper right corner of the first row of the shelf and the lower left foot and lower right foot of the bottom row respectively, and are used to locate the four vertices of the shelf to be identified.

S1.2: Perspective Transformation Processing Perspective transformation is performed through the position of the four-point back-shaped code to obtain the front view of the shelf, that is, the obtained framed area.

S2: Perform rough detection on the framed area through global detection to obtain the information of the desired product.

S2.1: Use the YOLOv5 network to perform global rough detection on the picture, and obtain the rough detection result. Specifically, the front view of the shelf is sent to the pre-trained YOLOv5 network to obtain the position and size information of some commodities in the picture. As a binary classification task, rough detection only detects the location of the product and does not identify the type.

S2.2: According to the position of the return code, the average height of the product center, and the number of shelf rows, the rough detection results are filtered and supplemented to obtain the position information and size information of the required product in the picture.

S3: Classify the products by row according to the information of the required products, and segment the image according to the classification results to obtain the images of the products to be detected; specifically, according to the recognition results of the trained binary classification network, that is, the position information of each product in the image, The image is further segmented to obtain an independent small image for each product for the next recognition process.

Image feature extraction and comparison:

The identification process of the product category is shown in Figure 3. The image of the product to be tested retrieves the first-level keywords according to the confidence level and obtains the ranking, and conducts the second-level search for the top three confidence levels in the first-level keywords. When the second-level search results are unique When the second-level search results are not unique, search for all kinds of lower-level keywords; when the third-level search results are unique, output the product category; when the third-level search results are not unique, add graphic feature comparison and color comparison As a result, the commodity category is output. In this embodiment, the first-level keyword is the brand name of the product, the second-level keyword is the taste of the product, and the third-level keyword is the net content of the product. The specific process is as follows:

S4: Perform character recognition on the image of the commodity to be detected to initially determine the category of the commodity.

S4.1: Perform text extraction on the segmented product image to be detected by using a pre-trained model; in this embodiment, PaddleOCR is used for text extraction.

Specifically, several commodity description information including commodity name, commodity flavor, and commodity net content are acquired. Retrieve the descriptive information words in the preset keyword database, and determine the commodity category according to the keyword priority, the space occupied by the keyword and other information.

S4.2: Query according to the extracted text information and keyword database to determine the commodity category.

S5: Perform image feature comparison and color comparison on the image of the product to be tested. In this embodiment, the combination of the traditional SIFT algorithm and the RANSAC algorithm is used to supplement the text recognition in the case of less text information and improve the accuracy. The color comparison process judges by converting the high-dimensional values of the RGB color space to the HSV color space.

S5.1: Compare the image features of the images of the products to be tested.

S5.1.1: Use the SIFT+BFmatcher algorithm to extract pattern features from the product image to be detected.

S5.1.2: Perform repeated feature filtering on the extracted features using the RANSAC algorithm to obtain comparison information.

S5.1.3: Match the comparison information with the features of the template gallery, and output the comparison information if the matching result is unique. If it is not unique, skip to S5.1.4 to perform color comparison on the image of the product to be tested.

S5.1.4: Carry out color comparison of the images of the products to be tested.

Specifically, the original RGB color space is converted and compressed into the HSV color space as color features by utilizing the insensitivity of the HSV color space to light, and the consistency between the color features and the main colors is compared, and the comparison information is output.

S6: Fusion of text recognition results, image feature comparison and color comparison information to obtain the final commodity category.

S6.1: Based on the text recognition results, when the features are similar, add pattern features and color comparison information to get the final result.

Specifically, when the text information is vague or the existing information cannot directly confirm the specific product, use pattern features and color comparison information to assist, further screen and confirm on the basis of text classification, and adopt a voting mechanism to determine the product to be detected according to the confidence level The final category of the product.

Analysis and display of recognition results:

S7: Judging the product category according to the template category information and the category information to be detected, and outputting a product identification result.

S7.1: Configure the template category information and the category information to be detected in the initial system parameter configuration process, specifically including commodity barcodes (that is, the type of acquisition), the number of shelf rows, the number of commodity types in each row, the number of commodities in each category, etc. , thus describing the entire shelf.

S7.2: Based on the template category information and the category information to be detected, further judgment is made with correct placement, reverse placement, out of stock, and multi-placement as the final category, and the result with the least number of errors per line is the final result output to obtain the product identification result .

Embodiment two:

Embodiment 2 of the present invention provides an identification system for goods on shelves, including:

The positioning module is configured to collect shelf images, locate the shelf to be tested and frame the area of the shelf to be tested;

The detection module is configured to roughly detect the framed area through global detection to obtain the information of the required commodity; classify the commodity by row according to the information of the required commodity, and segment the image according to the classification result to obtain the image of the commodity to be detected;

The identification module is configured to perform text recognition on the image of the product to be detected to initially determine the product category; perform image feature comparison and color comparison on the product image to be tested; perform information fusion of the text recognition result and the image feature comparison and color comparison result, The final product category is obtained, and the product category is judged according to the template category information and the category information to be detected to output a product recognition result.

Embodiment three:

Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, the steps in the method for identifying goods on shelves as described in Embodiment 1 of the present invention are implemented. The steps are:

Collect shelf images, locate the shelf to be tested and frame the area of the shelf to be tested;

Roughly detect the framed area through global detection to obtain the information of the required goods;

Classify the products by rows according to the information of the required products, and segment the images according to the classification results to obtain images of the products to be detected;

Carry out text recognition on the image of the commodity to be detected to initially determine the category of the commodity;

Carry out image feature comparison and color comparison on the image of the product to be tested;

Information fusion of text recognition results and image feature comparison and color comparison information to obtain the final product category;

According to the category information of the template and the category information to be detected, the product category is judged and the product identification result is output.

Optionally, the system initialization process includes reading and analyzing the ini configuration file and extracting features from the template library. Therefore, it is possible to judge the product category according to the template category information and the category information to be detected and output the product identification result.

The ini description file is mainly used to obtain information such as commodity barcodes (that is, to obtain types), number of shelf rows, number of commodity types in each row, and number of commodities in each category. By reading and analyzing the ini description file, an array that can describe the entire shelf, that is, a standard template array, is formed for comparison with the final test results. The basic format of the ini file is shown in Figure 5.

The template gallery feature extraction is an extension of the standard placement description array, accessing the corresponding position of the template map in the ini file, using the SIFT algorithm to extract the features of each template and storing them for comparison.

The detailed steps are the same as the method for identifying goods on the shelf provided in Embodiment 1, and will not be repeated here.

Embodiment four:

Embodiment 4 of the present invention provides a device, which includes a memory, a processor, and a program stored in the memory and operable on the processor. When the processor executes the program, the device described in Embodiment 1 of the present invention The steps in the method for identifying goods on the shelf, the steps are:

Collect shelf images, locate the shelf to be tested and frame the area of the shelf to be tested;

Roughly detect the framed area through global detection to obtain the information of the required goods;

Classify the products by rows according to the information of the required products, and segment the images according to the classification results to obtain images of the products to be detected;

Carry out text recognition on the image of the commodity to be detected to initially determine the category of the commodity;

Carry out image feature comparison and color comparison on the image of the product to be tested;

Information fusion of text recognition results and image feature comparison and color comparison information to obtain the final product category;

According to the category information of the template and the category information to be detected, the product category is judged and the product identification result is output.

The detailed steps are the same as the method for identifying goods on the shelf provided in Embodiment 1, and will not be repeated here.

The steps involved in the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and for specific implementation methods, please refer to the relevant description part of the embodiment 1. The term "computer-readable storage medium" shall be construed to include a single medium or multiple media including one or more sets of instructions; and shall also be construed to include any medium capable of storing, encoding, or carrying A set of instructions to execute and cause the processor to execute any method in the present invention.

Those skilled in the art should understand that each module or each step of the present invention described above can be realized by a general-purpose computer device, optionally, they can be realized by a program code executable by the computing device, thereby, they can be stored in a memory The device is executed by a computing device, or they are made into individual integrated circuit modules, or multiple modules or steps among them are made into a single integrated circuit module for realization. The invention is not limited to any specific combination of hardware and software.

Although the specific implementation of the present invention has been described above in conjunction with the accompanying drawings, it does not limit the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, those skilled in the art do not need to pay creative work Various modifications or variations that can be made are still within the protection scope of the present invention.

Claims (10)

1. The method for identifying the goods on the goods shelf is characterized by comprising the following steps:

acquiring a shelf image, positioning a shelf to be detected and framing the area of the shelf to be detected;

coarse detection is carried out on the framed area through global detection, and information of the required commodity is obtained;

classifying commodities according to the information of the required commodities according to rows, and dividing the image according to the classification result to obtain a commodity image to be detected;

performing character recognition on the commodity image to be detected to preliminarily determine commodity category;

image feature comparison and color comparison are carried out on the commodity image to be detected;

information fusion is carried out on the character recognition result and the image characteristic comparison and color comparison information, and a final commodity category is obtained;

judging the commodity category according to the template category information and the category information to be detected, and outputting a commodity identification result.

2. The method of claim 1, wherein the method comprises locating the shelf to be tested and framing the area of the shelf to be tested by a positioning of a loop code and a perspective transformation process.

3. The method of claim 1, wherein the step of obtaining information about the desired product by roughly detecting the framed area through global detection comprises:

performing global coarse detection on the picture by using a YOLOv5 network to obtain a coarse detection result;

and filtering and complementing the coarse detection result according to the position of the shape code, the average height of the commodity center and the line number information of the goods shelf to obtain the position information and the size information of the required commodity in the picture.

4. The method for identifying a shelf commodity according to claim 1, wherein the specific step of performing text recognition on the image of the commodity to be detected to preliminarily determine the commodity category comprises the steps of:

character extraction is carried out on the segmented commodity images to be detected through a pre-trained model;

and inquiring according to the extracted text information and the keyword library so as to determine the commodity category.

5. The method for identifying a shelf commodity according to claim 1, wherein the specific steps of comparing the image features of the commodity image to be tested are as follows:

extracting pattern features of the commodity image to be detected;

and carrying out repeated feature filtering processing on the extracted features to obtain comparison information.

6. The method for identifying a shelf commodity according to claim 1, wherein the specific step of performing color comparison on the image of the commodity to be tested is:

the original RGB color space is converted and compressed into the HSV color space to serve as color features by utilizing the characteristic that the HSV color space is insensitive to illumination, consistency of the color features and main colors is compared, and comparison information is output.

7. The method for identifying goods on shelves according to claim 1, wherein the specific steps of information fusion between the character identification result and the image feature comparison result and the color comparison result to obtain the final goods category are as follows:

and (3) taking a character recognition result as a main, and when the characteristics are similar, adding pattern characteristics and color comparison information to obtain a final result.

8. A system for identifying a shelf commodity, comprising:

the positioning module is configured to acquire images of the goods shelves, position the goods shelves to be tested and frame the areas of the goods shelves to be tested;

the detection module is configured to perform rough detection on the framed area through global detection to obtain information of the required commodity; classifying commodities according to the information of the required commodities according to rows, and dividing the image according to the classification result to obtain a commodity image to be detected;

the identification module is configured to perform character identification on the commodity image to be detected to preliminarily determine commodity category; image feature comparison and color comparison are carried out on the commodity image to be detected; and carrying out information fusion on the character recognition result and the image characteristic comparison result and the color comparison result to obtain a final commodity category, judging the commodity category according to the template category information and the category information to be detected, and outputting the commodity recognition result.

9. A computer readable storage medium, characterized in that a plurality of instructions are stored, which instructions are adapted to be loaded by a processor of a terminal device and to carry out the method of identifying a shelf commodity according to any one of claims 1 to 7.

10. A terminal device comprising a processor and a computer readable storage medium, the processor configured to implement instructions; a computer readable storage medium for storing a plurality of instructions adapted to be loaded by a processor and to perform the method of identifying a shelf commodity according to any one of claims 1 to 7.

Images