JP2019159433A - Image recognition system - Google Patents

Abstract

To provide an image recognition system for accurately identifying identity information of a target object, such as a product, in an image.SOLUTION: An image recognition system is provided, comprising: an image information processing unit configured to identify product identity information candidates of products in each of images of a display shelf having the products displayed thereon captured from multiple angles; and an identity information comparison processing unit configured to compare the identified identity information candidates for corresponding images to identify product identity information of the products in the images.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image recognition system for accurately identifying identification information of an object such as a product shown in image information.

  In the retail industry, it is known that the display status of products affects the sales of products. For this reason, as a product manufacturer and sales company, it is possible to connect to the development strategy and sales strategy of their own products by grasping what products of their own or other companies are displayed in the store.

  On the other hand, in order to achieve this, it is important to accurately identify the products displayed in stores. Therefore, it is conceivable that a store display shelf is photographed, and a product displayed on the basis of the image information is manually specified by a human. In this case, the product can be specified almost accurately. However, in order to keep track of the display status of products, it is necessary to monitor the display status at regular intervals. Humans specify products every time from image information of store shelves. Is burdensome and inefficient.

  Therefore, it is required to automatically identify the products displayed on the display information of the store shelves and grasp the display status of the products. For example, there is a method of using an image recognition technique for an image obtained by photographing a display shelf of a store based on a sample image for each product. As these conventional techniques, for example, there is a system that manages the display status of products using techniques as shown in Patent Documents 1 to 3 below.

  Also, outside the retail industry, it may be required to take an image of some object and specify the object shown in the captured image information.

JP-A-5-342230 JP-A-5-334409 International Publication 2012/029548

  The invention of Patent Document 1 is a system that supports a person who has no knowledge on which display shelf a product should be displayed. For this reason, it is impossible to grasp the products that are actually displayed. Patent Document 2 is a system that supports the input of product images in a shelf allocation support system that supports the display of products. However, the system of Patent Document 2 only supports the input of product images when using the shelf allocation support system, and even if this system is used, the products actually displayed cannot be grasped. .

  Patent Document 3 is an invention that, when there is a vacant space in a display shelf, specifies a product to be displayed in the vacant space, or notifies the user when a product to be displayed in the display shelf is misplaced. Even in this invention, the image matching process between the product and the product displayed on the display shelf is performed to identify the product displayed on the display shelf, but the reality is that the recognition accuracy is low.

  When using image recognition technology based on a sample image of a product for an image of a display shelf, the image features between the sample image and the partial image of the display part of the product in the image of the display shelf Recognize products using similarity. On the other hand, in the image or specimen image obtained by photographing the display shelf, if the surface of the product package reflects light, unnecessary objects such as lighting in the store, outside light, photographer, and photographing device are not positioned with the photographing device. Depending on the relationship, it may be reflected on the surface of the product. This is a factor affecting the accuracy of image recognition because the original image feature quantity cannot be obtained.

  Further, the same problem is not limited to the identification of products displayed on the display shelf, and the same problem also applies to the case where the object is identified from image information obtained by photographing a general object. And even if it uses each above-mentioned patent document, the subject by reflection of light etc. cannot be solved.

  In view of the above problems, the present inventor has invented an image recognition system capable of specifying identification information of an object such as a product with higher accuracy than in the past even when light or the like is reflected in an image.

  The invention according to claim 1 is an image information processing unit for identifying candidates for product identification information of products shown in the image information for each piece of image information obtained by photographing a display shelf displaying products from a plurality of angles. And an identification information comparison processing unit that identifies product identification information of a product shown in the image information by comparing the identified candidate product identification information in the corresponding image information. .

  With the configuration of the present invention, even if there is a reflection of light or the like in the image information obtained by photographing the display shelf, the product can be recognized more accurately than in the past.

  In the above-mentioned invention, the identification information comparison processing unit specifies the product identification information of the product shown in the image information based on the presence or the likelihood or the likelihood of the product identification information in the plurality of image information. It can be configured like a recognition system.

  When specifying product identification information from candidates for product identification information in each image information, it is preferable to specify as in the present invention. That is, if the product identification information candidate is specified or not specified, the specified product identification information candidate is specified as the product identification information. Further, when there is a candidate for product identification information for each, the product identification information with the highest likelihood can be specified. When there are a plurality of candidates according to the likelihood, the product identification information can be specified from the value based on the likelihood.

  In the above-described invention, the image information processing unit includes a captured image information normalization processing unit that performs normalization processing on captured image information obtained by capturing the display shelf and generates correct image information; Identification information for identifying candidate product identification information of the product in the area by comparing the image information of a part of the display shelf in the display image information with the sample information of the product And an image recognition system having a specific processing unit.

  Since the image information obtained by photographing the display shelf is distorted and the like, if the processing is performed after the positive value processing is performed, the recognition accuracy becomes high. In addition, since a plurality of products are often displayed on the display shelf, it is preferable to identify candidates for product identification information by executing, for example, sample information and comparison processing for the region for each product.

  In the above-described invention, the identification information specifying processing unit accepts input of product identification information and cannot input at least the input of the product identification information when the product identification information candidate of the product in the area cannot be specified. It is possible to configure the image recognition system such that the received product identification information is specified as the product identification information of the region without executing the processing in the identification information comparison processing unit.

  If the candidate for the product identification information cannot be specified, the input may be accepted. In this case, since the input product identification information is considered to have high accuracy, if the product identification information is identified (determined) as it is without executing the comparison process, the processing time can be reduced.

  In the above-described invention, the image information processing unit includes a face specifying processing unit that specifies a face area in the display shelf reflected in the in-place image information, and the identification information specifying processing unit includes: Comparing the image information of the face area in the in-place image information to be processed with the image information of the face area in a predetermined range from the face area in the other in-place image information corresponding to the in-place image information. Thus, it can be configured like an image recognition system that identifies candidates for product identification information of products in the face area in the in-place image information to be processed.

  Specifying product identification information by image matching processing with sample information has a large processing load. Therefore, if the faces that are the areas where the products are displayed are specified, comparing the faces can reduce the number of times of image matching processing and reduce the overall processing load. it can.

  The invention according to claim 6 is based on an image information processing unit that generates one image information based on a plurality of pieces of image information obtained by photographing the same display shelf from different angles, and on the basis of the generated image information, An image recognition system having an identification information specifying processing unit for specifying product identification information of a product shown in the image information.

  Even when configured as in the present invention, the same technical effect as that of the invention described in claim 1 can be obtained. That is, even if there is a reflection of light or the like in the image information obtained by photographing the display shelf, the product can be recognized more accurately than in the past.

  In the above-described invention, the image information processing unit identifies image information of a plurality of pieces of image information by comparing index values based on brightness for each corresponding reference unit region. By combining the image information of each area, the image information can be configured as an image recognition system that generates the one piece of image information.

  In the above invention, the image information processing unit divides each of the plurality of pieces of image information into meshes for each reference unit, calculates an index value based on lightness in the region of the mesh, and By comparing the index values in the corresponding mesh areas, the image information of the mesh areas is identified, and the image information of each identified mesh area is synthesized to generate the one image information , It can be configured like an image recognition system.

  When light is reflected, the brightness of the image information becomes high. Therefore, by configuring as in these inventions, it is possible to generate image information from which the reflection of light is removed, which leads to suppression of a decrease in recognition accuracy due to the reflection of light.

  The invention according to claim 9 changes the relative position between the product and the light source without changing the relative position between the product and the photographing device, and a part or all of the image information in which the display shelf displaying the product is shown. An image recognition system comprising: an identification information specifying processing unit for specifying product identification information of a product in image information showing the display shelf by comparing with sample information of the product based on a plurality of pieces of image information photographed It is.

  It is possible to improve accuracy by using sample information of products used in the comparison process that does not include light or the like. Therefore, it is possible to improve accuracy in comparison processing by using product sample information based on multiple pieces of image information taken by changing the relative position with the light source without changing the relative position between the product and the imaging device. be able to.

  The invention described in claim 10 accepts input of a plurality of image information taken by changing the relative position between the product and the light source without changing the relative position between the product and the photographing device. In the image recognition system, the sample information of the product is generated by cutting out a part of the image information and combining it into one piece of image information.

  It is possible to improve accuracy by using sample information of products used in the comparison process that does not include light or the like. Therefore, it is preferable to generate the sample information of the product as in the present invention.

  The invention according to claim 11 is a first processing unit for identifying candidates for identification information of a specific object reflected in the image information for each of image information obtained by photographing the object to be photographed from a plurality of angles, A second processing unit that identifies identification information of a specific object in the image information by comparing the identified identification information candidates in corresponding image information, Can be configured.

  In the first aspect of the present invention, the image recognition system identifies the identification information of the products displayed on the display shelf. However, the present invention is not limited to this, and image information obtained by photographing the object to be photographed from a plurality of angles. However, the present invention can be applied even when the identification information of the specific object shown in the image information is specified. That is, as in the first aspect of the invention, even if there is a reflection of light or the like in the captured image information, the identification information of the specific object can be recognized more accurately than in the past.

  The invention according to claim 12 is a first processing unit that generates one piece of image information based on a plurality of pieces of image information obtained by photographing a subject to be photographed from different angles, and based on the generated image information, And a second processing unit that identifies identification information of a specific object reflected in the image information.

  In the invention described in claim 6, although the image recognition system identifies the identification information of the commodity displayed on the display shelf, the present invention is not limited to this, and image information obtained by photographing the object to be photographed from a plurality of angles. However, the present invention can be applied even when the identification information of the specific object shown in the image information is specified. That is, as in the sixth aspect of the invention, even when there is a reflection of light or the like in the captured image information, the identification information of the specific object can be recognized more accurately than in the past.

  The invention described in claim 13 accepts input of a plurality of pieces of image information taken by changing the relative position between the object and the light source without changing the relative position between the object and the photographing device. In the image recognition system, a part of the image information is cut out and synthesized into one piece of image information, thereby generating sample information of the object.

  The invention according to claim 9 is an image recognition system for identifying identification information of commodities displayed on a display shelf, but is not limited thereto, and image information obtained by photographing an object to be photographed from a plurality of angles. However, the present invention can be applied even when the identification information of the specific object shown in the image information is specified. In other words, since the sample information of the product used in the comparison process uses information that does not include light or the like, which improves accuracy, a plurality of pieces of image information captured without changing the relative position between the product and the imaging device. By using the sample information of the object based on the above, it is possible to improve the accuracy in the comparison process.

  In the invention described in claim 14, the relative position between the object and the light source is changed without changing the relative position between the object and the photographing device, and a plurality of pieces of image information of the object are photographed. In this method, the sample information of the target is generated by cutting out a region in the image information where there is no reflection and combining it into one piece of image information.

  When performing image recognition processing, the use of sample information without reflection is a factor in improving recognition accuracy. Therefore, by generating sample information by the method of the present invention, sample information without reflection can be generated.

  The invention described in claim 1 can be realized by reading the program of the present invention into a computer and executing it. That is, for each piece of image information obtained by photographing a display shelf displaying a product from a plurality of angles, the image information processing unit for identifying candidates for product identification information of the product shown in the image information, corresponding image information In the image recognition program that functions as an identification information comparison processing unit that identifies the product identification information of the product shown in the image information by comparing the identified product identification information candidates. it can.

  The invention described in claim 6 can be realized by reading the program of the present invention into a computer and executing it. That is, the computer displays an image information processing unit that generates one piece of image information based on a plurality of pieces of image information obtained by photographing the same display shelf from different angles, and the image information is displayed based on the generated image information. It can be configured like an image recognition program that functions as an identification information specifying processing unit that specifies the product identification information of a product being sold.

  The invention according to claim 9 can be realized by reading the program of the present invention into a computer and executing it. In other words, the computer was photographed by changing the relative position between the product and the light source without changing the relative position between the product and the imaging device, and a part or all of the image information showing the display shelf displaying the product. Like an image recognition program that functions as an identification information identification processing unit that identifies product identification information of products in image information in which the display shelf is reflected by comparing with sample information of products based on a plurality of image information Can be configured.

  The invention described in claim 10 can be realized by reading the program of the present invention into a computer and executing it. That is, in the computer, the step of receiving a plurality of pieces of image information captured by changing the relative position between the product and the light source without changing the relative position between the product and the imaging device; By cutting out a part and synthesizing it into one piece of image information, it can be configured as an image recognition program that executes the step of generating sample information of the product.

  The invention described in claim 11 can be realized by reading the program of the present invention into a computer and executing it. That is, for each piece of image information obtained by photographing a subject to be photographed from a plurality of angles, a first processing unit for identifying candidates for identification information of a particular subject in the image information, By comparing the identified identification information candidates, it can be configured as an image recognition program that functions as a second processing unit that identifies identification information of a specific object shown in the image information. .

  The invention according to claim 12 can be realized by reading the program of the present invention into a computer and executing it. In other words, the computer captures a first processing unit that generates one piece of image information based on a plurality of pieces of image information obtained by photographing a subject to be photographed from different angles, and shows the image information based on the generated image information. The image recognition program can be configured to function as a second processing unit that identifies identification information of a specific target object.

  The invention described in claim 13 can be realized by reading the program of the present invention into a computer and executing it. In other words, in the computer, the step of receiving input of a plurality of pieces of image information obtained by changing the relative position of the object and the light source without changing the relative position of the object and the imaging device, and the image having received the input A part of the information is cut out and synthesized into one piece of image information, so that it can be configured as an image recognition program that executes the step of generating the sample information of the object.

  By using the image recognition system of the present invention, even when light or the like is reflected in image information, it is possible to specify an object such as a product with higher accuracy than before.

It is a block diagram showing typically an example of composition of an image recognition system of the present invention. It is a block diagram which shows typically an example of a structure of the image information processing part in the image recognition system of this invention. It is a block diagram which shows typically an example of the hardware constitutions of the computer used with the image recognition system of this invention. It is a flowchart which shows an example of the whole processing process in the image recognition system of this invention. It is a flowchart which shows an example of the processing process in the image information processing part of this invention. It is a flowchart which shows an example of the process in the identification information comparison process part of this invention. It is a figure which shows an example of a sample information storage part typically. It is a figure which shows an example of the picked-up image information image | photographed from the different angle on the same opportunity about the display shelf of a store. It is a figure which shows another example of the picked-up image information image | photographed from the different angle about the display shelf of the store at the same opportunity. FIG. 9 is a diagram illustrating an example of in-place image information on which trapezoidal correction processing has been performed for the captured image information in FIG. 8. It is a figure which shows an example of the in-place image information in which the trapezoid correction process was performed about the picked-up image information of FIG. It is a figure which shows the state by which the shelf area was specified in the picked-up image information of FIG. It is a figure which shows the state by which the shelf area was specified in the picked-up image information of FIG. It is an image figure of an example of the typical process in the image recognition system of this invention. It is a figure which shows an example of the state which set the feature-value collection area | region with respect to the N-1th picked-up image information. It is a figure which shows an example of the state which set the feature-value collection area | region with respect to the captured image information of the Nth time. It is a figure which shows the relationship of the pair of the N-th feature-value collection area | region and the N-th feature-value collection area | region. It is a figure which shows the state which projects the shelf position C in the N-1th picked-up image information as the shelf position D in the N-th picked-up image information by the function F. It is a figure which shows an example of picked-up image information. It is a figure which shows the state which set the feature-value collection area | region to the N-1th picked-up image information. It is a figure which shows the state which set the feature-value collection area | region to the captured image information of the Nth time. It is a figure which shows an example of the state which specified shelf position D1-D4 in the N-th imaging | photography in the square method. It is a figure which shows typically the Nth process in an identification information specific process part. FIG. 10 is a block diagram schematically illustrating an example of a configuration of an image recognition system according to a fifth embodiment. FIG. 10 is a block diagram schematically illustrating an example of a configuration of an image information processing unit according to a fifth embodiment. 10 is a flowchart illustrating an example of an overall processing process of an image recognition system according to a fifth embodiment. 14 is a flowchart illustrating an example of a processing process of processing for generating processed in-place image information according to the fifth exemplary embodiment. It is a figure which shows an example of the state divided | segmented with the mesh for every predetermined | prescribed reference unit in in-place image information. It is a figure which shows typically the process which produces | generates processed in-place image information by arranging the image information of the area | region of the mesh specified by satisfying predetermined conditions side by side. It is a figure which shows an example at the time of seeing the state which image | photographs goods from upper direction. It is a figure which shows typically the process which produces | generates suitable sample information by cutting out the part which is not reflected in several image information image | photographed, and synthesize | combining each other. It is a figure which shows typically an example of the external appearance of the goods used as a sample. It is a figure which shows an example of the imaging | photography method in case the external shape of goods is a 1st case. It is a figure which shows an example of the imaging | photography method in case the external shape of goods is a 2nd case. It is a figure which shows an example of the imaging | photography method in case the external shape of goods is a 3rd case.

  An example of the system configuration of the image recognition system 1 of the present invention is shown in FIG. The image recognition system 1 uses a management terminal 2 and a captured image information input terminal 3.

  The management terminal 2 is a computer used by an organization such as a company that operates the image recognition system 1. The photographed image information input terminal 3 is a terminal for inputting image information obtained by photographing a display shelf in a store.

  The management terminal 2 and the captured image information input terminal 3 in the image recognition system 1 are realized using a computer. FIG. 3 schematically shows an example of the hardware configuration of the computer. The computer is capable of inputting information, such as an arithmetic device 70 such as a CPU for executing arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk for storing information, a display device 72 such as a display for displaying information, and the like. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 that transmits and receives processing results of the arithmetic device 70 and information stored in the storage device 71 via a network such as the Internet or a LAN.

  When the computer includes a touch panel display, the display device 72 and the input device 73 may be configured integrally. The touch panel display is often used in, for example, a portable communication terminal such as a tablet computer or a smartphone, but is not limited thereto.

  The touch panel display is an apparatus in which the functions of the display device 72 and the input device 73 are integrated in that input can be performed directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

  The photographed image information input terminal 3 may include a photographing device such as a camera in addition to the above devices. As the photographed image information input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, or a tablet computer can be used.

  Each means in the present invention is only logically distinguished in function, and may be physically or virtually the same area. The processing order of each means of the present invention can be changed as appropriate. Further, a part of the processing may be omitted. For example, it is possible to omit the normalization processing described later. In this case, it is possible to execute processing for image information that has not been subjected to normalization processing.

  The image recognition system 1 includes a captured image information storage unit 20, a sample information storage unit 21, an identification information candidate storage unit 22, an identification information storage unit 23, an image information processing unit 24, and an identification information comparison processing unit 25.

  The photographed image information storage unit 20 stores the photographed image information received from the photographed image information input terminal 3, the photographing date and time, the store identification information, the image information identification information, and the like in association with each other. The captured image information only needs to be image information to be subjected to trapezoidal correction processing, and includes image information that is synthesized as a single image information when a single display shelf is captured in multiple sheets. . Also, the image information after the distortion correction processing is executed is included in the captured image information.

  The sample information storage unit 21 stores sample information for identifying which product is the product of each face on the shelf of the display shelf shown in the in-place image information. Specimen information is image information obtained by photographing products that may be displayed on display shelves from a plurality of angles such as up, down, left, and right. FIG. 7 shows an example of sample information stored in the sample information storage unit 21. In FIG. 7, the sample information shows a case where canned beer is taken from various angles, but is not limited to canned beer. The sample information storage unit 21 stores sample information and product identification information in association with each other.

  The specimen information storage unit 21 stores information necessary for calculation of similarity extracted from the specimen information together with the specimen information or in place of the specimen information, for example, information on a pair of image feature quantity and its position. You may remember. It is assumed that the sample information includes information necessary for calculating similarity. In this case, the identification information specifying processing unit 246 does not have to calculate the image feature amount every time the sample information is matched with the image information of the face area and the sample information, thereby reducing the calculation time. can do.

  The identification information candidate storage unit 22 stores, for each photographed image information, merchandise identification information candidates of merchandise displayed on each face of the shelf of the display shelf. For example, in association with candidates for product identification information, it is associated with shooting date / time information, store information, image information identification information of captured image information, image identification information of in-place image information, and face identification information for identifying a face. The identification information candidate storage unit 22 stores the identification information. The candidate product identification information stored here is stored for each photographed image information. For this reason, since there are a plurality of pieces of photographed image information for the same display shelf, candidate product identification information specified for each piece of photographed image information is stored in the corresponding face of the same display shelf shelf. .

  The identification information storage unit 23 stores product identification information in which the products displayed on each face of the shelf of the display shelf are specified (confirmed). For example, identification information associated with product identification information is associated with shooting date / time information, store information, image information identification information of captured image information, image identification information of in-place image information, and face identification information for identifying a face. Store in the storage unit 23.

  The image information processing unit 24 is a process of identifying candidate product identification information of products displayed on the shelf in the display shelf shown in the captured image information in the captured image information received from the captured image information input terminal 3 Execute.

  The image information processing unit 24 identifies the captured image information input reception processing unit 241, the captured image information normalization processing unit 242, the shelf position specifying processing unit 243, the shelf area extraction processing unit 244, and the face specifying processing unit 245. An information identification processing unit 246 and a shelf image matching processing unit 247 are included.

  The photographed image information input acceptance processing unit 241 accepts input of image information (photographed image information) of a store display shelf photographed by the photographed image information input terminal 3 and stores it in the photographed image information storage unit 20. From the photographed image information input terminal 3, it is preferable to accept input together with photographed image information, store identification information such as photographing date and time, store name, and image information identification information for identifying image information. 8 and 9 show examples of photographed image information. In FIG. 8 and FIG. 9, there is photographed image information in which a display shelf has three shelves and products are displayed there.

  The photographed image information normalization processing unit 242 generates a regular image information obtained by performing a trapezoidal correction process on the photographed image information stored in the photographed image information storage unit 20 and performing the normalization. The trapezoidal correction process is a correction process performed so that the shelf of the display shelf shown in the photographed image information is horizontal and the product tag for the product displayed there is vertical. In-place positioning means that the image information is deformed so that the optical axis of the lens of the photographing apparatus is sufficiently the same as when photographing from a far distance along the perpendicular direction of the plane to be photographed. There is a correction process.

  The trapezoid correction processing executed by the captured image information normalization processing unit 242 receives input of designation of four vertices in the captured image information, and executes the trapezoid correction processing using each vertex. The four vertices that accept the designation may be the four vertices of the shelf of the display shelf or the four vertices of the shelf position of the display shelf. Further, it may be the four vertices of a group of two or three shelves. Any four points can be designated as the four vertices. FIG. 10 and FIG. 11 show an example of photographed image information (orientated image information) subjected to the in-place processing.

  The shelf position identification processing unit 243 identifies a shelf area (shelf area) from the captured image information or the in-place image information. In other words, the display shelves shown in the photographed image information and the in-place image information include a shelf area in which products are displayed and a product tag placement area in which product tags for the products to be displayed are attached. . Therefore, the shelf area is specified from the in-place image information. As the specification of the shelf area, the operator of the management terminal 2 may manually specify the shelf area, and the shelf position specifying processing unit 243 may accept it, or the shelf that has been manually input for the first time. On the basis of the area information, the shelf area may be specified automatically after the second time. 12 and 13 show a state where the shelf area is specified.

  The shelf area extraction processing unit 244 extracts the image information of the shelf area specified by the shelf position specification processing unit 243 as the shelf area image information. The shelf area cutout processing unit 244 may actually cut out as image information, or actually cuts out by specifying coordinates constituting the shelf area without cutting out as image information. It's okay. When there are a plurality of shelves on the display shelf, each is cut out as shelf area image information. In addition, the coordinates indicating the shelf area are the coordinates of the vertices necessary for specifying the area. For example, the coordinates of the four points in the in-place image information, the coordinates of the upper right and lower left, the upper left and lower right, etc. Good. Further, the relative coordinates are based on a predetermined location in the image information (for example, the top left vertex of the display shelf) such as a display shelf in the in-place image information.

  The face identification processing unit 245 identifies the face on which the product is placed (the region on which the product is placed) for each shelf in the shelf area in the in-place image information. The face identification processing unit 245 is divided into an initial face identification process and a second and subsequent face identification process.

  In the face identification processing unit 245 for the first time, products are placed within a range (preferably a rectangular region) constituted by the coordinates of the shelf specified by the shelf position specification processing unit 243. Specify the area (face). Specifically, the narrow and narrow shadows that occur between products are specified, the image repeat pattern is specified, the top edge of the package is specified, the product width is the same, and so on. The face area is specified by specifying the position. As the face identification processing, any method can be adopted depending on the product category and the product form, and the method is not limited to the above. Moreover, you may receive the correction input by a person in charge with respect to the face identified automatically. Furthermore, an input of the face position from the person in charge may be accepted. The coordinates of the area constituting the identified face are the shooting area information, the store information, the image information identification information of the captured image information, the image identification information of the in-place image information, the face, and the coordinates of the face area in the in-place image information. The face identification information for identification is associated and managed. The coordinates indicating the face area are the coordinates of the vertices necessary for specifying the rectangular area, and may be, for example, the coordinates of the four points in the in-place image information, the coordinates of the upper right and lower left, the upper left and lower right, etc. . Further, the relative coordinates are based on a predetermined location in the image information (for example, the top left vertex of the display shelf) such as a display shelf in the in-place image information.

  In the face identification processing for the second and subsequent times in the face identification processing unit 245, the coordinates of the face area identified by the previous (N-1th) in-place image information for the same shelf of the same display shelf are the current ( The coordinates of the face area specified by the (Nth) position image information may be used.

  The identification information specification processing unit 246 specifies candidates for the product identification information of the products displayed on the face for each shelf of the display shelf. The product identification information includes, in addition to the product name, a JAN code assigned to the product, but is not limited thereto. Any information can be used as long as it can identify the product.

  The identification information specifying processing unit 246 executes the following processing. That is, for each face, by matching the image information of the face with the sample information of the product stored in the sample information storage unit 21, the product identification information candidates of the product displayed on the face are specified. Specifically, first, the similarity between the image information of the region formed by the coordinates of the face to be processed and the sample information stored in the sample information storage unit 21 is determined, and the sample information having the highest similarity is determined. Is identified as a candidate for product identification information of the product displayed on the face composed of the above coordinates.

  Here, in order to determine the similarity between the face image information and the sample information, the following processing is performed. First, in the process before the identification process for identifying the product identification information in the identification information identification processing unit 246, the image information of the face area in the shelf of the in-place image information and the sample information have the same direction (the rollover or And the size of each image information is approximately the same (if the size of the image information is different within a predetermined range, before each similarity determination, The image information is adjusted so that the size of the image information is within a predetermined range).

  The identification information specifying processing unit 246 determines the similarity between the face image information and the sample information, and the feature point based on the image feature amount (for example, local feature amount) of the face image information and the image of the sample information Feature points based on feature quantities (for example, local feature quantities) are extracted. Then, a pair having the highest similarity between the feature point of the face image information and the feature point of the sample information is detected, and the difference between the coordinates of the corresponding points is obtained. Then, the average value of the differences is obtained. The average value of the difference indicates the total average movement amount between the face image information and the sample information. Then, the coordinate differences of all feature point pairs are compared with the average coordinate difference, and pairs with a large degree of deviation are excluded. Then, the similarity is ranked by the number of remaining corresponding points.

  The similarity between the face image information and the sample information can be calculated by the above method. Further, in order to improve the accuracy, an EMD (Earth Moves Distance) between color histograms may be obtained and used as a measure of similarity. As a result, it is possible to compare comparatively strong similarities to environmental changes such as brightness information of photographed image information, and to specify with high accuracy.

  As another determination of similarity, EMD between image information signatures (a set of image feature amounts and weights) in each face area may be obtained and used as a measure of similarity. As the image feature quantity of the signature, for example, the frequency distribution in the HSV color space of the image information of the face area is obtained, the hue and the saturation are grouped, and the number of the features and the image feature quantity by the area in the HSV color space are determined. can do. The reason for grouping hue and saturation is to reduce the dependence on lightness so that it is not greatly affected by shooting conditions.

  Further, in order to speed up the processing, similarities such as L2 distance between image feature quantities such as color correlograms and color histograms of image information in an appropriate color space can be used instead of the signature and EMD.

  The determination of similarity is not limited to the above. The identified product identification information candidates are identification information candidates in association with shooting date / time information, store information, image information identification information of captured image information, image identification information of in-place image information, and face identification information for identifying a face. Store in the storage unit 22. Note that the candidate product identification information specified here is not the confirmed product identification information because it is a candidate for the product identification information of the product shown on the face in the in-place image information based on certain photographed image information.

  The candidate product identification information identified as described above corresponds to shooting date / time information, store information, image information identification information of the photographed image information, image identification information of the in-place image information, and face identification information for identifying the face. Then, it is stored in the identification information candidate storage unit 22.

  The shelf image matching processing unit 247 includes image information on the shelf area in the previous (N-1) th in-place image information and image information on the shelf area in the current (Nth) in-place image information. If the similarity is high, the candidates for the product identification information of each face on the shelf are determined to be the same. As described above, this similarity determination processing is performed in the image feature amount of the image information of the shelf area in the previous (N-1) th in-place image information and the current (Nth) in-place image information. Similarity determination based on image information of a shelf area may be used, or EMD between color histograms may be used. Moreover, it is not limited to them. Then, instead of the identification processing for each face unit in the identification information identification processing unit 246, the identification information identification processing unit 246 sends the N-th in-place image information candidates for the product identification information of each face on the shelf to N− The product is stored in the identification information candidate storage unit 22 as being the same as the product identification information candidate for each face on the same first shelf. As a result, it is possible to omit processing for a shelf that hardly changes, such as a shelf that does not move much or a shelf that is managed in a very short cycle. Note that the processing by the shelf image matching processing unit 247 may not be provided.

  The processing in the shelf-level image matching processing unit 247 is used for the N-th comparison and the (N-1) -th comparison, and for the N-th comparison of the in-place image information obtained by photographing the same display shelf at different angles. It may be used.

  Based on the candidate product identification information specified by the identification information specification processing unit 246, the identification information comparison processing unit 25 uses the captured image information (orientated image information) obtained by photographing the same display shelf at the same opportunity at different angles. , The product identification information candidates of the corresponding faces are compared, and the product identification information of the product shown on the face is specified (confirmed). An image diagram of the processing of the identification information comparison processing unit 25 is shown in FIG.

  The identification information comparison processing unit 25 stores the product identification information for each face specified by the image information processing unit 24 in the in-place image information based on different photographed image information photographed on the same display shelf at the same opportunity, for example, on the same day. Each candidate is specified from the identification information candidate storage unit 22. For example, candidate product identification information corresponding to the same face identification information is specified. Then, each product identification information candidate is compared with a predetermined condition, and the product identification information corresponding to the face is specified (determined). For example, as shown in FIG. 14, it is assumed that there are two pieces of photographed image information (photographed image information 1, photographed image information 2) obtained by photographing the same display shelf at the same opportunity and having different angles. At this time, the product identification information 1 is associated as a candidate for product identification information with the face A in the normal image information 1 in which the captured image information 1 is aligned, and the captured image information 2 is aligned. It is assumed that the product identification information candidate does not exist on face A in image information 2 (the product identification information on face A could not be specified based on the in-place image information 2). In this case, the product identification information 1 based on the in-place image information 1 is specified as the product identification information of the face A. As another example, product identification information 1 is associated with 40% likelihood as a candidate for product identification information to face A in the in-place image information 1 obtained by arranging the captured image information 1. When the product identification information 2 is associated with the face A in the normal image information 2 obtained by arranging the information 2 as a candidate for product identification information with a likelihood of 90%, the product identification information with the highest likelihood is used. 2 is specified as product identification information of face A.

  In this way, the identification information comparison processing unit 25 determines the identified product identification information candidates, the presence / absence of the product identification information candidates, and the product identification information in the image information at different angles obtained by photographing the same display shelf at the same opportunity. Based on the likelihood of the candidate, the product identification information corresponding to the face is specified. When there are a plurality of product identification information candidates corresponding to the likelihood in one face in one image information, the likelihood may be calculated respectively. For example, product identification information 1 is associated with a likelihood of 65% and product identification information 2 with a likelihood of 30% as a candidate for the product identification information to face A in the regular image information 1 obtained by arranging the photographed image information 1 in a regular manner. The face A in the normal image information 2 obtained by arranging the photographed image information 2 is associated with the product identification information 2 as a candidate for product identification information with a likelihood of 70% and the product identification information 1 with a likelihood of 25%. In the case of the same product identification information, the values based on the likelihood are calculated by adding, multiplying, etc., and specifying one of the product identification information based on the value. Also good. For example, in the above-described case, the product identification information 1 may be specified as 90 and the product identification information 2 may be set as 100 and may be specified as the product identification information 2. In this way, the identification information comparison processing unit 25 compares the product identification information candidates identified from a plurality of pieces of image information at different angles obtained by photographing the display shelves that are the same object to be photographed at the same opportunity under a predetermined condition. , Product identification information corresponding to the face is specified. Although the accuracy is improved by executing the processing in units of faces, the present invention is not limited to this, and it is only necessary to compare the product identification information candidates.

  Note that the image recognition system 1 in the present specification can be applied to a process of identifying the product identification information of the product shown in the image information obtained by photographing the product displayed on the display shelf, but is not limited thereto. The present invention can also be applied to the case where identification information of an object is specified from image information in which some object is reflected.

  A processing process using the image recognition system 1 of the present invention will be described with reference to the block diagrams of FIGS. 1 and 2 and the flowcharts of FIGS. In the following description, a case where two pieces of photographed image information photographed from two directions are used for the same display shelf will be described. However, even when there are three or more pieces, the same processing can be executed.

  The person in charge of photographing the display shelf of the store performs photographing from a plurality of angles so that the reflection of light or the like does not overlap on the same display shelf at the same opportunity. And about the display shelf of a store, the picked-up image information (FIG. 8, FIG. 9) image | photographed from the different angle at the same opportunity is input from the picked-up image information input terminal 3, and is image | photographed in the image information processing part 24 of the management terminal 2. The image information input reception processing unit 241 receives the input (S100, S110). Also, input of image information identification information such as shooting date / time, store identification information, and shooting image information is accepted. Then, the photographed image information input acceptance processing unit 241 stores the photographed image information, the photographing date and time, the store identification information, and the image information identification information of the photographed image information, which are accepted, in the photographed image information storage unit 20 in association with each other.

  When the management terminal 2 accepts a predetermined operation input, the photographed image information normalization processing unit 242 extracts the photographed image information stored in the photographed image information storage unit 20 and performs a shelf that is a vertex for performing trapezoid correction processing. The input of the four points of the position (the position of the display shelf) is received, and the keystone correction process is executed (S120). FIGS. 10 and 11 show examples of photographed image information (rectified image information) that has been subjected to the trapezoidal correction process in this way.

  Then, the shelf position identification processing unit 243 identifies the shelf area by receiving a predetermined operation input in the management terminal 2 for the in-place image information (S130). That is, the input of the shelf area in the in-place image information is accepted. 12 and 13 are diagrams showing a state where the shelf region is specified.

  When the shelf area is specified as described above, the shelf area extraction processing unit 244 extracts the image information of the shelf area from the in-place image information based on the shelf area received in S130. (S140). Then, a process for specifying a face is executed for each shelf in the shelf area image information (S150). Specifically, for the shelves in the shelf area, identify the narrow and narrow shadow that occurs between the products within the rectangular area consisting of the coordinates of the four points. The face is specified by specifying a step on the upper side of the package, specifying a separation position based on constraints such as the same product width, and the like. Face identification information for identifying the face is attached to the specified face. The face identification information may be the same face identification information for image information obtained by photographing the same display shelf in order to identify the corresponding face even if the image information is different. The correspondence of identification information is stored.

  The coordinates of each identified face are stored in association with the shooting date and time, the store identification information, the image information identification information of the photographed image information, the image information identification information of the in-place image information, and the face identification information for identifying the face. Let The coordinates of the face may be two points that can specify a rectangular area without storing four points.

  As described above, when each face of each shelf in the shelf area image information of the in-place image information is specified, the identification information specifying processing unit 246 performs the sample information and matching processing stored in the sample information storage unit 21 for each face. To identify the product identification information corresponding to the sample information having the highest similarity. And it specifies as a candidate of the merchandise identification information of the merchandise displayed on the face (S160). That is, each image feature amount is calculated from the image information in the rectangular area of the face of a certain shelf (the face identification information of the face of this area is X) and each sample information stored in the sample information storage unit 21. Then, similarity is determined by obtaining a pair of feature points. Then, the sample information with the highest similarity is specified, and if the similarity at that time is equal to or greater than a predetermined threshold, the product identification information corresponding to the sample information is specified based on the sample information storage unit 21. Then, the specified product identification information is set as a candidate for product identification information of the product displayed on the face of the face identification information X. Then, the identification information identification processing unit 246 associates the identified candidate product identification information with the shooting date and time, the store identification information, the image information identification information of the photographed image information, the image information identification information of the in-place image information, and the face identification information. And stored in the identification information candidate storage unit 22 (S170).

  In addition, the product identification information of all faces cannot be specified. Therefore, for a face that has not been identified (such as a face having the highest similarity less than a predetermined threshold), the product identification information is input, and the received product identification information includes the shooting date / time, the store identification information, and the captured image. The image information identification information of the information, the image information identification information of the in-place image information, and the face identification information may be stored in the identification information candidate storage unit 22 as product identification information candidates. Similarly, an input may be accepted for the correction process for the identified candidate product identification information. Since such a face is originally highly reliable, it is determined as product identification information without executing the processing of the identification information comparison processing unit 25 described later, and the identification information candidate storage unit 22, the identification information storage unit 23 may be stored. At this time, a flag indicating that manual input has been accepted is attached to the identification information candidate storage unit 22, and if the flag is present, the processing of the identification information comparison processing unit 25 can be skipped.

  By performing the processing as described above, candidates for product identification information of products displayed on the shelf of the display shelf shown in the captured image information can be specified.

  When the product identification information candidate can be specified for the face in the in-place image information based on each captured image information, the identification information comparison processing unit 25 selects the product identification information candidate between the corresponding faces as the identification information candidate storage unit 22. Are identified as product identification information in the face by comparing them based on predetermined conditions (S200).

  Specifically, first, the identification information comparison processing unit 25 specifies corresponding faces in a plurality of pieces of in-place image information at different angles obtained by photographing the same display shelf at the same opportunity (S210). In identifying the corresponding face, there are methods such as specifying based on the same face identification information and specifying a face associated as a face.

  And the candidate of the product identification information about the specified face is extracted from the identification information candidate storage unit 22, and the candidate of the product identification information of the corresponding faces is compared if a predetermined condition is satisfied (S220). (S230). Then, the identified product identification information is stored in the identification information storage unit 23 (S240).

  By performing the processing as described above, it is possible to specify the product identification information of the products displayed on the shelf of the display shelf shown in the captured image information. Moreover, since it identifies based on the picked-up image information image | photographed from several angles, it becomes easy to eliminate the influence of reflections, such as light, and can improve the specific precision of goods.

  In the second or second and subsequent processes, the process in the shelf image matching processing unit 247 can be executed as appropriate.

  In the first embodiment described above, the trapezoidal correction process is executed by designating four points. However, it is burdensome to designate and input a vertex as a reference every time. Therefore, a configuration may be adopted in which a vertex serving as a reference for trapezoid correction processing is automatically specified. Processing in this case will be described.

  In this case, the captured image information normalization processing unit 242 is divided into an initial trapezoid correction process and a second and subsequent trapezoid correction processes. In addition to the first time, the first time includes a case where it is manually performed at an arbitrary timing in order to correct a deviation when automatically specifying the vertex. The second and subsequent times are other than the first time. In the present invention, the same display shelf is photographed from a plurality of angles at the same opportunity. Therefore, the trapezoid correction process is executed by specifying the vertex for the first sheet (N-1), and the vertex that is the standard for the trapezoid correction process is automatically specified for the second and subsequent sheets (Nth). You may make it do.

  The first trapezoid correction process in the captured image information normalization processing unit 242 accepts input of designation of the four vertices of the rectangular area of the display shelf, as in the first embodiment. The four vertices of the rectangular area of the display shelf may be the four vertices of the shelf position of the display shelf, the four vertices of the shelf stage, or the four vertices of the area to which the product tag is attached. Further, it may be the four vertices of a group of two or three shelves. The four vertices that have received the designation are stored in association with the image information identification information of the shooting date / time information, the store information, and the shot image information. Then, the captured image information normalization processing unit 242 performs trapezoidal correction processing on the captured image information based on the coordinates of the four vertices that have received the designation, and obtains in-place image information.

  The captured image information normalization processing unit 242 executes the second and subsequent trapezoidal correction processes as follows.

  First, the captured image information normalization processing unit 242 uses the vertex coordinates of the (N−1) th captured image information obtained by capturing the same (substantially the same) area corresponding to the Nth captured image information in the previous processing. Specify from the stored information. The vertex coordinates of the (N-1) th imaged image information obtained by imaging the same (substantially the same) area corresponding to the Nth imaged image information are stored in store identification information, image identification information, imaged date / time information, etc. corresponding to the imaged image information. Identify based on. Then, with respect to the (N-1) th captured image information, a rectangular region having a predetermined size including the identified vertex coordinates of the four vertices, for example, a square having a width of about 1/5 of the shelf width is selected as the feature amount collection region 2420. Set as. FIG. 15 shows an example of a state in which the feature amount collection area 2420 is set for the (N-1) th captured image information. The feature amount collection area 2420 may be a rectangular area including vertex coordinates. On the other hand, if the backgrounds of the display shelves match each other, the background will be greatly displaced by only a slight shift in the shooting position. Therefore, it is preferable to set the feature amount collection region 2420 at a position including as much inside the display shelf as possible. That is, it is preferable that the vertex coordinates are located in the outward direction of the display shelf in the feature amount collection region 2420 from the center point of the feature amount collection region 2420.

  For example, the four vertex coordinates are located at the upper left, upper right, lower left, and lower right. Then, when the rectangular area of the feature quantity collection area 2420 is divided into a total of 4 areas each divided into two at the vertical and horizontal centers, in the feature quantity collection area 2420 including the upper left vertex coordinates, the vertex coordinates are the upper left areas of the rectangular areas. The feature amount collection region 2420 is set to be located at Similarly, in the feature quantity collection area 2420 including the upper right vertex coordinates, the feature quantity collection area 2420 is set so that the vertex coordinates are located in the upper right area of the rectangular area, and the feature quantity collection including the lower left vertex coordinates is performed. In the area 2420, the feature quantity collection area 2420 is set so that the vertex coordinates are located in the lower left area of the rectangular area. In the feature quantity collection area 2420 including the lower right vertex coordinates, the vertex coordinates of the rectangular area are The feature amount collection region 2420 is set so as to be located in the lower right region. As a result, the vertex coordinates are located in the feature amount collection region 2420 in the outer direction of the display shelf with respect to the center point of the feature amount collection region 2420.

  Next, the captured image information normalization processing unit 242 includes a feature amount collection area 2420 set in the N−1th captured image information in the Nth captured image information, and the N−1th captured image information. The feature amount collection region 2421 having a size larger than the feature amount collection region 2420 is set. The feature amount collection area 2421 set in the Nth captured image information does not exceed half the short side. Further, if the image information is outside the captured image information, the range is trimmed. FIG. 16 shows an example of a state in which the feature amount collection area 2421 is set for the Nth captured image information.

  Then, the captured image information normalization processing unit 242 collects local feature amounts in each feature amount collection region 2420 set for the (N-1) th captured image information, and extracts feature points and their coordinates based on the local feature amounts. Memorize the set. In addition, in each feature amount collection area 2421 set for the Nth captured image information, a local feature amount is collected, and a feature point based on the local feature amount and a set of coordinates thereof are stored.

  The captured image information normalization processing unit 242 corresponds to the local feature amount of the feature point in the feature amount collection region 2420 of the N-1th captured image information and the feature amount collection region 2420 of the N-1th captured image information. The local feature amount of the feature point in the feature amount collection region 2421 of the Nth captured image information at the position to be compared is compared. Then, the feature point of each local feature amount of the N-th captured image information that is closest to each local feature amount of each feature point of the N-1th captured image information is specified. Then, the feature points of the nearest local feature values are paired, and the feature point coordinates by the paired local feature values are associated. At this time, pairs whose local feature quantities are close (similarity) less than a predetermined threshold are excluded. As a result, it is possible to identify a pair of feature points of the local feature amount in the (N-1) th feature amount collection region 2420 and a feature point of the closest local feature amount in the Nth feature amount collection region 2421. FIG. 17 shows a pair relationship between the feature points of the local feature amount of the N-1th feature amount collection region 2420 and the feature points of the local feature amount of the Nth feature amount collection region 2421. In FIG. 17, a point group of feature points based on local feature amounts in the N-1th feature amount collection region 2420 is A, and a point group of feature points based on local feature amounts in the Nth feature amount collection region 2421 is B, N−. The vertex used for the first trapezoid correction process is indicated by C.

  The coordinates of the point group A of the feature points based on the local feature amounts in the N-1th feature amount collection region 2420 and the point group B of the feature points based on the local feature amounts in the Nth feature amount collection region 2421 corresponding to the point group A And a function F (perspective transformation / homography) for projecting the point group A to the point group B is obtained. The function F includes, but is not limited to, a method of repeating sampling estimation and using Ransac of OpenCV which is a kind of robust estimation. It should be noted that a pair having a large deviation from the relationship line in the projection relationship is excluded from the processing target.

  After obtaining the function F in the captured image information normalization processing unit 242, the captured image information normalization processing unit 242 determines the coordinates of the vertex C used in the N−1th trapezoidal correction process based on the function F. The image information is projected onto the Nth captured image information and specified as the coordinates of the vertex D for the Nth trapezoidal correction process. This is schematically shown in FIG.

  By performing the above processing on the feature amount collection areas 2420 and 2421, the four vertices of the shelf position for the trapezoidal correction process in the Nth captured image information are specified. Then, the captured image information normalization processing unit 242 executes trapezoidal correction processing for the Nth captured image information based on the identified four vertices to generate the correct image information and store it. At this time, the captured image information corrective processing unit 242 associates with the correct image information, and corresponds to the image date / time information, the store information, the image information identification information of the captured image information, and the image identification information of the correct image information. Let me remember. The coordinates of the vertex corresponding to the specified Nth captured image information are stored in association with the image information identification information of the captured date / time information, the store information, and the captured image information.

  Note that the vertex specifying process used in the trapezoidal correction process in the captured image information normalization processing unit 242 is not limited to the case of specifying a product from the image information obtained by capturing the display shelf as in the present invention, but the same object to be captured. The present invention can also be applied to the image recognition system 1 that corrects a plurality of pieces of image information obtained by photographing the image and generates the in-place image information. As a result, a plurality of pieces of image information obtained by photographing the same object to be photographed can be arranged in order to generate in-place image information of the object to be photographed.

  In addition, in order to specify the four vertices of the shelf position for the trapezoidal correction process in the Nth captured image information, in the above description, the feature points based on the local feature amounts in the feature amount collection area 2420 in the N−1th captured image information. The function F is obtained using the point group A and the point group B of the feature points based on the local feature amounts in the feature amount collection area 2421 in the N-th captured image information, and used in the N-1th trapezoid correction process. The process of projecting the coordinates of the vertex C onto the Nth captured image information by the function F and specifying the coordinates as the vertex D for the Nth trapezoid correction process has been described. However, in this process, it is only necessary to find the coordinates (positions) of corresponding points of similar image information in the (N-1) th captured image information and the Nth captured image information. However, any type of feature amount that identifies a location in the image information may be used. For example, there are sharp points in image information, highlight points, and the like. In this specification, a feature quantity that identifies a location in image information, such as a local feature quantity, is referred to as an image feature quantity (position-specific image feature quantity). In the description of the present specification, a case where the local feature amount is used as the image feature amount as described above will be described.

  Next, a vertex specifying process for performing the trapezoidal correction process in the second embodiment will be described. In this case, the vertex for performing the trapezoidal correction process has already been specified in the captured image information obtained by photographing an arbitrary display shelf, and the captured image information obtained at the same opportunity and at different angles is used for the same display shelf. Will be explained.

  The photographed image information obtained by photographing the display shelf of the store is input from the photographed image information input terminal 3, and the photographed image information input reception processing unit 241 of the management terminal 2 receives the input. FIG. 19 shows an example of photographed image information. Also, input of image information identification information such as shooting date / time, store identification information, and shooting image information is accepted. Then, the photographed image information input acceptance processing unit 241 stores the photographed image information, the photographing date / time, the store identification information, and the image information identification information that have received the input in the photographed image information storage unit 20 in association with each other.

  When the management terminal 2 accepts a predetermined operation input, the photographed image information normalization processing unit 242 extracts the photographed image information stored in the photographed image information storage unit 20 and changes the shelf position for executing the trapezoid correction process. A process of specifying the vertex D (D1 to D4) is executed.

  Vertex coordinates (for example, vertex coordinates C1 to C4) of the previous (N-1) th imaged image information (FIG. 20) in which the same or substantially the same area corresponding to the current (Nth) imaged image information (FIG. 19) is imaged. Identify). The vertex coordinates of the previous photographed image information may be specified based on the photographing date / time, the store identification information, the image identification information of the photographed image information, and the like.

  The captured image information normalization processing unit 242 extracts the (N-1) th captured image information from the captured image information storage unit 20, and for each vertex C1 to C4, a rectangle having a predetermined size including one vertex. The area is set as the feature amount collection area 2420 and is set to the (N-1) th captured image information. FIG. 20 shows a state where the feature amount collection area 2420 is set in the (N-1) th captured image information.

  Also, the captured image information normalization processing unit 242 extracts the N-th captured image information (FIG. 19) from the captured image information storage unit 20, and has a wider range of features than the N-1th feature amount collection region 2420. The quantity collection area 2421 is set to the Nth captured image information. FIG. 21 shows a state in which the feature amount collection area 2421 is set in the Nth captured image information. Each feature amount collection region 2421 in the Nth captured image information includes one N−1th feature amount collection region 2420 one by one. In FIG. 21, the inclusion relationship is shown by indicating the (N−1) th feature value collection region 2420 in the Nth feature value collection region 2421.

  The captured image information normalization processing unit 242 collects local feature amounts in each feature amount collection region 2420 set for the (N-1) th captured image information, and sets feature points and coordinates based on the local feature amounts. Remember. In addition, local feature amounts are collected in each feature amount collection region 2421 set for the Nth captured image information, and feature points and coordinate sets based on the local feature amounts are stored.

  The captured image information normalization processing unit 242 includes the feature amount collection region 2421 of the Nth captured image information closest to each local feature amount of each feature point in the feature amount collection region 2420 of the N−1th captured image information. The feature points of the local feature quantity in are identified, and the coordinates are associated with them as the feature points of the local feature quantity as a pair. FIG. 17 shows a pair relationship between the (N−1) th (N−1) th feature amount collection region 2420 and the Nth feature amount collection region 2421.

  A point group of feature points based on local feature amounts in the N-1th feature amount collection region 2420 is A, and a point group of feature points based on local feature amounts in the Nth feature amount collection region 2421 is B, N−1th time. If the vertex used in the trapezoidal correction processing of C is C (C1 to C4), the captured image information normalization processing unit 242 converts the point group A to the point group B based on the coordinates of the point group A and the point group B. A function F (perspective transformation / homography) to be projected onto is obtained.

  Then, the captured image information normalization processing unit 242 projects the coordinates of the vertex C (C1 to C4) used in the (N-1) th trapezoid correction process based on the obtained function F, and the Nth trapezoid correction process. Is specified as the coordinates of the vertex D (D1 to D4).

  By performing the above processing on the feature amount collection areas 2420 and 2421, the four vertices D (D1 to D4) for the Nth trapezoid correction process can be automatically specified. The coordinates of the identified Nth vertex D (D1 to D4) are stored in association with the image information identification information of the shooting date / time, the store identification information, and the captured image information. FIG. 22 shows the identified vertices D1 to D4.

  As described above, when the vertex D (D1 to D4) of the shelf position for the trapezoidal correction process for the Nth captured image information is specified, the captured image information alignment processing unit 24222 performs the vertex D (D1 to D4). ) To perform trapezoidal correction processing on the Nth captured image information.

  By executing the above process, the corresponding vertices can be automatically identified without specifying the four vertices used in the trapezoid correction process for the second and subsequent trapezoid correction processes. The burden on the user can be reduced.

  Furthermore, a modification of the face specifying process in the face specifying processor 245 of the first embodiment will be described. In the present embodiment, the face specifying process of the first embodiment may be performed as the first process, and the face specifying process may be automatically performed as the second and subsequent face specifying processes. Processing in this case will be described.

  In addition to the first time, the first time includes a case where the processing of the first embodiment is performed at an arbitrary timing in order to correct a shift when automatically specifying. The second and subsequent times are other than the first time. In addition, the second and subsequent times include processing of captured image information and in-place image information photographed at different angles with respect to captured image information and in-place image information captured on the same display shelf at the same opportunity. .

  The face identification processing unit 245 executes the same process as that of the first embodiment as the first face identification process. Then, the face identification processing for the second and subsequent times in the face identification processing unit 245 is performed by determining the coordinates of the area of the face specified by the previous (N-1th) in-place image information for the same shelf of the same display shelf. The extracted coordinates are used as the coordinates of the face area specified by the current (Nth) in-place image information.

  The coordinates of the face area are relative coordinates based on a predetermined position (for example, the vertex C1 at the upper left of the display shelf) in the display shelf in the in-place image information, similarly to the coordinates of the position of the shelf.

  Further, as a modification of the first embodiment, the process of the first embodiment is performed as the first process in the identification information specifying processing unit 246 for specifying the product identification information of the product displayed on the face for each shelf of the display shelf. The following process is executed as the product identification information specifying process and the product identification information specifying process for the second and subsequent times.

  The identification information identification processing unit 246 first identifies the face identification information of the face to be processed in the identification processing of the product identification information of the face in the N-th in-place image information. Let X denote the identified face identification information. Then, the image information of the area of the face identification information X in the Nth position image information is compared with the image information of the area in the (N−1) th position image information at a position corresponding to the face identification information X. For the similarity determination, in addition to the processing in the first to third embodiments described above, EMD between image information signatures (a set of image feature values and weights) of each face area is obtained, can do. As the image feature quantity of the signature, for example, the frequency distribution in the HSV color space of the image information of the face area is obtained, the hue and the saturation are grouped, and the number of the features and the image feature quantity by the area in the HSV color space are determined. can do. The reason for grouping hue and saturation is to reduce the dependence on lightness so that it is not greatly affected by shooting conditions.

  Further, in order to speed up the processing, similarities such as L2 distance between image feature quantities such as color correlograms and color histograms of image information in an appropriate color space can be used instead of the signature and EMD.

  If the similarity of the image information in the face area is equal to or greater than a certain threshold, the product identification information candidate corresponding to the face in that area in the N-1th in-place image information is extracted from the identification information candidate storage unit 22; The product identification information candidate of the face identification information X in the N-th in-place image information is used. Thereby, the candidate of the product identification information of the face identification information X in the Nth position image information to be processed can be specified. If the similarity is less than a certain threshold value, the image information of the area of the face identification information X in the Nth in-place image information and the sample information stored in the sample information storage unit 21 are obtained as in the first case. In comparison, the product identification information having similarity that is equal to or higher than a predetermined threshold and the highest similarity is specified as a candidate for product identification information of the face of the face identification information X in the N-th in-place image information.

  In comparison with the face area image information in the (N-1) th in-place image information, not only a comparison with the corresponding face position but also a face within a predetermined range may be included as a comparison target. For example, when comparing with the image information of the area of face identification information X in the N-th position image information, the comparison object includes the area of face identification information X in the N−1-th position image information and the area. An area of a face within a predetermined range, for example, a face located at a position spaced one or more in the left-right direction, and an area of a face located on upper and lower shelves may be included. Further, in addition to the area of face identification information X in the N-1th in-place image information, a plurality of adjacent face areas such as face identification information X-2, X-1, X, X + 1, and X + 2 are included. May be.

  In this case, the image information of the area of the face identification information X in the Nth position image information, and the image information of the area of the face range of the range to be compared in the N−1th position image information, And the face identification information of the N-1th in-place image information with the highest similarity is specified. Note that the similarity may be a condition that it is equal to or greater than a certain threshold. And the candidate of the product identification information corresponding to the face identification information of the specified N-1th in-place image information is extracted from the identification information candidate storage unit 22, and the product identification of the face identification information X in the N-th in-place image information Candidate for information. This process is schematically shown in FIG. FIG. 23A shows the previous (N-1th) in-place image information, and FIG. 23B shows the current (N-th) in-place image information. Then, the image information of each face area of the shelf 1 of the N-th position image information is compared with the image information of each face area of the shelf 1 of the N-1th position image information, respectively. The similarity is determined in step N, and the product identification information candidate for the face of shelf 1 in the N-1st in-place image information having the highest similarity is used as the product in the face of shelf 1 in the N-th place image information. Indicates that identification is made as a candidate for identification information. FIG. 23 shows a case in which comparison is made between the left and right faces in addition to the N-1th face image information face at the position corresponding to the Nth face image information face. . In addition, you may compare with the image information of the face position of not only the same shelf but an upper and lower shelf. For example, in the case of FIG. 23, when specifying the product identification information candidate of the face position at the center of the shelf 2 of the Nth position image information, the center of the shelf 2 of the N−1th position image information and In addition to comparison with the image information of the left and right face areas, the image information of the center of the shelf 1 of the N-1th in-place image information and the left and right face areas, N− Similarity may be compared with the image information of the center of the shelf 3 of the first position image information and the two face areas on the left and right sides thereof.

  As a result of the comparison of the N-1th in-place image information with the face image information, if the product identification information candidate cannot be specified because the similarity does not satisfy the threshold, the N-th in-place image information The image information in the area of the face identification information X in the sample information and the sample information stored in the sample information storage unit 21 are compared, and the product identification information having the highest similarity that is equal to or higher than a predetermined threshold is represented by N The face is identified as a candidate for product identification information of the face of the face identification information X in the in-place image information. The similarity determination process in this case can be performed in the same manner as the initial product identification information specifying process.

  The candidate product identification information identified as described above corresponds to shooting date / time information, store information, image information identification information of the photographed image information, image identification information of the in-place image information, and face identification information for identifying the face. Then, it is stored in the identification information candidate storage unit 22.

  In Example 1 to Example 4, in each piece of image information obtained by photographing the same display shelf at different angles at the same opportunity, by specifying candidates for product identification information in the face and comparing the corresponding faces, The merchandise identification information in the face is confirmed. In this embodiment, instead of specifying candidates for product identification information on the face for each piece of image information taken at different angles, the image information taken at different angles is first synthesized to eliminate the reflected portion of light or the like. A case will be described in which product identification information specifying processing in the face is executed on one piece of image information obtained after the insertion. An example of a block diagram of the system configuration of the image recognition system 1 in this case is shown in FIGS.

  The image recognition system 1 of the present embodiment includes a photographed image information storage unit 20, a specimen information storage unit 21, an identification information storage unit 23, and an image information processing unit 24. The captured image information storage unit 20, the sample information storage unit 21, and the identification information storage unit 23 have the same configuration as in the first to fourth embodiments.

  The image information processing unit 24 includes a captured image information input acceptance processing unit 241, a captured image information positive value processing unit, a shelf position specifying processing unit 243, a shelf area extraction processing unit 244, a face specifying processing unit 245, and identification information. A specific processing unit 246, a shelf image matching processing unit 247, and an image information processing unit 248 are included.

  The photographed image information input acceptance processing unit 241 and the photographed image information positive value processing unit execute the same processing as in the first to fourth embodiments. That is, the photographed image information input acceptance processing unit 241 accepts input of image information (photographed image information) of a store display shelf photographed by the photographed image information input terminal 3 and stores it in the photographed image information storage unit 20. In addition, the captured image information normalization processing unit 242 generates correct image information obtained by performing trapezoidal correction processing on the captured image information stored in the captured image information storage unit 20 and performing normalization.

  The image information processing unit 248 executes a process of combining a part or all of the in-position image information that has been subjected to the positive value processing based on the captured image information into one piece of image information. That is, a part or the whole range of each of the in-place image information obtained by photographing the same display shelf at the same opportunity at different angles is divided by a mesh using a predetermined reference unit, for example, about 10 to 100 pixels as a reference unit. And the average value of the brightness of the image information of each mesh area is calculated. The average value of brightness is compared for each mesh corresponding to each of the in-place image information photographed at different angles, and the mesh of the image information having the smallest average value of brightness is specified. This process is performed for all meshes. Image information with reduced influence of light reflection can be generated by combining the image information of the specified mesh area side by side in the vertical or horizontal direction as single image information (processed in-place image information). . In addition, about the image information of a mesh area | region, what is necessary is just to identify the mesh to synthesize | combine using not only the average value of lightness but the index value by the other evaluation criteria about lightness. The image information to be processed by the image information processing unit 248 is preferably corrected so that shooting conditions such as shutter speed and aperture value are the same or equivalent between the pieces of image information. When the above-described process using brightness is performed between image information with different shooting conditions or not corrected equally, image information with a low index value (darkness) based on brightness is selected for the part where there is no reflection. Will be. Even in this case, the reflection itself can be removed, but problems such as low contrast or brightness of the entire image information to be generated, or conspicuous roughness may occur, which may affect recognition accuracy. It is.

  Each processing in the face identification processing unit 245, the identification information identification processing unit 246, and the shelf image matching processing unit 247 is the same as that in the first to fourth embodiments, but the processing target is the shelf area of the in-place image information. Instead, it is performed on the processed in-place image information generated by the image information processing unit 248. In addition, in the first to fourth embodiments, the identification information specification processing unit 246 specifies the product identification information candidates for the products in the face area and stores them in the identification information candidate storage unit 22, but in this embodiment, The product identification information of the product in the face area is specified, and the specified information is stored in the identification information storage unit 23.

  By executing the processing on the processed in-place image information in this way, as in the first to fourth embodiments, the image of the range of the shelf area can be obtained without specifying the product identification information candidates in the face. Since the information is processed by the image information processing unit 248, the influence of light reflection on the shelf area can be reduced. Therefore, the product identification information on the face can be directly specified with high accuracy.

  The image information processing processing unit 248 performs processing on the in-place image information. The image information of the shelf position area specified by the shelf position specification processing unit 243 and the image of the shelf area cut out by the shelf area extraction processing unit 244 are as follows. However, the present invention is not limited to this. However, the present invention is not limited to this, and the information may be executed on a part or the whole range of the in-place image information generated by the captured image information positive value processing unit.

  Next, an example of the processing process in the present embodiment will be described with reference to the flowcharts of FIGS.

  The person in charge of photographing the display shelf of the store photographs the same display shelf from a plurality of angles so that the reflection of light or the like does not overlap. And about the display shelf of a store, the picked-up image information (FIG. 8, FIG. 9) image | photographed from the different angle at the same opportunity is input from the picked-up image information input terminal 3, and is image | photographed in the image information processing part 24 of the management terminal 2. The image information input reception processing unit 241 receives the input (S300). Also, input of image information identification information such as shooting date / time, store identification information, and shooting image information is accepted. Then, the photographed image information input acceptance processing unit 241 stores the photographed image information, the photographing date and time, the store identification information, and the image information identification information of the photographed image information, which are accepted, in the photographed image information storage unit 20 in association with each other.

  When the management terminal 2 accepts a predetermined operation input, the photographed image information normalization processing unit 242 extracts the photographed image information stored in the photographed image information storage unit 20 and performs a shelf that is a vertex for performing trapezoid correction processing. The input of 4 points of the position (the position of the display shelf) is received, and the keystone correction process is executed (S310). FIGS. 10 and 11 show examples of photographed image information (rectified image information) that has been subjected to the trapezoidal correction process in this way.

  Then, by receiving a predetermined operation input in the management terminal 2 for the in-place image information, the shelf position specification processing unit 243 specifies the shelf area (S320). That is, the input of the shelf area in the in-place image information is accepted. 12 and 13 are diagrams showing a state where the shelf region is specified.

  When the shelf area is specified as described above, the shelf area extraction processing unit 244 extracts the image information of the shelf area from the normal image information based on the shelf area received in S320. (S330).

  The image information processing unit 248 executes a processed in-place image information generation process for the image information of the shelf area cut out by the shelf area extraction processing unit 244 in S330 (S340).

  First, the image information processing unit 248 changes the magnification so that the vertical and horizontal sizes match or substantially match each corresponding shelf area with respect to the in-place image information of the extracted shelf area. Keep the size the same. Then, the in-place image information of the cut shelf area is divided by a predetermined reference unit, for example, every 10 pixels (S400). FIG. 28 schematically shows this state. FIG. 28A shows a state where FIG. 12 and FIG. 28B are meshed with respect to the shelf area of the in-place image information of FIG.

  Then, the image information processing unit 248 calculates, for each mesh, an average value of brightness for each image information of the mesh area (S410). The average value may be a simple average, a weighted average, or the like, or may be a value obtained by a calculation other than the average as long as it is a reference value based on the lightness of each image information in the mesh area.

  Then, the image information processing unit 248 compares the average value (reference value) of the brightness for each corresponding mesh (S420), and specifies the image information of the mesh with the lowest brightness among the comparisons (S430). In the case of FIG. 28, in each of the in-place image information in FIG. 28 (a) and FIG. 28 (b), for each mesh in the shelf area of each in-place image information, the brightness of each mesh area image information. The average value is calculated. Since the mesh A1 (FIG. 28A) and the mesh A2 (FIG. 28B) correspond to each other, the average brightness value of the mesh A1 is compared with the average brightness value of the mesh A2. For example, if the average value of lightness of the mesh A1 is small, the mesh A1 is specified. Further, since mesh B1 (FIG. 28A) and mesh B2 (FIG. 28B) correspond, the average brightness value of mesh B1 is compared with the average brightness value of mesh B2. For example, if the average value of the brightness of the mesh B2 is small, the mesh B2 is specified. In this way, the lightness comparison and specific processing for the corresponding mesh are performed for all meshes.

  Then, by arranging the specified meshes side by side vertically and horizontally, one piece of image information (processed in-place image information) is generated (S440). In the above example, since mesh A1, mesh B2, etc. are specified, meshes specified in S430 such as mesh A1, mesh B2, etc. are arranged side by side in the vertical and horizontal directions, so It is synthesized and generated as image information. This process is schematically shown in FIG. In FIG. 29, for the sake of easy understanding, the processing for the highest shelf area in the shelf area of FIG. 28 is shown as an example.

  By executing the processing as described above, it is possible to generate image information with reduced reflection of light for the shelf region.

  In this way, for the processed in-place image information, as in the first to fourth embodiments, the face identification processing unit 245 performs face identification processing, and the identification information identification processing unit 246 performs product identification information for each face. Specific processing is executed (S350, S360). That is, the face identification processing unit 245 identifies the face area in the processed in-place image information, and the identification information identification processing unit 246 stores the sample information stored in the specimen information storage unit 21 and the image information in the face area. The product identification information of the product in the face area is specified by executing the product identification information specifying process by the image matching process. Then, the identified product identification information is stored in the identification information storage unit 23 as in the first to fourth embodiments (S370).

  Even if the processing according to the present embodiment is used, the product identification information of all the faces cannot be specified. Therefore, for a face that has not been specified, input of product identification information is accepted, and the product identification information for which input has been accepted is taken as shooting date / time, store identification information, image information identification information of captured image information, and image information of in-place image information. The identification information may be stored in the identification information candidate storage unit 22 as a candidate for product identification information in association with the identification information and face identification information. Similarly, an input may be accepted for the correction process for the identified candidate product identification information.

  By performing the processing as described above, it is possible to specify the product identification information of the products displayed on the shelf of the display shelf shown in the captured image information.

  Also in this embodiment, the deformation process of each processing unit in the first to fourth embodiments can be used.

  If the sample information stored in the sample information storage unit 21 includes reflection of light or surrounding colors, it affects the image characteristics (image feature amount) of the appearance of the product (for example, package or wrapping). Affects the accuracy of the matching process. For this reason, it is preferable to use image information photographed so that the original color of the product is not impaired by the reflection of light and surrounding colors as the image information used as the sample information. However, depending on the shape and material of the appearance of the product, reflection may be unavoidable due to light reflection. For example, in the case of a metal can, since the reflectance is high, it is inevitable that the reflected light from the illumination is reflected in the photograph.

  Elements that give disturbances such as reflection when photographing sample information of a product include the shooting environment of the product and the appearance of the product. In addition to light sources such as lighting devices, the shooting environment includes the background of the product and the state of the surface on which the product is placed. In addition, the appearance of the product may reflect light from a photographing device, a lighting device, or the like depending on the direction of the surface constituting the appearance of the product.

  Therefore, in this embodiment, a process for obtaining sample information without reflection will be described. The processing in the present embodiment can also be applied to the sample information in the first to fifth embodiments.

  The following photographing methods can be applied as a method for reflection according to the photographing environment.

  In order to prevent distortion of the specimen information, the photographing with the photographing device is performed from the opposite direction of the product in the horizontal direction. In this case, since it may be unavoidable that the appearance of the product is reflected due to reflection or the like, the reflected image is a flat plate having no specific hue such as white or gray. Specifically, the product is photographed as sample information with white or gray having no specific hue as the background. White or gray paper or cloth may be used as the background of the product. The white or gray paper, cloth, etc. used as the background should be spread out to the front of the product until there is no crease on the surface on which the product is placed and the result of the reflection is a flat plate. This can reduce the effect of reflection due to the shooting environment.

  The following photographing methods can be applied as a method for reflection according to the appearance of the product.

  For products that are subject to sample information shooting, the product and lighting are not changed without changing the relative position between the product and the imaging device (for example, the relative position between the product surface captured by the imaging device and the imaging device). The product is photographed a plurality of times by changing the relative position with the light source (for example, the surface of the product irradiated with light from the light source and the relative position with the light source). In other words, the product is photographed a plurality of times by changing the relative position between the photographing device and the light source without changing the relative position between the product and the photographing device. In other words, the product is captured at the same position, the same size (size), and the same angle in the captured image information, and light is emitted from a light source such as an illumination to the product from different angles. The image recognition system 1 accepts the input. FIG. 30 shows an example of a state where a product is photographed as viewed from above. FIG. 30A is a diagram illustrating an example of a state in which illumination is performed from the left side of the product, and FIG. 30B is a diagram illustrating an example of a state in which illumination is performed from the right side of the product. FIG. 30 shows a case where the image is taken from two places on the left and right sides with the photographing apparatus interposed therebetween, but the image may be taken from three or more places. In FIG. 30, a light path of a reflection that is generated at a position closest to the center axis of the product due to light emitted from the light source closest to the photographing device is indicated by a broken line arrow. In this case, the angle of incident light (incident angle) and the angle of reflected light (reflective angle) are the same with the perpendicular to the surface of the product as the axis of symmetry.

  Then, the image recognition system 1 obtains appropriate sample information by cutting out portions where no reflection has occurred in a plurality of pieces of image information that have received inputs and synthesizing them. Appropriate sample information is stored in the sample information storage unit 21. FIG. 31 schematically shows this processing. FIG. 31A shows image information taken in the state of FIG. 30A, FIG. 31B shows image information taken in the state of FIG. 30B, and FIG. 31 (a) and 31 (b) are appropriate sample information obtained by synthesizing the image information. That is, in FIG. 30A, since the product is photographed by applying light from the right side, the reflection of light tends to occur on the right side of the product. Further, in FIG. 30B, since the product is photographed by applying light from the left side, the reflection of light tends to occur on the left side of the product. In other words, in FIG. 31A, there is no reflection of light on the left side (A) of the product, and in FIG. 31B, there is no reflection of light on the right side (B) of the product. Therefore, the image information on the left side (A) of the product in FIG. 31A and the image information on the right side (B) of the product in FIG. 31B are cut out and combined into one piece of image information. (FIG. 31 (c)). As a result, it is possible to synthesize appropriate sample information that does not reflect light.

  In FIGS. 30A and 30B, the sum α and β of the incident angle and the reflection angle are preferably the same or substantially the same. As a result, in the photographed image information, if the product appears in the vicinity of the center in the horizontal direction of the image information, the left half or the right half where the photographed image information does not appear are cut out from each other. Since it is only necessary to combine the images, the image combining process is simplified. When the sums α and β of the incident angle and the reflection angle are the same or substantially the same, the accuracy of the image information combining process can be improved, but the angles α and β may not be the same or substantially the same.

  The processing of FIGS. 30 and 31 will be described more specifically as follows. In general, the shape of the appearance of a product to be photographed can be roughly divided into three types as shown in FIG. 32 because the horizontal sectional view of the product is convex or substantially convex. That is, in the case of a rectangular parallelepiped or a substantially rectangular parallelepiped (a front surface is a plane) (FIG. 32 (a)), like a product in a box, a columnar shape, etc., and a side surface is vertical (a figure). 32 (b)), when a part of the side surface is not vertical, such as a wine bottle, such as a curved surface such as a spherical surface (FIG. 32 (c)). For convenience, FIG. 32A is referred to as a first case, FIG. 32B as a second case, and FIG. 32C as a third case.

  FIG. 33 is a diagram illustrating a case where the first case is photographed by the photographing apparatus. FIG. 33 (a) is a view from above when photographing a product, and FIG. 33 (b) is a view from the side when photographing a product. In the case of the first case, for example, one light source such as an illumination is used so that the photographing apparatus faces the product and the light that illuminates the product comes from one direction. Light emitted from a light source such as an illuminating device is reflected on a surface (reflecting surface in a range indicated by a one-dot chain line) facing the image capturing device in the product and is incident on the focus of the image capturing device.

  Then, if a straight line connecting the focal point of the imaging device and the right or left side of the reflection surface of the product is assumed to be a straight line with the perpendicular to the imaging device from the reflection surface of the product as the axis of symmetry, If the light source is installed at any position in the side area (area X, area Y), the product will not be reflected. Here, an angle formed by a straight line connecting the focal point of the photographing apparatus and the right side or the left side of the reflection surface of the product and the perpendicular line, and an angle between the line serving as a boundary line between the regions X and Y and the perpendicular line are incident. From the relationship between the angle and the reflection angle, both are angle a. Therefore, it is only necessary to take a picture from either one of the area X and the area Y (composition is not necessary). Alternatively, as shown in FIG. 33 (b), a region above the horizontal line from the top of the product and a straight line (the line with the horizontal line as the axis of symmetry) having the same angle b as the angle b from the imaging device to the top of the product. A region Z is set, and a light source is installed in the region Z to perform shooting. In the case of the first case, if the photographing is performed as shown in FIG. 33 (a) or FIG. 33 (b), the light source is not reflected on the product.

  FIG. 34 is a diagram illustrating a case where the second case is photographed by the photographing apparatus. In the second case, as shown in FIG. 34 (a), as in FIG. 33 (b), the horizontal line from the top of the product is the same angle b as the angle b from the photographing device to the top of the product. An area above the straight line (a line with the horizontal line as the axis of symmetry) is defined as an area Z, and a light source is installed in the area Z to perform photographing. If the image cannot be captured as shown in FIG. 34 (a), the image is captured as shown in FIG. 34 (b). FIG. 34 (b) is a view from above when photographing a product. In FIG. 34 (b), as in FIG. 30, the relative position between the product and the photographing device (for example, the relative position between the product surface photographed by the photographing device and the photographing device) is not changed. The product is photographed a plurality of times by changing the relative position between the product and a light source such as illumination (for example, the surface of the product illuminated by the light source and the relative position between the light source). Then, the image information photographed at each position is combined with a portion where there is no reflection to obtain sample image information.

  That is, the light emitted from the illumination is reflected by the reflection surface of the product, and enters the focus of the photographing apparatus. Therefore, when illumination is installed in the left area (area X) of the product, light emitted from the illumination causes reflection on the left side of the product, and illumination is installed in the right area (area Y) of the product. , The light emitted from the illumination causes reflection on the right side of the product. Therefore, the right side is specified in the image information when the light source is positioned on the left side of the product, and the left side is specified in the image information when the light source is positioned on the right side of the product, and combined into one image information. Specimen information.

  FIG. 35 is a diagram illustrating a case where the third case is photographed by the photographing apparatus. In FIG. 35, as in FIGS. 30 and 34 (b), the relative position between the product and the photographing device (for example, the relative position between the product surface photographed by the photographing device and the photographing device) is changed. Instead, the product is photographed a plurality of times by changing the relative position between the product and the light source such as lighting (for example, the relative position between the light source and the light source and the light source). Then, the image information photographed at each position is combined with a portion where there is no reflection to obtain sample image information.

  As described above, when photographing sample information of a product, a photographing method corresponding to the first to third cases may be applied in accordance with the shape of the appearance of the product. However, in general, since the appearance of the product has various shapes, it is not classified into three cases, but the third case (that is, the same photographing method as that in FIGS. 30, 34B, and 35 described above). ) Can be used for general purposes.

  By photographing the sample information of the product by the above-described photographing method, it is possible to obtain sample information in which disturbance due to both the photographing conditions and the shape of the appearance of the product is reduced.

  As a modification of the sixth embodiment, the process of the fifth embodiment may be used. That is, in order to generate appropriate specimen information, the same information as in the fifth embodiment is used for a plurality of pieces of image information obtained by changing the position of the light source without changing the relative positional relationship between the product and the imaging device. Execute the process. Specifically, among a plurality of pieces of image information obtained by photographing a product as in the fifth embodiment, all or part of the image information (region including the product) is divided into meshes, and an average value of brightness in the mesh is calculated. . Then, an average value of brightness is calculated for each corresponding mesh, and image information of the mesh having the smallest value is specified. Appropriate specimen information is obtained by arranging the image information of the mesh specified in this way side by side.

  The sample information generated by using the processing in the present embodiment performs processing similar to that in the first to fifth embodiments, thereby specifying the product identification information of the product shown in the image information, as well as the captured image information. Or you may comprise like the image recognition system 1 which performs the process which specifies merchandise identification information by performing an image matching process simply with an in-place image information and sample information.

  In the first to sixth embodiments described above, the case where the product identification information of the product displayed on the display shelf such as a store is specified from the image information obtained by photographing the display shelf has been described. Not only processing but also identification information of a specific object can be specified from image information. That is, image information obtained by photographing a photographing object (for example, an object having a plurality of specific objects arranged) or a specific object (an object for which identification information is specified) from a plurality of angles at the same opportunity, Can be applied to the image recognition system 1 that identifies the identification information of the specific object shown in the image information. As specific processing in that case, the processing of the first to sixth embodiments can be applied. In this case, the process for the display shelf may be read as the process for the photographing object, and the process for the product may be read as the process for the specific object.

  By using the image recognition system 1 of the present invention, even when light or the like is reflected in image information, an object such as a product can be specified with higher accuracy than in the past.

1: Image recognition system 2: Management terminal 3: Captured image information input terminal 20: Captured image information storage unit 21: Sample information storage unit 22: Identification information candidate storage unit 23: Identification information storage unit 24: Image information processing unit 25: Identification information comparison processing unit 70: arithmetic device 71: storage device 72: display device 73: input device 74: communication device 241: photographed image information input reception processing unit 242: photographed image information normalization processing unit 243: shelf position specification Processing unit 244: Shelf-level region extraction processing unit 245: Face identification processing unit 246: Identification information identification processing unit 247: Shelf-level image matching processing unit 248: Image information processing unit 2420: Features in the N-1th image information Quantity collection area 2421: Feature quantity collection area in the Nth image information

The invention according to claim 1, for the captured image information obtained by photographing a display shelf that commodities are displayed, and the photographed image information positive置化processing unit running a positive置化process produces a positive location image information, the positive In the display shelf shown in the image information, a face identification processing unit that identifies the face area, image information of the face area in the normal image information to be processed, and other information corresponding to the normal image information By comparing the image information of the face area within the predetermined range from the face area in the in-place image information, the product identification information of the product in the face area in the in-place image information to be processed An identification information specifying processing unit for specifying a candidate, the in-place image information based on the captured image information, and the in-place image information based on captured image information captured at an angle different from the captured image information; Oite, by at corresponding face each other to compare the candidate of the specified product identification information, the image recognition system having an identification information comparing unit for identifying the merchandise identification information item that is reflected in the photographed image information It is.

With the configuration of the present invention, even if there is a reflection of light or the like in the image information obtained by photographing the display shelf, the product can be recognized more accurately than in the past.
Since the image information obtained by photographing the display shelf is distorted or the like, the recognition accuracy increases if the processing is performed after performing the normalization processing. In addition, since a plurality of products are often displayed on the display shelf, it is preferable to identify candidates for product identification information by executing, for example, sample information and comparison processing for the region for each product.
Specifying product identification information by image matching processing with sample information has a large processing load. Therefore, if the faces that are the areas where the products are displayed are specified, comparing the faces can reduce the number of times of image matching processing and reduce the overall processing load. it can.

In the above-described invention, the identification information comparison processing unit corresponds to the in- place image information based on the captured image information and the in-place image information based on the captured image information captured at an angle different from the captured image information. It can be configured as an image recognition system that specifies the product identification information of the product shown in the photographed image information based on the presence or the likelihood of the specified product identification information between the faces to be identified.

In the above-described invention, the identification information specifying processing unit accepts input of product identification information when the product identification information candidate of the product shown in the area of the face cannot be specified, and at least the product identification information For an area that has received the input, the image identification system is configured to identify the received product identification information as the product identification information of the area without executing the processing in the identification information comparison processing unit. it can.

In the invention described above, the identification information identifying processor includes specific image information of the area of the face, by comparing the sample information of the product, the candidate of the merchandise identification information item that is reflected in the region of the face It can be configured like an image recognition system.

The invention described in claim 1 can be realized by reading the program of the present invention into a computer and executing it. In other words, the captured image information obtained by photographing the display shelf on which the product is displayed is subjected to the normalization process to generate the correct image information. In the display shelf, the face identification processing unit for identifying the face area, the image information of the face area in the normal image information to be processed, and other normal image information corresponding to the normal image information Identification information that identifies candidates for product identification information of products appearing in the face area in the in-place image information to be processed by comparing image information of the face area within a predetermined range from the face area The specific processing unit includes the normal image information based on the captured image information and the normal image information based on the captured image information captured at an angle different from the captured image information. Te, In with the corresponding face each other to compare the candidate of the specified product identification information, the image recognition program to function as an identification information comparing unit, for identifying the merchandise identification information item that is reflected in the photographed image information It can be constituted as follows.

The image information processing unit 24 identifies the captured image information input reception processing unit 241, the captured image information normalization processing unit 242 , the shelf position specifying processing unit 243, the shelf area extraction processing unit 244, and the face specifying processing unit 245. An information identification processing unit 246, a shelf image matching processing unit 247, and an image information processing unit 248 are included.

The photographed image information input acceptance processing unit 241 and the photographed image information regularization processing unit 242 perform the same processing as in the first to fourth embodiments. That is, the photographed image information input acceptance processing unit 241 accepts input of image information (photographed image information) of a store display shelf photographed by the photographed image information input terminal 3 and stores it in the photographed image information storage unit 20. In addition, the captured image information normalization processing unit 242 generates correct image information obtained by performing trapezoidal correction processing on the captured image information stored in the captured image information storage unit 20 and performing normalization.

The image information processing section 248, the scope of some or all of the Sei置image information running Sei置 process, a process for synthesizing a single image information, respectively, based on the captured image information. That is, a part or the whole range of each of the in-place image information obtained by photographing the same display shelf at the same opportunity at different angles is divided by a mesh using a predetermined reference unit, for example, about 10 to 100 pixels as a reference unit. And the average value of the brightness of the image information of each mesh area is calculated. The average value of brightness is compared for each mesh corresponding to each of the in-place image information photographed at different angles, and the mesh of the image information having the smallest average value of brightness is specified. This process is performed for all meshes. Image information with reduced influence of light reflection can be generated by combining the image information of the specified mesh area side by side in the vertical or horizontal direction as single image information (processed in-place image information). . In addition, about the image information of a mesh area | region, what is necessary is just to identify the mesh to synthesize | combine using not only the average value of lightness but the index value by the other evaluation criteria about lightness. The image information to be processed by the image information processing unit 248 is preferably corrected so that shooting conditions such as shutter speed and aperture value are the same or equivalent between the pieces of image information. When the above-described process using brightness is performed between image information with different shooting conditions or not corrected equally, image information with a low index value (darkness) based on brightness is selected for the part where there is no reflection. Will be. Even in this case, the reflection itself can be removed, but problems such as low contrast or brightness of the entire image information to be generated, or conspicuous roughness may occur, which may affect recognition accuracy. It is.

The image information processing processing unit 248 performs processing on the in-place image information. The image information of the shelf position area specified by the shelf position specification processing unit 243 and the image of the shelf area cut out by the shelf area extraction processing unit 244 are as follows. However, the present invention is not limited to this. However, the present invention is not limited to this, and the information may be executed on a part or the entire range of the in-place image information generated by the captured image information in-place processing unit 242 .

Claims (21)

An image information processing unit for identifying product identification information candidates of products shown in the image information for each piece of image information obtained by photographing a display shelf displaying products from a plurality of angles;
An identification information comparison processing unit for identifying product identification information of a product shown in the image information by comparing the identified product identification information candidates in corresponding image information;
An image recognition system comprising: The identification information comparison processing unit
Identifying the product identification information of the product in the image information based on the presence or likelihood of the product identification information candidates in the plurality of image information,
The image recognition system according to claim 1. The image information processing unit
A photographed image information normalization processing unit that executes positive value processing on the photographed image information obtained by photographing the display shelf and generates correct image information;
Identification for identifying candidate product identification information of products in the area by comparing the image information of a part of the display shelf in the display image information with the sample information of the product An information identification processing unit;
The image recognition system according to claim 1, further comprising: The identification information specifying processing unit
If the product identification information candidate of the product shown in the area cannot be specified, accept the input of the product identification information,
Identifying the received product identification information as the product identification information of the region without executing the process in the identification information comparison processing unit for at least the region of receiving the input of the product identification information;
The image recognition system according to claim 3. The image information processing unit
The display shelf shown in the in-place image information has a face identification processing unit for identifying a face area;
The identification information specifying processing unit
By comparing the image information of the face area in the in-place image information to be processed with the image information of the face area in a predetermined range from the face area in other in-place image information corresponding to the in-place image information. , Identify candidate product identification information for products in the face area of the in-place image information to be processed,
The image recognition system according to claim 3 or 4, characterized by the above. An image information processing unit for generating one piece of image information based on a plurality of pieces of image information obtained by photographing the same display shelf from different angles;
Based on the generated image information, an identification information specifying processing unit for specifying product identification information of a product shown in the image information;
An image recognition system comprising: The image information processing unit
For each of the plurality of image information, by comparing the index value based on lightness for each corresponding reference unit area, the image information of that area is identified,
By combining the image information of each identified area, the one image information is generated.
The image recognition system according to claim 6. The image information processing unit
For each of the plurality of image information, a reference unit is divided into meshes, and an index value based on the lightness in the mesh area is calculated,
For the plurality of image information, by comparing the index values in the corresponding mesh region, the image information of the mesh region is specified,
By combining the image information of each identified mesh area, the one image information is generated.
The image recognition system according to claim 6. A part or all of the image information showing the display shelf on which the product is displayed and a plurality of pieces of image information taken by changing the relative position between the product and the light source without changing the relative position between the product and the photographing device. An identification information specifying processing unit for specifying the product identification information of the product in the image information showing the display shelf by comparing with the sample information of the product based on
An image recognition system comprising: Accepts input of multiple image information taken by changing the relative position of the product and the light source without changing the relative position of the product and the imaging device,
The sample information of the product is generated by cutting out a part of the image information that has received the input and combining it into one image information.
An image recognition system characterized by that. A first processing unit for identifying candidates for identification information of specific objects shown in the image information for each piece of image information obtained by photographing the imaging object from a plurality of angles;
A second processing unit for identifying identification information of a specific object in the image information by comparing the identified identification information candidates in corresponding image information;
An image recognition system comprising: A first processing unit that generates one piece of image information based on a plurality of pieces of image information obtained by photographing a subject to be photographed from different angles;
Based on the generated image information, a second processing unit for identifying identification information of a specific object reflected in the image information;
An image recognition system comprising: Accepts input of multiple image information taken by changing the relative position of the object and the light source without changing the relative position of the object and the imaging device,
The sample information of the object is generated by cutting out a part of the image information that has received the input and combining it into one image information.
An image recognition system characterized by that. Change the relative position between the object and the light source without changing the relative position between the object and the imaging device, and capture multiple image information of the object,
The sample information of the object is generated by cutting out a region where there is no reflection in the captured image information and combining it into one image information.
A method for generating specimen information of an object characterized by the above. Computer
An image information processing unit that identifies candidates for product identification information of the product shown in the image information for each of the image information obtained by photographing the display shelf displaying the product from a plurality of angles;
An identification information comparison processing unit that identifies the product identification information of the product shown in the image information by comparing the identified product identification information candidates in the corresponding image information;
An image recognition program characterized in that it functions as a computer program. Computer
An image information processing unit for generating one piece of image information based on a plurality of pieces of image information obtained by photographing the same display shelf from different angles;
Based on the generated image information, an identification information specifying processing unit for specifying product identification information of a product shown in the image information;
An image recognition program characterized in that it functions as a computer program. Computer
A part or all of the image information showing the display shelf on which the product is displayed and a plurality of pieces of image information taken by changing the relative position between the product and the light source without changing the relative position between the product and the photographing device. An identification information specifying processing unit for specifying the product identification information of the product in the image information showing the display shelf by comparing with the sample information of the product based on
An image recognition program characterized in that it functions as a computer program. In the computer
Receiving a plurality of pieces of image information captured by changing the relative position between the product and the light source without changing the relative position between the product and the photographing device;
Generating a piece of sample information of the product by cutting out a part of the image information that has received an input and combining it into one image information;
An image recognition program characterized in that Computer
A first processing unit for identifying candidates for identification information of specific objects shown in the image information for each piece of image information obtained by photographing the imaging object from a plurality of angles;
A second processing unit for identifying identification information of a specific object shown in the image information by comparing the identified identification information candidates in corresponding image information;
An image recognition program characterized in that it functions as a computer program. Computer
A first processing unit that generates one piece of image information based on a plurality of pieces of image information obtained by photographing a subject to be photographed from different angles;
A second processing unit for identifying identification information of a specific object in the image information based on the generated image information;
An image recognition program characterized in that it functions as a computer program. In the computer
Receiving a plurality of pieces of image information captured by changing the relative position between the object and the light source without changing the relative position between the object and the imaging device;
Generating sample information of the object by cutting out a portion of the image information that has received an input and combining it into one image information;
An image recognition program characterized in that