CN110347772A - Article condition detection method, device and computer readable storage medium - Google Patents

Abstract

The embodiment of the present invention proposes a method, device, and computer-readable storage medium for item state detection, wherein the method includes tracking the human body of each user who enters the set area, so as to determine the real-time position of each user in the set area information; when the target item in the set area is detected to be moved, acquire the position information of the moved target item; according to the position information of the target item, and The real-time location information is matched to obtain the identification information of the target user who moves the target item. When the target item is moved, the location information of each user can be matched with the location information of the target item, so as to determine the target user who moved the target item. In this way, the confirmation and matching of target items and target users can be realized in an unmanned environment, realizing unmanned and automatic.

Description

Object state detection method, device and computer-readable storage medium

technical field

The present invention relates to the technical field of automation, in particular to a method, a device and a computer-readable storage medium for object state detection.

Background technique

Smart retail store is a consumption management and marketing service platform that integrates big data, intelligent software and hardware allocation to realize Internet, data and electronic stores. It can operate 24 hours a day, with low operating costs, which is convenient for the actual needs of residents. It is mainly aimed at the unmanned transformation of 24-hour convenience stores near communities and communities, which can greatly save costs compared with the current 24-hour convenience stores.

According to the development process, the existing smart retail stores include the following categories: unmanned retail cabinets, RFID-based convenience stores, unmanned retail cabinets for door-to-door shopping, and unmanned convenience stores. However, the above-mentioned types of solutions still have their own defects. For example, the number of products stored in the unmanned retail cabinet is limited, and each product in an RFID-based convenience store needs to be manually labeled with an RFID tag before it is put on the shelf. Relying on the camera, the detection error rate for the selected product is high, and it cannot be truly unmanned.

Contents of the invention

Embodiments of the present invention provide a method, device, and computer-readable storage medium for object state detection, so as to solve one or more technical problems in the prior art.

In a first aspect, an embodiment of the present invention provides a method for detecting an item state, including:

Perform human body tracking on each user entering the set area to determine the real-time position information of each user in the set area;

If it is detected that the target item in the set area is moved, acquiring the position information of the moved target item;

According to the position information of the target item and the real-time position information of each user in the set area, the identification information of the target user who moves the target item is obtained through matching.

In one embodiment, human body tracking is performed on each user entering the set area to determine the real-time position information of each user in the set area, including:

acquiring facial images of each of the users entering the set area;

Acquiring identification information of each user according to the facial image of each user;

According to the identification information of each of the users, human body tracking is performed on each of the users by means of pedestrian re-identification ReID, so as to determine the real-time position information of each of the users in the set area.

In one embodiment, according to the location information of the target item and the real-time location information of each user in the set area, matching to obtain the identification information of the target user who moves the target item includes:

Comparing the real-time location information of each user in the set area with the location information of the target item to determine the distance between each user and the target item;

Acquiring the identification information of the user with the closest distance to the target item.

In one embodiment, it also includes:

Obtain the color image and depth image of each user collected by the depth image acquisition device;

extracting key points of the human body from the color images of each of the users;

Corresponding the key points of the human body with the depth image to determine the position information of each user's hand;

Comparing the position information of the hands of each user with the position information of the target item to obtain the distance between the hands of each user and the target item;

Acquiring the identification information of the user whose hand is closest to the target item.

In one embodiment, it also includes:

In the case of detecting that the target item in the set area is moved, acquiring a facial image of the target user who moves the target item;

performing facial recognition on the facial image of the target user by using the existing images of the users in the set area, and acquiring the identification information of the target user;

Comparing whether the identification information of the target user acquired through facial recognition is consistent with the matching identification information of the target user who moves the target item.

In one embodiment, it also includes:

receiving trigger information from the gravity sensing sensor, the trigger information including the first moment when the item is touched and the second moment when the item is taken away;

Acquiring target position information corresponding to the gravity sensing sensor;

In the item images collected at the target location corresponding to the target location information, search for the first item image corresponding to the first moment and the second item image corresponding to the second moment;

comparing the difference between the image of the first item and the image of the second item to determine the target item taken away at the target position corresponding to the target position information.

In one embodiment, it also includes:

Obtaining the first weight of the gravity sensing sensor before the target item is taken away and the second weight after being taken away;

obtaining the difference between the first weight and the second weight;

Query the standard weight of the target item;

Comparing whether the difference is consistent with the standard weight of the target item.

In a second aspect, an embodiment of the present invention provides an item state detection device, including:

A user real-time location determination module, configured to track the human body of each user entering the set area, so as to determine the real-time position information of each user in the set area;

A target item position information determining module, configured to obtain the position information of the moved target item when it is detected that the target item in the set area is moved;

A matching module, configured to match the identification information of the target user who moves the target item according to the position information of the target item and the real-time position information of each user in the set area

In one embodiment, the user real-time location determination module includes:

The first facial image acquisition submodule is used to acquire the facial images of each of the users entering the set area;

The first identification information acquisition submodule is used to acquire the identification information of each user according to the facial image of each user;

The real-time position information determination sub-module is used to track the human body of each user by means of pedestrian re-identification ReID according to the identification information of each user, so as to determine the real-time position information of each user in the set area.

In one embodiment, the matching module includes:

The first distance determination submodule is used to compare the real-time location information of each user in the set area with the location information of the target item, and determine the distance between each user and the target item;

The second identification information acquisition submodule is configured to acquire the identification information of the user with the closest distance to the target item.

In one embodiment, it also includes:

An image acquisition module, configured to acquire the color image and depth image of each user collected by the depth image acquisition device;

A human body key point extraction module, used to extract human body key points from the color images of each of the users;

A hand position information determining module, configured to correspond the key points of the human body to the depth image, so as to determine the position information of each user's hand;

The second distance determination module is used to compare the position information of each user's hand with the position information of the target item, and obtain the distance between each user's hand and the target item;

The third identification information acquisition submodule is configured to acquire the identification information of the user whose hand is closest to the target item.

In one embodiment, it also includes:

The second facial image acquisition module is used to acquire the facial image of the target user who moves the target item when the target item in the set area is detected to be moved;

A facial recognition module, configured to perform facial recognition on the facial image of the target user by using the existing images of each user in the set area;

The fourth identification information acquisition sub-module is used to acquire the identification information of the target user according to the recognition result of the facial recognition module;

A judging module, configured to compare whether the identification information of the target user acquired through the fourth identification information acquisition sub-module is consistent with the identification information of the target user who moves the target item through matching by the matching module.

In one embodiment, it also includes:

The trigger information receiving module is used to receive the trigger information of the gravity sensor, and the trigger information includes the first moment when the item is touched and the second moment when the item is taken away;

A target position information acquisition module, configured to acquire target position information corresponding to the gravity sensing sensor;

An item image search module, configured to search for the first item image corresponding to the first moment and the second item image corresponding to the second moment in the item images collected at the target location corresponding to the target location information;

The target item determination module is configured to compare the difference between the first item image and the second item image to determine the target item taken away at the target position corresponding to the target position information.

In one embodiment, it also includes:

a weight acquisition module, configured to acquire the first weight of the gravity sensing sensor before the target item is taken away and the second weight after being taken away;

a difference determination module, configured to obtain the difference between the first weight and the second weight;

A standard weight inquiry module, used to inquire about the standard weight of the target item;

A weight comparison module, configured to compare whether the difference is consistent with the standard weight of the target item.

In a third aspect, an embodiment of the present invention provides an item status detection system, including:

The server includes the aforementioned item state detection device;

A human body tracking device, configured to collect tracking images of users in a set area and transmit them to the server;

The gravity sensing sensor is used to output sensing information to the server when it detects that the target item in the set area is moved.

In one embodiment, it also includes at least one of the following:

A facial image acquisition device, configured to acquire a facial image of a target user who moves the target item;

The depth image collection device is used to collect the depth images of each user.

In one embodiment, it also includes:

A core switch, communicatively connected to the server;

A plurality of secondary switches are communicatively connected to the core switch, and each of the secondary switches is connected to a specified type of front-end equipment;

Wherein, the secondary switch is used to control the data interaction between different front-end devices connected to itself, and the data interaction between the front-end device and the core switch; the front-end device includes a human body tracking device, a gravity sensor, a facial image One or more of an acquisition device and a depth image acquisition device.

In the fourth aspect, the embodiment of the present invention provides an item state detection system, and the functions of the system can be implemented by hardware, or can be implemented by executing corresponding software on the hardware. The hardware or software includes one or more modules corresponding to the above functions.

In a possible design, the structure of the system includes a processor and a memory, the memory is used to store a program that supports the system to execute the above object state detection method, and the processor is configured to execute the programs stored in memory. The system may also include a communication interface for communicating with other devices or a communication network.

In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium for storing computer software instructions used by an object state detection device, which includes the programs involved in executing the above object state detection method.

One of the above technical solutions has the following advantages or beneficial effects: when the target item is moved, the location information of each user can be used to match the location information of the target item, so as to determine the target user who moved the target item. In this way, the confirmation and matching of target items and target users can be realized in an unmanned environment, realizing unmanned and automatic.

Another technical solution in the above technical solutions has the following advantages or beneficial effects: in the detection process, the position of each user's hand is obtained, and the target user is judged according to the distance between the received position and the position of the target item. Therefore, when the target item is moved, multiple users in front of the shelf where the target item is located may be the target user.

The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the drawings and the following detailed description.

Description of drawings

In the drawings, unless otherwise specified, the same reference numerals designate the same or similar parts or elements throughout the several drawings. The drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in accordance with the present invention and should not be taken as limiting the scope of the present invention.

Fig. 1 shows a flowchart of a method for detecting an item state according to an embodiment of the present invention.

Fig. 2 shows a flowchart of a method for detecting an item state according to an embodiment of the present invention.

Fig. 3 shows a flowchart of a method for detecting an item state according to an embodiment of the present invention.

Fig. 4 shows a flowchart of a method for detecting an item state according to an embodiment of the present invention.

Fig. 5 shows a schematic diagram of images collected by a depth image collection device according to an embodiment of the present invention.

Fig. 6 shows a schematic diagram of images collected by a depth image collection device according to an embodiment of the present invention.

Fig. 7 shows a flowchart of a method for detecting an item state according to an embodiment of the present invention.

Fig. 8 shows a flowchart of a method for detecting an item state according to an embodiment of the present invention.

Fig. 9 shows a flowchart of a method for detecting an item state according to an embodiment of the present invention.

Fig. 10 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Fig. 11 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Fig. 12 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Fig. 13 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Fig. 14 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Fig. 15 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Fig. 16 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Fig. 17 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Fig. 18 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Fig. 19 shows a structural block diagram of an object state detection device according to an embodiment of the present invention.

Detailed ways

In the following, only some exemplary embodiments are briefly described. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature and not restrictive.

Fig. 1 shows a flowchart of an object detection method according to an embodiment of the present invention. This embodiment is described by taking a smart retail store as a scenario, as shown in Figure 1, the method includes the following steps:

S101: Perform body tracking on each user entering a set area, so as to determine real-time position information of each user in the set area.

For example, the setting area may be inside a smart retail store, or inside a three-dimensional space such as a warehouse or a warehouse. Set up multiple user tracking devices in the set area to track the human body of each user entering the set area. The user tracking device may use the opening of the gate at the entrance of the set area as a trigger instruction. The position information of each user tracking device in three-dimensional space is pre-stored in the database. The image of the user collected by the user tracking device may be called a tracking image.

The user tracking device can upload the collected tracking images of each user to the background server. The background server calculates the location information of each user based on the tracking image. The location information of the user may be expressed in the coordinate form of the world coordinate system.

S102: Acquiring position information of the moved target item when it is detected that the target item in the set area is moved.

For example, after placing items on a shelf, each item has corresponding location information. A gravity sensing sensor may be provided on the surface of the shelf for carrying items, and the items are placed on the gravity sensing sensor. For example, if the shelf includes cargo lane A and cargo lane B, gravity sensing sensors are respectively arranged on the surfaces of the cargo lane A and cargo lane B for carrying items. The location information of shelves, cargo lanes and gravity sensing sensors can be stored in the database in advance. The cargo lane and the gravity sensing sensor on the cargo lane can use the same position information. The shelf may use the position information of any aisle on the shelf, or the position information of the aisle in the middle of the shelf.

When the user moves to a certain target item, the gravity sensing sensor at the bottom of the item detects the weight change of the item, and can output a sensing information. The sensing information may include one or more of the time information when the weight change of the item is detected, the weight change value, and the identification information of the gravity sensing sensor. The user's operation of moving the item may include moving in the shelf, putting it in the shelf, taking it out from the shelf, and the like.

After obtaining the sensing information sent by the gravity sensing sensor, the background server queries the location information of the gravity sensing sensor in the database according to the identification information of the gravity sensing sensor. Since the position of the gravity sensor is close to the moved target item, the position of the gravity sensor is used as the position information of the moved target item.

S103: According to the location information of the target item and the real-time location information of each user in the set area, match to obtain identification information of a target user who moves the target item.

At a certain moment, for example, when the target object is detected to be moved, each human body tracking device may acquire the location information of each user within the set area at the same moment. Calculate the distance between each user and the target item being moved. Matching can be performed according to the distance, and the user who is the closest to the target item is selected as the target user.

Through the above embodiments, when the target item is moved, the location information of each user can be used to match with the location information of the target item, so as to determine the target user who moved the target item. In this way, the confirmation and matching of target items and target users can be realized in an unmanned environment, realizing unmanned and automatic.

As shown in Figure 2, in an implementation manner, step S101 includes:

S1011: Acquire facial images of each of the users entering the set area.

For example, a user can pre-register through a terminal such as a mobile phone or a handheld computer, and upload the registration information to the background server. When registering, users can directly input registration information, or import registration information by binding social software. The registration information may include information such as the user's name, phone number or social software account number and password. During the registration process, the registration information can also be associated with the user's facial image. A first image acquisition device may be installed at the entrance of the set area to collect facial images of users who are about to enter the set area. The first image collection device transmits the collected facial images of each user to the background server.

S1012: Obtain identification information of each user according to the facial image of each user.

The background server recognizes the acquired facial images of each user, thereby completing the identity authentication of each user. After the user enters the set area, in order to protect the privacy of the user, identification information may be assigned to the user. Replace the user's registration information with identification information.

S1013: According to the identification information of each of the users, use the pedestrian re-identification technology ReID to perform body tracking on each of the users, so as to determine the real-time position information of each of the users in the set area.

Multiple user tracking devices cooperate to collect images to achieve full coverage of image collection in a set area. The background server receives the tracking images collected by multiple user tracking devices, and uses the pedestrian re-identification technology ReID to track the human body of each user, so as to determine the location information of each user in real time. The real-time location information of each user is associated with the identification information of the user.

Through the above embodiments, it is possible to implement human body tracking for each user within the set area, and acquire location information of each user in real time.

As shown in Figure 3, in an implementation manner, step S103 includes:

S1031: Compare the real-time location information of each user in the set area with the location information of the target item, and determine the distance between each user and the target item.

S1032: Obtain the identification information of the user with the closest distance to the target item.

Through the above embodiments, the target user who moves the target item is determined by using the distance between the location of the target item and the location of each user.

As shown in Figure 4, in one embodiment, the method also includes:

S401: Acquire the color image and depth image of each user collected by the depth image collection device.

For example, the depth image acquisition device may be a binocular VSLAM depth camera. As shown in Figure 5 and Figure 6, a depth camera that shoots vertically downward and obliquely is set up above each shelf in the set area. The depth camera can capture the color image and the depth image in front of the shelf, and can correspond the color image and the depth image. Among them, the key points of the human body of each user can be extracted based on the color image, and the positions of the key points of the human body can be obtained based on the depth image. All depth cameras can perform extrinsic calibration, so that the positions of key points of each user's human body in the depth image can be mapped in three-dimensional space by using extrinsic calibration. In this way, the location information of the key points of the user's body can be obtained.

S402: Extract key points of the human body from the color images of each user.

Human body key point detection and tracking are performed in the acquired color images. The key points of the human body can only retain the key points of the upper body of the human body, such as the head, shoulders, elbows, hands, etc. By connecting the above key points, it can be guaranteed that each user corresponds to a set of key point connection diagrams, and different users can be distinguished based on the key point connection diagrams. The key points of the hand are extracted from the key point connection graph of each user, and the key points of each user's hand detected on the color image are corresponding to the depth image. Using the external parameter calibration, the position information of each user's hand is calculated by knowing the position of each user's hand in the depth image.

S403: Correspond the key points of the human body with the depth image, so as to determine the position information of each user's hand.

S404: Compare the position information of the hand of each user with the position information of the target item to obtain the distance between the hand of each user and the target item.

As shown in Fig. 5 and Fig. 6, there are three people in front of the shelves including user A, user B and user C. After calculating the position information of the hands of the three users respectively, the distances between the positions of the hands of the three users and the items on the shelf can be obtained.

S405: Obtain identification information of a user whose hand is closest to the target item.

Through the above embodiments, the position of each user's hand is obtained, and the target user is determined according to the distance between the received position and the position of the target item. Therefore, when the target item is moved, multiple users in front of the shelf where the target item is located may be the target user. For example, when a couple is moving items, they stand side by side in front of the shelf. Or two users stand in front of the shelf in a front-to-back positional relationship, and the user who is far away from the shelf reaches out to pick and place items. Or in complex scenarios where multiple users cross pick and place items at the same time, it is necessary to judge based on the depth: which user the hand that picks up the item belongs to, and which shelf is the item picked and placed on.

As shown in Figure 7, in one embodiment, the method also includes:

S501: Acquiring a facial image of a target user who moves the target item when it is detected that the target item in the set area is moved.

At different shelves or different aisles of the shelves, second image acquisition devices facing the front of the shelves may be provided respectively. The installation height should be suitable for photographing users standing in front of the shelf, for example, it can be about 1.4 meters. The location information of the second image capture device may be acquired in advance and stored in the database.

The back-end server obtains the position information of the moved target item, and queries the shelf or aisle corresponding to the position information of the moved target item in the database. Outputting a photographing control instruction to the second image acquisition device installed at the shelf or aisle, controlling it to take pictures of the user standing in front of the shelf. The image sent back by the second image acquisition device is received. Alternatively, the second image acquisition device acquires the image of the user standing in front of the shelf in real time. When the back-end server acquires that the gravity sensor detects that the target item is moved, it queries the database for the shelf or aisle corresponding to the position information of the moved target item. The image retrieval instruction is output to the second image acquisition device arranged at the shelf or the aisle. The image sent back by the second image acquisition device is received.

S502: Perform facial recognition on the image of the target user by using the existing images of each user in the set area, and acquire identification information of the target user.

The backend server receives the image collected by the second image collection device, and compares it with the existing images of each user in the set area. Determine the user standing in front of the shelf, and obtain the identification information of the user.

S503: Compare whether the identification information of the target user obtained through facial recognition is consistent with the matching identification information of the target user who moves the target item.

The identification information of the target user who is matched to move the target item obtained in step S103 is retrieved, and compared with the identification information of the user acquired in step S502. Thus, the recognition result in step S103 is verified. If they are the same, determine the target user who moves the target item obtained through matching in step S103. If not, an auxiliary determination may be performed according to the determination result of step S404.

Through the above embodiments, the identification information of the user is determined by using the image of the user captured by the image sensor that identifies the corresponding position of the item when the item is moved. The determined identification information is verified in step 103 to match the identification information of the target user of the moving target item. Therefore, the accuracy rate of identifying the target user can be improved.

As shown in Figure 8, in one embodiment, the method also includes:

S601: Receive trigger information from a gravity sensing sensor, where the trigger information includes a first moment when an item is touched and a second moment when the item is taken away.

When the user touches the item, due to the external force exerted on the item, the gravity sensing sensor will generate a trigger information at the first moment when the item is touched. After the user takes the item away, the gravity sensing sensor will detect a weight data at the second moment when the item is taken away, thereby generating another trigger information. The trigger information may include identification information of the gravity sensing sensor and (first or second) time information.

S602: Obtain target position information corresponding to the gravity sensing sensor.

After the backend server obtains the trigger information, it can query the location information of the gravity sensor in the database according to the identification information of the gravity sensor included in the trigger information.

S603: Search for the first item image corresponding to the first moment and the second item image corresponding to the second moment in the item images collected at the target location corresponding to the target location information.

A third image acquisition device facing the items is arranged inside the shelf. The third image acquisition device may be a color sensor RGB sensor. The third image acquisition device can be arranged according to the cargo lane. For example, if the shelf includes cargo aisle A and cargo aisle B, a third image acquisition device facing the cargo aisle A and cargo aisle B is respectively provided. The location information of the third image capture device may be acquired in advance and stored in the database. The third image acquisition device may use the same location information as the cargo lane it faces.

Using the position information of the gravity sensing sensor, the third image acquisition device with the same position information can be queried in the database. The backend server may send an image retrieval instruction to the third image acquisition device at the target location corresponding to the target location information at the first moment and the second moment, so as to obtain the first item image and the second image at the above two moments. Item image.

S604: Compare the difference between the image of the first item and the image of the second item to determine the target item taken away at the target position corresponding to the target position information.

If the user takes away the target item, compared with the first item image, the missing item in the second item image is the target item. Image recognition can be performed on the image of the target item, so as to determine the target item taken away at the target position corresponding to the target position information. Image recognition can pre-establish recognition models. The recognition model can be configured as follows: input images of different items on the shelf at the input end, and output relevant information of the item at the output end of the model. The relevant information may be one or more of information such as the name, weight, and amount of the item.

Through the above technical solution, the image changes before and after the item is taken are compared, and the moved item is determined according to the image change.

As shown in Figure 9, in one embodiment, the method also includes:

S701: Obtain a first weight of the gravity sensing sensor before the target item is taken away and a second weight after the target item is taken away.

S702: Obtain a difference between the first weight and the second weight.

S703: Query the standard weight of the target item.

The standard weights of different items can be obtained in advance and stored in the database.

S704: Compare whether the difference is consistent with the standard weight of the target item.

For example, the standard weight of a bottle of beverage in a can is 350g. In a case where it is determined that the target item to be taken away from cargo lane A is a bottle of beverage in a can, the difference between the first weight and the second weight is obtained. Assuming that the difference is 350g, the difference is consistent with the standard weight, and it is verified that the target item is a beverage in a can. Assuming that the difference is 500g, the difference is inconsistent with the standard weight, and retesting is required.

The re-inspection may include: the back-end server obtains the adjacent location of the target location, for example, it may be the B cargo lane adjacent to the A cargo lane. Repeat steps S601 to S604. The difference is that in step S603, among the item images captured by the third image acquisition device at the adjacent position (cargo aisle B), search for the first item image corresponding to the first moment, and the item image corresponding to the second moment. The second item image is used to determine the target item taken away at the target position corresponding to the target position information. For example, if the target item photographed by the third image acquisition device of B cargo lane is a bottle of beverage in a pop-top can, then the target item is determined to be a bottle of beverage in a pop-top can. And the target item photographed by the third image acquisition device of B cargo lane is a bag of food, then the weight of the food can be further queried in the database, if it is 500g, then it is determined that the bag of food is the target item.

Through the above-mentioned embodiment, in the case that the inquired standard weight does not match the difference, an auxiliary determination is made with the aid of the image captured by the third image capture device at a position adjacent to the target position. Since the angle between the vector position and the target position is different, the target item is finally determined through multi-angle picture comparison. In order to improve the confidence of determining the target item.

Fig. 10 shows a structural block diagram of an object state detection device according to an embodiment of the present invention. As shown in Figure 10, the device includes:

The user real-time position determination module 1001 is used to track the human body of each user entering the set area, so as to determine the real-time position information of each user in the set area.

The target item position information determination module 1002 is configured to acquire the position information of the moved target item when it is detected that the target item in the set area is moved.

The matching module 1003 is configured to match and obtain the identification information of the target user who moves the target item according to the position information of the target item and the real-time position information of each user in the set area.

As shown in FIG. 11 , in an implementation manner, the user real-time location determining module 1001 includes:

The first facial image acquisition sub-module 10011 is used to acquire the facial images of the users entering the set area.

The first identification information acquisition sub-module 10012 is configured to acquire the identification information of each user according to the facial image of each user.

The real-time location information determination sub-module 10013 is used to track the human body of each user by using the pedestrian re-identification ReID method according to the identification information of each user, so as to determine the real-time location information of each user in the set area .

As shown in FIG. 12, in an implementation manner, the matching module 1003 includes:

The first distance determination sub-module 10031 is used to compare the real-time location information of each user in the set area with the location information of the target item, and determine the distance between each user and the target item;

The second identification information acquisition sub-module 10032 is configured to acquire the identification information of the user with the closest distance to the target item.

As shown in Figure 13, in one embodiment, the device also includes:

An image acquisition module 1301, configured to acquire the color image and depth image of each user collected by the depth image acquisition device;

Human body key point extraction module 1302, for extracting human body key points from the color images of each user;

A hand position information determination module 1303, configured to correspond the key points of the human body to the depth image, so as to determine the position information of each user's hand;

The second distance determination module 1304 is configured to compare the position information of the hands of each user with the position information of the target item, and obtain the distance between the hands of each user and the target item;

The third identification information acquisition submodule 1305 is configured to acquire the identification information of the user whose hand is closest to the target item.

As shown in Figure 14, in one embodiment, the device also includes:

The second facial image acquisition module 1401 is used to acquire the facial image of the target user who moves the target item when it is detected that the target item in the set area is moved;

A facial recognition module 1402, configured to perform facial recognition on the facial image of the target user by using the existing images of each user in the set area;

The fourth identification information acquisition sub-module 1403 is used to acquire the identification information of the target user according to the recognition result of the facial recognition module;

The judging module 1404 is configured to compare whether the identification information of the target user acquired through the fourth identification information acquisition sub-module is consistent with the identification information of the target user who moves the target item through matching by the matching module.

As shown in Figure 15, in one embodiment, the device also includes:

The trigger information receiving module 1501 is configured to receive the trigger information of the gravity sensor, the trigger information includes the first moment when the item is touched and the second moment when the item is taken away;

Gravity sensing sensor position information acquisition module 1502, configured to acquire target position information corresponding to the gravity sensing sensor;

Item image search module 1503, configured to search for the first item image corresponding to the first moment and the second item image corresponding to the second moment in the item images collected at the target location corresponding to the target location information;

A target item determination module 1504, configured to compare the difference between the first item image and the second item image to determine the target item taken away at the target position corresponding to the target position information.

As shown in Figure 16, in one embodiment, the device further includes:

A weight acquisition module 1601, configured to acquire the first weight of the gravity sensing sensor before the target item is taken away and the second weight after being taken away;

A difference determination module 1602, configured to obtain the difference between the first weight and the second weight;

A standard weight query module 1603, configured to query the standard weight of the target item;

A weight comparison module 1604, configured to compare whether the difference is consistent with the standard weight of the target item.

Fig. 17 shows a structural block diagram of an object detection system according to an embodiment of the present invention. As shown in Fig. 17, the system includes:

The server 1701 includes the above-mentioned item state detection device;

The human body tracking device 1702 is used to collect tracking images of users in the set area and transmit them to the server;

The gravity sensing sensor 1703 is configured to output sensing information to the server when it detects that the target item in the set area is moved.

In one embodiment, the system also includes at least one of the following:

A facial image acquisition device, configured to acquire a facial image of a target user who moves the target item;

The depth image collection device is used to collect the depth images of each user.

In one embodiment, it also includes:

A core switch, communicatively connected to the server;

A plurality of secondary switches are communicatively connected to the core switch, and each of the secondary switches is connected to a specified type of front-end equipment;

Wherein, the secondary switch is used to control the data interaction between different front-end devices connected to itself, and the data interaction between the front-end device and the core switch; the front-end device includes a human body tracking device, a gravity sensor, a facial image One or more of an acquisition device and a depth image acquisition device.

As shown in FIG. 18 , for example, the system may include a secondary switch, a background server, and a storage server that are communicatively connected to the core switch, respectively. The secondary switch has two functions, one is to transmit data to the front-end equipment connected to it, and the other is to supply power to the front-end equipment connected to it. The front-end equipment may be a gravity sensing sensor, a first image sensor, a second image acquisition device, a third image acquisition device, a depth image acquisition device, a user tracking device, a gate, and the like. PoE (Power Over Ethernet) networking can be used between the front-end equipment and the secondary switch, that is, only one network cable is required for data transmission and power supply. Therefore, the difficulty of wiring can be reduced, and the reliability of the overall system can be increased. Among them, the core switch and the background server can be connected by 10G optical fiber, which has the advantages of low delay and strong data forwarding ability, which greatly improves the efficiency of network transmission. Gigabit network cables can be used to connect the core switch and the secondary switch. Wireless communication is used between the core switch and the storage server, such as 4G network, 5G network or WiFi network.

Secondary switches can include:

Rack secondary switches can correspond to the number of racks. The shelf secondary switch is connected with front-end equipment such as the gravity sensor on the corresponding shelf, the second image acquisition device, the third image acquisition device, and the depth image acquisition device.

By constructing the above-mentioned network topology structure, it is convenient to route the network. For example, all item identification information and user identification information of a shelf are transmitted in a secondary switch. Only one network cable is needed to connect to the core switch. It is convenient for engineering wiring, troubleshooting and maintenance.

The secondary switch for human body tracking can be set according to the number of user tracking devices. For example, the set area may be equally divided into N areas according to the deployment position of the user tracking device. Each area corresponds to a human body tracking secondary switch, which is connected to each user tracking device distributed in the area.

The human-computer interaction secondary switch is respectively connected to front-end equipment with human-computer interaction functions such as the first image sensor and the gate. As an extended embodiment, the front-end equipment with human-computer interaction function may also include UPS equipment, electric door equipment, display equipment and other front-end equipment connections.

According to the functional classification, the front-end devices under this function can communicate with each other, and the data can be exchanged without passing through the core switch. For example, in one embodiment, the first image sensor is integrated with a recognition function, and after the first image sensor recognizes the user, it can output control information to open the gate. Alternatively, after the first image sensor recognizes the user, the identification information of the user is sent to the display device. The display device pushes relevant information and the like on the display device according to the recognized item acquisition habit of the user.

All front-end devices perform network broadcasting. For example, when synchronous triggering is required to capture pictures, because they are on the same secondary switch, the network delay is consistent, so the time points for obtaining pictures are synchronized.

Storage servers can be used to store log files and data, etc. The log files may include record files or file collections of background servers, core switches, and secondary switches that record system operation events. Data can include registration information, location information, etc.

Fig. 19 shows a structural block diagram of an item status detection system according to an embodiment of the present invention. As shown in FIG. 19 , the system includes: a memory 1910 and a processor 1920 , and the memory 1910 stores computer programs that can run on the processor 1920 . When the processor 1920 executes the computer program, the object status detection method in the above-mentioned embodiments is implemented. The number of the memory 1910 and the processor 1920 may be one or more.

The system also includes:

The communication interface 1930 is used to communicate with external devices for interactive data transmission.

The memory 1910 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory.

If the memory 1910, the processor 1920, and the communication interface 1930 are independently implemented, the memory 1910, the processor 1920, and the communication interface 1930 may be connected to each other through a bus to complete mutual communication. The bus may be an Industry Standard Architecture (ISA, Industry Standard Architecture) bus, a Peripheral Component Interconnect (PCI, Peripheral Component Interconnect) bus, or an Extended Industry Standard Architecture (EISA, Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in FIG. 19 , but it does not mean that there is only one bus or one type of bus.

Optionally, in specific implementation, if the memory 1910, the processor 1920, and the communication interface 1930 are integrated on one chip, the memory 1910, the processor 1920, and the communication interface 1930 may communicate with each other through the internal interface.

An embodiment of the present invention provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, the method described in any one of the above-mentioned embodiments is implemented.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order depending on the functions involved, which shall It is understood by those skilled in the art to which the embodiments of the present invention pertain.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium, For use with instruction execution systems, devices, or devices (such as computer-based systems, systems including processors, or other systems that can fetch instructions from instruction execution systems, devices, or devices and execute instructions), or in conjunction with these instruction execution systems, devices or equipment for use. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use in or in conjunction with an instruction execution system, device or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Read Only Memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, since the program can be read, for example, by optically scanning the paper or other medium, followed by editing, interpretation or other suitable processing if necessary. processing to obtain the program electronically and store it in computer memory.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. During execution, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of its various changes or modifications within the technical scope disclosed in the present invention. Replacement, these should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (19)

1. A method for detecting the state of an article, characterized in that, comprising: Perform human body tracking on each user entering the set area to determine the real-time position information of each user in the set area; If it is detected that the target item in the set area is moved, acquiring the position information of the moved target item; According to the position information of the target item and the real-time position information of each user in the set area, the identification information of the target user who moves the target item is obtained through matching. 2. The method according to claim 1, wherein the human body tracking is performed on each user who enters the set area to determine the real-time position information of each user in the set area, including: acquiring facial images of each of the users entering the set area; Acquiring identification information of each user according to the facial image of each user; According to the identification information of each of the users, human body tracking is performed on each of the users by means of pedestrian re-identification ReID, so as to determine the real-time position information of each of the users in the set area. 3. The method according to claim 1, characterized in that, according to the location information of the target item and the real-time location information of each user in the set area, the location of the target user who moves the target item is matched. Identification information, including: Comparing the real-time location information of each user in the set area with the location information of the target item to determine the distance between each user and the target item; Acquiring the identification information of the user with the closest distance to the target item. 4. The method according to claim 1, further comprising: Obtain the color image and depth image of each user collected by the depth image acquisition device; extracting key points of the human body from the color images of each of the users; Corresponding the key points of the human body with the depth image to determine the position information of each user's hand; Comparing the position information of the hands of each user with the position information of the target item to obtain the distance between the hands of each user and the target item; Acquiring the identification information of the user whose hand is closest to the target item. 5. The method according to claim 1, further comprising: In the case of detecting that the target item in the set area is moved, acquiring a facial image of the target user who moves the target item; performing facial recognition on the facial image of the target user by using the existing images of the users in the set area, and acquiring the identification information of the target user; Comparing whether the identification information of the target user obtained through facial recognition is consistent with the identification information of the target user who moves the target item obtained through matching. 6. The method according to claim 1, further comprising: receiving trigger information from the gravity sensing sensor, the trigger information including the first moment when the item is touched and the second moment when the item is taken away; Acquiring target position information corresponding to the gravity sensing sensor; In the item images collected at the target location corresponding to the target location information, search for the first item image corresponding to the first moment and the second item image corresponding to the second moment; The difference between the first item image and the second item image is compared to determine the target item taken away at the target position corresponding to the target position information. 7. The method according to claim 6, further comprising: Obtaining the first weight of the gravity sensing sensor before the target item is taken away and the second weight after being taken away; obtaining the difference between the first weight and the second weight; Query the standard weight of the target item; Comparing whether the difference is consistent with the standard weight of the target item. 8. An object state detection device, characterized in that, comprising: A user real-time location determination module, configured to track the human body of each user entering the set area, so as to determine the real-time position information of each user in the set area; A target item position information determining module, configured to obtain the position information of the moved target item when it is detected that the target item in the set area is moved; The matching module is configured to match and obtain the identification information of the target user who moves the target item according to the position information of the target item and the real-time position information of each user in the set area. 9. The device according to claim 8, wherein the module for determining the real-time position of the user comprises: The first facial image acquisition submodule is used to acquire the facial images of each of the users entering the set area; The first identification information acquisition submodule is used to acquire the identification information of each user according to the facial image of each user; The real-time position information determination sub-module is used to track the human body of each user by means of pedestrian re-identification ReID according to the identification information of each user, so as to determine the real-time position information of each user in the set area. 10. The device according to claim 8, wherein the matching module comprises: The first distance determination submodule is used to compare the real-time location information of each user in the set area with the location information of the target item, and determine the distance between each user and the target item; The second identification information acquisition submodule is configured to acquire the identification information of the user with the closest distance to the target item. 11. The device according to claim 8, further comprising: An image acquisition module, configured to acquire the color image and depth image of each user collected by the depth image acquisition device; A human body key point extraction module, used to extract human body key points from the color images of each of the users; A hand position information determining module, configured to correspond the key points of the human body to the depth image, so as to determine the position information of each user's hand; The second distance determination module is used to compare the position information of each user's hand with the position information of the target item, and obtain the distance between each user's hand and the target item; The third identification information acquisition submodule is configured to acquire the identification information of the user whose hand is closest to the target item. 12. The device according to claim 8, further comprising: The second facial image acquisition module is used to acquire the facial image of the target user who moves the target item when the target item in the set area is detected to be moved; A facial recognition module, configured to perform facial recognition on the facial image of the target user by using the existing images of each user in the set area; The fourth identification information acquisition sub-module is used to acquire the identification information of the target user according to the recognition result of the facial recognition module; A judging module, configured to compare whether the identification information of the target user acquired through the fourth identification information acquisition sub-module is consistent with the identification information of the target user who moves the target item after being matched by the matching module. 13. The device of claim 8, further comprising: The trigger information receiving module is used to receive the trigger information of the gravity sensor, and the trigger information includes the first moment when the item is touched and the second moment when the item is taken away; A gravity sensing sensor position information acquisition module, configured to acquire target position information corresponding to the gravity sensing sensor; An item image search module, configured to search for the first item image corresponding to the first moment and the second item image corresponding to the second moment in the item images collected at the target location corresponding to the target location information; The target item determination module is configured to compare the difference between the first item image and the second item image to determine the target item taken away at the target position corresponding to the target position information. 14. The device of claim 8, further comprising: a weight acquisition module, configured to acquire the first weight of the gravity sensing sensor before the target item is taken away and the second weight after being taken away; a difference determination module, configured to obtain the difference between the first weight and the second weight; A standard weight inquiry module, used to inquire about the standard weight of the target item; A weight comparison module, configured to compare whether the difference is consistent with the standard weight of the target item. 15. An article state detection system, characterized in that, comprising: A server, comprising the object state detection device according to claims 8-14; A human body tracking device, configured to collect tracking images of users in a set area and transmit them to the server; The gravity sensing sensor is used to output sensing information to the server when it detects that the target item in the set area is moved. 16. The system according to claim 15, further comprising at least one of the following: A facial image acquisition device, configured to acquire a facial image of a target user who moves the target item; The depth image collection device is used to collect the depth images of each user. 17. The system of claim 16, further comprising: A core switch, communicatively connected to the server; A plurality of secondary switches are communicatively connected to the core switch, and each of the secondary switches is connected to a specified type of front-end equipment; Wherein, the secondary switch is used to control the data interaction between different front-end devices connected to itself, and the data interaction between the front-end device and the core switch; the front-end device includes a human body tracking device, a gravity sensor, a facial image One or more of an acquisition device and a depth image acquisition device. 18. An article state detection system, characterized in that, comprising: one or more processors; storage means for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the method according to any one of claims 1-7. 19. A computer-readable storage medium storing a computer program, characterized in that, when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.