CN108537094B

CN108537094B - Image processing method, device and system

Info

Publication number: CN108537094B
Application number: CN201710123786.XA
Authority: CN
Inventors: 廖可; 于海华; 王炜; 杨林举
Original assignee: Ricoh Co Ltd
Current assignee: Ricoh Co Ltd
Priority date: 2017-03-03
Filing date: 2017-03-03
Publication date: 2022-11-22
Anticipated expiration: 2037-03-03
Also published as: CN108537094A

Abstract

An image processing method, an image processing device and an image processing system are provided, wherein the method comprises the following steps: receiving inertial navigation data of a mobile device carried by a first moving body and a plurality of panoramic image frames shot by a panoramic camera in a synchronous time window; obtaining first information related to a moving track of a mobile device based on inertial navigation data of the mobile device carried by a first moving body, wherein the first information comprises moving amount and steering information of the moving track of the mobile device in a synchronous time window; obtaining second information related to the moving track of one or more moving bodies to be recognized detected in the panoramic image frame based on the panoramic image frame, wherein the second information comprises the moving amount and the steering information of the moving track of the one or more moving bodies to be recognized in a synchronous time window; the similarity of the first information and the second information is compared to match a moving object to be recognized in the panoramic image frame with the first moving object carrying the mobile device.

Description

Image processing method, device and system

Technical Field

The present disclosure relates to the field of image processing, and more particularly, to an image processing method, apparatus, and system for recognizing an identity of a moving body.

Background

Video surveillance is currently in widespread use. For example, the activities of persons in a specific area are monitored by a monitoring camera, and a specific person is recognized by face recognition or the like. However, the accuracy of the result of such recognition depends on the accuracy of the face recognition algorithm, and the variation in the orientation, shade, color, etc. of the face of a moving person may be large, and if the person faces away from the camera, or moves out of the camera and into the camera, this may result in an inaccurate recognition of a particular person, or a recognition of the same person as a different person.

Disclosure of Invention

According to an aspect of the present invention, there is provided an image processing method including the steps of: receiving inertial navigation data of a mobile device carried by a first moving body and a plurality of panoramic image frames shot by a panoramic camera in a synchronous time window; obtaining first information related to a movement track of a mobile device carried by the first moving body based on inertial navigation data of the mobile device, wherein the first information comprises a movement amount and steering information of the movement track of the mobile device in the synchronous time window; obtaining second information related to the moving track of one or more moving bodies to be recognized detected in the panoramic image frame based on the panoramic image frame, wherein the second information comprises the moving amount and steering information of the moving track of the one or more moving bodies to be recognized in the one synchronous time window; and comparing the similarity of the first information and the second information to match the moving body to be identified in the panoramic image frame with the first moving body carrying the mobile equipment.

According to another aspect of the present invention, there is provided a mobile device including: a data collector configured to gather inertial navigation data from a mobile device; a communicator configured to transmit the collected inertial navigation data to an image processing apparatus, wherein the image processing apparatus receives inertial navigation data of the mobile device carried by a first moving body and a plurality of panoramic image frames photographed by a panoramic camera in one synchronized time window; obtaining first information related to a movement track of the mobile device based on inertial navigation data of the mobile device carried by the first moving body, wherein the first information comprises movement amount and steering information of the movement track of the mobile device in the synchronous time window; obtaining second information related to the moving track of one or more moving bodies to be recognized detected in the panoramic image frame based on the panoramic image frame, wherein the second information comprises the moving amount and steering information of the moving track of the one or more moving bodies to be recognized in the one synchronous time window; and comparing the similarity of the first information and the second information to match the moving object to be identified in the panoramic image frame with the first moving object carrying the mobile equipment.

According to another aspect of the present invention, there is provided an image processing apparatus comprising: a receiver configured to receive inertial navigation data of a mobile device carried by a first moving body and a plurality of panoramic image frames photographed by a panoramic camera in one or more synchronized time windows; a first calculator configured to obtain first information related to a movement trajectory of a mobile device carried by the first mobile body based on inertial navigation data of the mobile device, the first information including a movement amount and steering information of the movement trajectory of the mobile device in the one synchronized time window; a second calculator configured to obtain, based on the panoramic image frame, second information related to a movement trajectory of one or more moving bodies to be recognized detected in the panoramic image frame, the second information including a movement amount and steering information of the movement trajectory of the one or more moving bodies to be recognized in the one synchronized time window; a comparator configured to compare the similarity of the first information and the second information to match a moving object to be recognized in the panoramic image frame with a first moving object carrying the mobile device.

According to another aspect of the present invention, there is provided an image processing system comprising: one or more processors; one or more memories having one or more computer-executable programs stored therein that, when executed by the one or more processors, perform the steps of: receiving inertial navigation data of a mobile device carried by a first moving body and a plurality of panoramic image frames shot by a panoramic camera in one or more synchronous time windows; obtaining first information related to a movement track of a mobile device carried by the first moving body based on inertial navigation data of the mobile device, wherein the first information comprises a movement amount and steering information of the movement track of the mobile device in the synchronous time window; obtaining second information related to the moving tracks of one or more moving bodies to be recognized detected in the panoramic image frame based on the panoramic image frame, wherein the second information comprises the moving amount and steering information of the moving tracks of the one or more moving bodies to be recognized in the synchronous time window; and comparing the similarity of the first information and the second information to match the moving body to be identified in the panoramic image frame with the first moving body carrying the mobile equipment.

Drawings

FIG. 1A illustrates an example scene diagram to which an image processing scheme according to an embodiment of the invention is applied. Fig. 1B shows an example of an image frame taken by a panoramic camera.

FIG. 2 illustrates an example hardware block diagram of an application of an image processing scheme in accordance with various embodiments of the present invention.

3A-3D illustrate example flow diagrams of image processing methods according to various embodiments of the invention.

Fig. 4 shows a detailed flowchart of an example of an image processing method according to an embodiment of the present invention.

Fig. 5 shows a synchronization process flow diagram of one example of an image processing method according to one embodiment of the present invention.

FIG. 6 shows a schematic diagram of a synchronized time window of an example of an image processing method according to an embodiment of the invention.

Fig. 7 shows a flowchart of steps of obtaining a movement trajectory based on a panoramic video stream including panoramic image frames according to one example of an image processing method according to one embodiment of the present invention.

Fig. 8 is a diagram showing an example of information (including step size and turn) about a movement trajectory obtained in the step shown in fig. 7.

Fig. 9 is a diagram illustrating calculation of an actual distance of a moving body from the center of an area in a panoramic image frame in the step shown in fig. 7.

Fig. 10 is a diagram showing the calculation of the shift amount from the distance shown in fig. 9.

Fig. 11 is a schematic view showing the determination of the turning of the mobile body in the step shown in fig. 7.

Fig. 12 is a flowchart showing a procedure of obtaining a movement trajectory based on inertial navigation data according to an example of an image processing method according to an embodiment of the present invention.

Fig. 13A and 13B are diagrams showing a steering judgment based on inertial navigation data in the step shown in fig. 12.

Fig. 14 shows a flowchart for filtering impossible movement trajectories and identification of a moving body in an image processing method according to an embodiment of the present invention.

Fig. 15 shows a schematic diagram of a comparison time window in an image processing method according to an embodiment of the invention.

Fig. 16 shows a block diagram of an image processing apparatus according to an embodiment of the present invention.

Fig. 17 shows a block diagram of a mobile device according to an embodiment of the invention.

FIG. 18 illustrates a block diagram of an exemplary image processing system suitable for use in implementing embodiments of the present invention.

Detailed Description

Reference will now be made in detail to the present embodiments of the invention, examples of which are illustrated in the accompanying drawings. While the invention will be described in conjunction with the specific embodiments, it will be understood that they are not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims. It should be noted that the method steps described herein may be implemented by any functional block or functional arrangement, and that any functional block or functional arrangement may be implemented as a physical entity or a logical entity, or a combination of both.

In order that those skilled in the art will better understand the present invention, the following detailed description of the invention is provided in conjunction with the accompanying drawings and the detailed description of the invention.

Note that the example to be presented next is only a specific example, and does not limit the embodiments of the present invention by the specific shapes, hardware, connections, steps, values, conditions, data, order, and the like, which are shown and described. Those skilled in the art can, upon reading this specification, utilize the concepts of the present invention to construct more embodiments than those specifically described herein.

Embodiments of the present invention primarily utilize a panoramic video stream (panoramic image frame sequence) captured by a panoramic camera (Theta 1, theta2 in fig. 1, a type of area sensor) deployed in the center of an area (such as a semicircular-like area in fig. 1) and inertial navigation data sensed by a mobile device (such as a user

mobile device client

1, 2, 3 in fig. 1A, a type of user sensor) carried by a mobile body (such as a user who is a person) for identity authentication of the mobile body. The field of view of a panoramic camera may for example cover its corresponding area, in which there may be a plurality of moving bodies carrying mobile devices or possibly other moving bodies (disturbing factors). For example, the moving body will be described later with a specific user example, but the moving body is not limited to a human being but may be other moving bodies such as an animal, a machine, and the like. Each user carries a mobile device having stored therein identity information identifying the user. The mobile device and the panoramic camera may communicate with each other in a wireless or wired network and send data to a server. The server can receive panoramic image frames from the panoramic camera and inertial navigation data from the mobile device in a wireless or wired mode, and the similarity of tracks from the panoramic video stream and the inertial navigation data is processed and compared through the algorithm according to the embodiments of the invention, so that the most similar tracks are subjected to user identity authentication and binding, and an identity information authentication result is obtained.

FIG. 2 illustrates an example hardware block diagram in which an image processing scheme according to various embodiments of the present invention may be applied.

The hardware block diagram shown in fig. 2 includes: 1) A data server U1 and a device client U2; 2) In the data server, there is an algorithm module U11 for processing panoramic video stream and inertial navigation data; the data center module U12 is used for data management; the communication module U13 is used for wireless or wired network communication, and receiving and sending data; 3) There are two different device clients, user sensor U22 and area sensor U21: the user sensor U22 may be a mobile device carried by a user, and the area sensor U21 may be a panoramic camera disposed at a ceiling. Each device client has two functional modules: data collection modules U221, U211 and communication modules U222, U212. The data collection modules U221, U211 are used for data sampling and preprocessing, and the communication modules U222, U212 are used for data transmission and reception.

Fig. 3A-3D illustrate an example flow diagram of an image processing method 30 in accordance with various embodiments of the present invention.

The image processing method 30 shown in fig. 3A includes: step S31, receiving inertial navigation data of a mobile device carried by a first moving body and a plurality of panoramic image frames shot by a panoramic camera in one or a plurality of synchronous time windows; step S32, obtaining first information related to the movement track of the mobile equipment based on inertial navigation data of the mobile equipment carried by the first moving body, wherein the first information comprises the movement amount and steering information of the movement track of the mobile equipment in the synchronous time window; a step S33 of obtaining, based on the panoramic image frame, second information related to a movement trajectory of one or more moving bodies to be recognized detected in the panoramic image frame, the second information including a movement amount and steering information of the movement trajectory of the one or more moving bodies to be recognized in the one synchronized time window; and step S34, comparing the similarity of the first information and the second information to match the moving object to be identified in the panoramic image frame with the first moving object carrying the mobile equipment.

Here, in one embodiment, the synchronization time window may be a time when the moving body, specifically, for example, the person takes one step, so that the moving trajectory of the person may be judged following the rule of taking one step.

In this way, the similarity of the tracks of the plurality of moving bodies to be identified detected in the panoramic video stream and the first moving body carrying the mobile device for generating the inertial navigation data can be processed and compared, and the moving body (for example, a user) identity authentication and binding are carried out on the closest track, so that an identity information authentication result is obtained. For example, if the degree of similarity between the trajectory of one moving body to be recognized among the plurality of trajectories of the plurality of moving bodies to be recognized detected in the panoramic video stream and the trajectory of a certain moving body carrying the mobile device that generates the inertial navigation data is the greatest, the moving body to be recognized may be considered to be the first moving body carrying the mobile device.

Note that the panoramic image frame mentioned in the present specification is not limited to the entirety of the panoramic image frame, but may include at least a part of the panoramic image frame, such as the entirety of the panoramic image frame (360 degree view angle), and a part of the panoramic image frame (e.g., 180 degree view angle, 90 degree view angle, arbitrary view angle, etc.). For example, although a 360-degree panoramic image frame is captured by a panoramic camera, attention may be paid only to a moving object at a certain area (angle of view), and therefore only the one area (angle of view) in the panoramic image frame may be extracted to perform the image processing mentioned in the present disclosure, so that costs such as the amount of computation and the amount of memory may be saved.

In one embodiment, as shown in fig. 3B, the step S32 of obtaining first information related to a movement trajectory of a mobile device carried by the first moving body based on inertial navigation data of the mobile device may include: step S321, obtaining Pedestrian Dead Reckoning (PDR) data based on the inertial navigation data, which indicates gait, step length, and steering information of a movement trajectory of the mobile device.

In this way, the gait, step length, and turning information of the movement trajectory of the mobile device can be obtained as the first information relating to the movement trajectory of the mobile device as the movement trajectory information of the first mobile body.

In one embodiment, as shown in fig. 3B, the step S321 of analyzing the behavior of the first moving body based on the inertial navigation data to obtain Pedestrian Dead Reckoning (PDR) data indicating gait, step length, and steering information of the moving trajectory of the mobile device includes: step S3211 of analyzing a gait of the first mobile body based on the inertial navigation data; s3212, filtering out a part of impossible acceleration values and gyroscope values to obtain alternative acceleration values and alternative gyroscope values based on the acceleration values, the gyroscope values, the acceleration value threshold, the gyroscope value threshold and the gait of the first moving body in the inertial navigation data; s3213, in a synchronous time window, obtaining a step length based on the candidate acceleration value and the speed at the beginning in the synchronous time window; s3214, filtering out impossible step sizes based on the step size threshold to obtain alternative movement amounts; and S3215, obtaining a rotation magnitude value as steering information based on the alternative gyroscope value.

In this way, gait, step size and steering information of the mobile device is obtained in particular.

In one embodiment, as shown in fig. 3C, in the case that there are a plurality of moving objects to be recognized, the step S33 of obtaining the second information related to the moving trajectory of the moving object to be recognized in the panoramic image frame based on the panoramic image frame may include: a step S331 of obtaining all coordinate positions of the detected plurality of moving bodies to be recognized in the plurality of panoramic image frames in the one synchronization time window; step S332, connecting all coordinate positions of the plurality of moving objects to be recognized in the two panoramic image frames at the beginning and the end in the one synchronous time window in the panoramic image frame to obtain all possible trajectories; step S333 of filtering out impossible trajectories from all the possible trajectories to obtain candidate trajectories based on a displacement threshold value of the moving body to be recognized within a predetermined time length.

Here, in a synchronized time window (e.g., one step), there may be multiple panoramic image frames, and the first panoramic image frame at the beginning and the last panoramic image frame at the end are taken for coordinate position detection and trajectory calculation to reduce the amount of calculation. Of course other two panoramic image frames may be computed. Generally, if the real-time performance of the system is high and the synchronization time window is small (e.g., within one step), it is possible to skip frames appropriately, even processing only the first frame and the last frame of each of a plurality of consecutive synchronization time windows; if the system is powerful, requires high precision, or the synchronization time window is large (e.g., multiple steps), then multiple panoramic image frames may be taken at intervals within a synchronization time window for computation. However, in the present disclosure, one step is generally used as the synchronization time window, and the general step is 1 second, and the amount of movement of a person within the synchronization time window as 1 second is not large, so that it is possible to directly calculate using two panoramic image frames at the start and end in one synchronization time window.

Here, it is possible to perform only the position recognition of the moving object to be recognized without performing the face recognition to obtain one or more positions of one or more moving objects to be recognized in each panoramic image frame, and to exhaust all possible moving trajectories of the one or more moving objects to be recognized by connecting the positions one by one in the time sequence of the plurality of panoramic image frames. Since the panoramic image frames are shot at known time intervals, the moving amount of all moving tracks between the time lengths of one step of the person can be compared with the length of one step of the person (as a displacement threshold value within a preset time length), and the track of which the moving amount between the time lengths of one step exceeds the displacement threshold value within the preset time length is filtered, so that the alternative track which is more in line with the walking rule of the person can be obtained.

Of course, only the displacement threshold for taking a step and the time length for taking a step of the person are exemplified here, however, the present solution is not limited thereto, and may also be other types of displacement thresholds and time lengths, such as a displacement threshold for a predetermined time length while the person is running, a displacement threshold for a predetermined time length during which a certain animal moves, and the like.

In one embodiment, as shown in fig. 3C, the step S33 of obtaining second information related to a moving trajectory of a moving body to be recognized in the panoramic image frame based on the panoramic image frame may further include: step S334 of obtaining actual distances of the plurality of moving objects to be recognized from the center of a panoramic image frame based on the panoramic image; step S335, obtaining rotation angles of the plurality of moving bodies to be identified relative to the panoramic camera based on the alternative tracks; step S336 of obtaining the movement amount alternatives of the plurality of moving bodies to be recognized based on the distances and the rotation angles.

Here, since the panoramic image frames are similar to photographs taken by a fisheye camera, the distance between two moving objects in the two panoramic image frames is not the distance that is actually moved. The fish eye imaging is characterized in that the closer to the center, the smaller the distortion is, and the more the image is consistent with the reality; the further away from the center, the greater the distortion, the less the image will be fitted to reality (e.g., the black arc on the left in fig. 1B, which is actually the office partition border line, should be straight, the image distorted to a curve). Therefore, in order to convert the distance represented in the panoramic image frame into the actual moving distance, the alternative moving amount between the two moving objects to be recognized identified in the two panoramic image frames may be calculated as the actual moving amount by the triangle geometry principle through the respective distances of the positions of the two moving objects to be recognized identified in the two panoramic image frames and the center of the panoramic image frame (representing, for example, the vicinity of the installation position of the panoramic camera), and the rotation angles of the two moving objects to be recognized with respect to the panoramic camera.

In one embodiment, as shown in fig. 3C, the step S33 of obtaining the second information related to the moving trajectory of the moving object to be recognized in the panoramic image frame based on the panoramic image frame may further include: step S337, judging the steering information of the moving body to be recognized based on the size difference of the actual distance between the moving body to be recognized and the center of the panoramic image frame in the two panoramic image frames at the beginning and the end of a synchronous time window; step S338, if the size difference is zero, determining steering information of the moving body to be recognized in the synchronized time window based on the size difference of the position coordinates of the moving body to be recognized in the two panoramic image frames at the beginning and the end of the synchronized time window.

This is a simple and efficient way of determining the steering information of the moving body. By a simple comparison judgment of once to twice, it is possible to obtain steering information to know whether the moving body moves in the clockwise direction or the counterclockwise direction, or whether the moving body moves leftward or rightward, or the like.

In one embodiment, as shown in fig. 3D, the step S34 of comparing the similarity of the first information and the second information to match the moving object to be recognized in the panoramic image frame with the first moving object carrying the mobile device may include: step S341, determining a comparison time window, the length of the comparison time window being a multiple of the length of the synchronization time window; step S342, in the comparison time window, comparing the movement amount in the first information with all the alternative movement amounts in the second information to obtain a distance similarity; step S343, in the comparison time window, comparing the steering information in the first information with the steering information in the second information to obtain steering similarity; and step S344, judging which of the one or more moving bodies to be identified is matched with the first moving body carrying the mobile equipment based on the distance similarity and the steering similarity.

In this way, the similarity of the movement amount and the steering information is compared, and it is possible to determine which of the moving bodies to be recognized matches the first moving body carrying the mobile device, thereby recognizing the moving body to be recognized photographed from the panoramic camera using information such as identification information identifying the first moving body stored in the mobile device. The Means for comparing similarity may include minimum variance, classification, and clustering algorithms, such as K Nearest Neighbor (KNN) and K Means (K-Means), etc., which are not described in detail herein.

In this way, the similarity of the trajectories of the plurality of moving bodies to be identified detected in the panoramic video stream and the first moving body carrying the mobile device for generating the inertial navigation data can be processed and compared, and the moving body (such as a user) identity authentication and binding are carried out on the closest trajectory, so that an identity information authentication result is obtained.

Details of a specific flow of an example of an image processing method according to an embodiment of the present invention are described below with reference to fig. 4 to 15, in which a user is used as an example of a moving body and a time length of one step of the user is used as an example of a synchronization time window, which is, of course, only an example and not a limitation, and the present invention is not limited thereto.

Fig. 4 shows a detailed flowchart of one example of an image processing method according to an embodiment of the present invention.

The work flow chart of one example of the image processing method comprises the following steps: 1) Step S1, initializing a platform of an image processing system (hereinafter referred to as a system); step S11 of obtaining a panoramic video stream including panoramic image frames taken by a panoramic camera, and step S12 of obtaining inertial navigation data from an inertial navigation sensor of a mobile device carried by a first person; 2) Step S211, when the system receives the binding trigger condition, the system starts motion detection to detect a moving person from the panoramic video stream, and step S212, based on the coordinate point (position) of the moving person, estimates trajectory information; 3) Step S221, simultaneously or in parallel, the system starts PDR analysis from the inertial navigation data to analyze the behavior of the person, and step S222, in each step (as an example of a synchronized time window), calculates step size (as an example of movement amount) and steering (information); 4) S3, in a comparison time window (equal to the multiple of the synchronization time window), the system compares the track information from the panoramic video stream and the track information from the mobile equipment, and filters out impossible tracks; 5) And S4, binding the identity information of the first person stored in the mobile equipment to a certain person detected with high track similarity based on the track similarity, and finishing the identity authentication process.

The similarity of the tracks of a plurality of persons with identities to be identified detected in the panoramic video stream and the first person carrying the mobile device for generating inertial navigation data can be processed and compared, the most similar (high-similarity) tracks are subjected to identity authentication and binding of the persons, and an identity information authentication result is obtained.

In addition, the binding trigger condition mentioned herein is a signal for starting a synchronization and recognition process, and may be determined based on proximity of a wireless signal, for example, determined by Bluetooth Low Energy (BLE) proximity, determined by video proximity determination processing, determined by PDR analysis proximity, and the like. In this manner, the mechanism for starting image processing according to the embodiment of the present invention based on the binding trigger condition, the image processing system does not need to be synchronized in the entire period of time, and power consumption can be reduced.

Fig. 5 shows a flowchart of a synchronization process with a binding trigger according to an example of an image processing method according to an embodiment of the present invention.

The synchronization process flow diagram includes: 1) The prerequisites are: s11, a panoramic video camera obtains a panoramic video stream comprising panoramic image frames, and S12, inertial navigation data of an inertial navigation sensor of mobile equipment carried by a first person is obtained; 2) Step S221-a, when a binding trigger condition (such as user approach, wireless approach judgment, video approach judgment and the like) comes, processing inertial navigation data, obtaining step number through PDR analysis, and further obtaining step length and steering in a step; and step S221-b, setting the synchronous time window as one step or several steps; 3) Step S211, identifying a person in a synchronized time window, and step S212 tracking the person in the synchronized time window; 4) Simultaneously or in parallel, step S222, tracking of the person is performed based on the PDR data.

The use of inertial navigation data to derive the number of steps and further derive the step size and steering in one step is well known to those skilled in the art and will not be described in detail herein.

In this manner, the mechanism for starting image processing according to the embodiment of the present invention based on the binding trigger condition, the image processing system does not need to be synchronized in the entire period of time, and power consumption can be reduced.

Here, a human step is taken as an example of the synchronized time window, and thus 3 frames of panoramic image frames in the panoramic video stream data and inertial navigation data i2 to i6 may be collected in one synchronized time window between "new step" and another "new step" as shown in fig. 6.

And the length of the synchronization time window as a step can be obtained in various ways. For example, the time length of one step is obtained as the synchronized time window by performing PDR analysis on the inertial navigation data to detect the number of steps, or the time length of one step is obtained as the synchronized time window from the step sensing data of the step sensor, or the predetermined time length for which a person normally takes one step may also be set in advance as the synchronized time window.

Here, the track processing of the panoramic image frame of the panoramic video stream and the track processing of the inertial navigation data are synchronized by using the synchronization time window, so that the start time point and the end time point of the similarity of the comparison tracks are the same, and the similarity of the tracks can be more accurately compared, thereby more accurately judging whether the two tracks are the same or similar.

The flow chart for obtaining a trajectory based on a panoramic video stream includes: 1) Step S212-a, connecting the position coordinate points of the detected people in the panoramic image frame in a synchronous time window according to the position coordinate points in time sequence, thereby obtaining all possible path tracks of all people; 2) Step S212-b, in each synchronous time window, assuming that the person moves in one step, obtaining the starting point and the final point of each moving path of the person in each step, and filtering out the path with the length longer than the threshold value of the length of one step of the common person so as to reduce the calculated amount; 3) Step S212-c, calculating the actual distance between the detected person and the center of the area (usually the point where the installation position of the panoramic camera is projected to the ground, for example, by the distance between the installation position point of the panoramic camera and the projected point, namely, the installation height of the panoramic camera); 4) Step S212-d, based on the results of step S212-c, calculates information related to the trajectory, including the actual step size (as an example of the amount of movement) and the steering.

Here, the information related to the movement trajectory is obtained from the trajectory processing of the panoramic video stream, and is composed of the following steps, as shown in fig. 8:

c-1) obtaining all position coordinate points of all detected persons in each synchronous time window;

c-2) waiting for all possible trajectories based on these position coordinate points, the number of trajectories being as in equation (1):

NumPath＝∏length(pt _i ) (1)

wherein pt is _i All the position coordinate points representing all the detected persons.

c-3) filtering out the repeated tracks in each synchronization time window with the first and last points as track paths to obtain possible tracks, for example, all possible tracks from two points 2 (pt21. X, pt21.Y, pt22.X, pt22.Y, respectively) of the 2 nd synchronization time window to one point 3 (pt3. X, pt3. Y) of the 3 rd synchronization time window to three points 4 (pt41. X., pt41. Y), [ 42.x., pt42.Y ], [ pt43.X., pt43.y ]) of the 4 th synchronization time window (each line listed at the lower side of fig. 8 represents one possible track) on the left side of fig. 8;

c-4) filtering tracks with large jump (namely tracks with the length of track paths larger than the step length threshold of one step of a common person) between different synchronous time windows by combining the possible tracks obtained in the step c-3 to obtain alternative tracks;

c-5) calculating an actual distance of a person in the area from the center of the area (panoramic camera position) based on the characteristics of the panoramic image, as shown in equation (2):

where H is the height of the panoramic camera from the ground, H is the height of a person (an average value, such as 1.7m, can be estimated in the case of multiple persons), p is the size of a pixel of the image sensor, and f is the focal length of the fisheye lens of the panoramic camera.

c-6) calculating the step size and the turn of the person in the synchronized time window based on the result of c-5.

Fig. 9 is a schematic diagram illustrating calculation of an actual distance of a moving object from the center of the area in the panoramic image frame in the step shown in fig. 7.

Referring to fig. 9, the intersection point of the optical axis of the panoramic camera, which is usually located at the ceiling, and the ground can be used as the area center, and assuming that the pixel size of the detected person is p, the actual height of the person is h, the pixel distance between the position coordinate point of the detected person and the area center is r, the pixel height of the position coordinate point is n, and the pixel width is m.

As such, the actual distance d of the coordinate position of the person from the center of the area can be calculated by the following formula.

Fig. 10 is a diagram showing calculation of an actual moving amount of a person from the distance shown in fig. 9.

The actual movement amount of the person is referred to herein as a step size within a certain synchronization time window, and can be calculated by equation (3), as shown in fig. 10.

d ₁ ² +d ₂ ² +2×d ₁ ×d ₂ ×cos α ₁ ＝SL ² (3)

Here, d ₁ Is the actual distance, d, of the detected person from the center of the region in the previous panoramic image frame ₂ SL, which is the actual distance of the detected person from the center of the area in the next succeeding panoramic image frame, represents the actual amount of movement of the person from the preceding panoramic image frame to the succeeding panoramic image frame, α ₁ Representing a rotation angle of the detected person relative to the panoramic camera from a previous panoramic image frame to a subsequent panoramic image frame. x is the number of ₁ 、y ₁ And x ₂ 、y ₂ Respectively the coordinates of the detected person on the panoramic image frames in the two panoramic image frames within the same synchronized time window. Since the panoramic image frame is distorted in terms of distance and movement, but the panoramic image corresponds to the angle, especially the angle based on the center of the area, the angle α on the panoramic image frame is used here ₁ As is the actual included angle.

Steering refers to the direction of rotation within a certain synchronization time window, and can be obtained by the following steps:

c-6-1) obtaining historical rotation direction;

c-6-2) comparing the actual distance of the detected person from the center of the area in the current synchronization time window (comparing the actual distance of the first detected coordinate point of the synchronization actual window from the center of the area with the actual distance of the last detected coordinate point of the synchronization actual window from the center of the area), and judging a possible rotation mode based on the comparison result of the actual distances;

c-6-3) comparing the detected position coordinates of the person in the current synchronous time window, and judging a possible rotation mode based on the coordinate comparison;

c-6-4) obtaining the steering of each track based on the judgment results of the first two items.

Specifically, the specific process of comparison is explained in detail with reference to fig. 11.

In the steering calculation, the known quantities are: coordinate points (xi, yi) of the detected person in each panoramic image frame, the actual distance di of the detected person from the center of the area, where i is the number of the panoramic image frame.

The following steps are carried out:

through di comparison between panoramic image frames, first determine whether a person is far from center or close to center:

for example, if d2 is greater than d1, indicating that the actual distance of the person from the center in the second panoramic image frame is greater than the actual distance of the person from the center in the first panoramic image frame, the person may be considered to be off center.

Then the range represented by the thin solid line segment below and to the left of the possible range of motion for the person.

2, more specifically judging the possible motion range of the human through the comparison of xi and yi:

for example, if x2 is greater than x1 and y1 is greater than y2, then the range of possible motion for the person is that represented by the thick solid line segment below.

3. A possible turn is determined by a historical value (x 0, y 0), i.e. the coordinate values of the detected person in the previous synchronization time window, or a future value (x 3, y 3), e.g. the coordinate values of the detected person in the following synchronization time window.

For example, if X0< X1, Y0> Y1 or X3< X2, Y3< Y2, it can be determined that the person is rotating clockwise, as indicated by the arrow in fig. 11.

4, calculating the amplitude (specific angle value) of the human rotation through xi, yi. The magnitude or angle value is calculated from coordinates on the image. The distortion brought by the fish-eye image can cause certain errors, but because the rough range of rotation is determined according to the first two qualitative comparisons, and the moving range of people in a short time is limited, the actual steering angle can be roughly estimated by using the steering angle in the image.

First, the (X0, y 0) and (X1, y 1) linear equations are performed:

next, the (X2, y 2) and (X1, y 1) circular equations are performed:

(x′-x ₁ ) ² +(y′-y ₁ ) ² ＝(x ₂ -x ₁ ) ² +(y ₂ -y ₁ ) ² (6)

then, the solution of the two is added with the orientation judgment to obtain (x ', y').

Then, the (X ', y'), (X2, y 2) and (X1, y 1) trigonometric equations are solved, wherein the steering angle α is obtained:

note that in a system with low precision requirement and requiring quick response, under the condition that the rough judgment of the first three steps can meet the requirement, the 4 th step of calculation is not needed, or a steering angle can be directly estimated through first verification in the 4 th step, so as to achieve the real-time effect of quick response.

The trajectory processing flow chart based on inertial navigation data comprises the following steps: 1) Step S221-a, obtaining inertial navigation data: acceleration, gyroscopes and magnetic sensors; 2) Step S221-b, based on the inertial navigation data, the PDR analysis process obtains further information: step number, step length, steering; 3) Step S222-a, based on user behaviors, an inverse function relation exists between the acceleration value and the steering value, and based on the relation, unreasonable inertial navigation data are filtered; 4) In each step time window, step S222-b, the step size is obtained, and step S222-c, the turn is obtained.

The PDR data is based on inertial navigation data of the mobile device and is derived in conjunction with PDR analysis. The inertial navigation data comprises result data of an accelerometer, a gyroscope and a magnetic sensor. The PDR analysis includes gait detection, step size analysis, and steering analysis.

The amount of movement is the step size of the person within a certain synchronization time window, obtained by the following steps, as shown in fig. 12:

d-1) analyzing user behaviors based on basic inertial navigation data to obtain PDR data;

d-2) filtering to a portion of the acceleration values and gyroscope values (as rotation vectors) based on a relationship between the acceleration values, gyroscope values, and user behavior;

d-3) in the synchronous time window, obtaining a step length SL according to a formula (8), and judging the feasibility of the step length according to the user behavior:

where a denotes the acceleration value, v _0 denotes the initial velocity value at the beginning of the synchronization time window, and t denotes the time length of the synchronization time window. The initial velocity value can be obtained by, for example, three methods: one is a priori authentication presetting, and the walking speed of a person is generally 1-1.5m/s; the other is to keep the final speed at the previous step (or the previous synchronization time window) and then use the speed as the initial speed of the next step (or the next synchronization time window); the last is to consider a short stay for each step of the person, with an initial velocity of 0. The first, more accurate, and less error can be used here.

Steering within a certain synchronization time window, usually referred to as a rotation vector, is obtained by the following steps, as shown in fig. 13:

d-2) filtering to a portion of the acceleration values and gyroscope values (rotation vectors) based on a relationship between the acceleration values, gyroscope values, and user behavior;

d-3) within this synchronization time window, the rotation vector values are obtained, as shown in FIG. 13A.

Specifically, in the step of obtaining the steering, a steering rotation amount rotation vector (rv) can be obtained by the value of the gyroscope in the inertial navigation data as steering information, which is an arctangent value of an angle, ranging from-1 to 1. If only the gyroscope value is output in the inertial navigation data, the related software module can be called to generate a similar steering vector through the traditional PDR algorithm. The specific solving method can solve the inertial navigation data through a quaternion equation:

where θ represents the rotation angle about the y-axis (relative to the specified initial direction at system initialization), Δ θ represents the rotation angle between two inertial navigation data samples, (x, y, z, w) represents the quaternion for the corresponding rotation,

psi denotes the rotation angles around the x-axis and y-axis, respectively (as shown in fig. 13B).

In this way, the step length and the steering information related to the moving track are obtained through the inertial navigation data.

Note that although some specific formulas are illustrated above, these formulas are merely examples, and those skilled in the art may construct other formulas according to the principles of the present application, which are not illustrated herein.

Fig. 14 shows a flowchart of filtering impossible movement trajectories and identification of a moving body in an image processing method according to an embodiment of the present invention. Fig. 15 shows a schematic diagram of a comparison time window in an image processing method according to an embodiment of the invention.

The process of identity recognition and identity binding by comparing the movement tracks includes the following steps, as shown in fig. 14:

e-1) step S411, obtaining the movement amount (here, step size) and the turn information from the detection result from the panoramic video stream within each synchronization time window;

e-2) step S421, obtaining the movement amount (step size here) and steering information from inertial navigation data and PDR analysis in each synchronous time window;

e-3) determining comparison time windows, which are multiples of the synchronization time windows, as shown in FIG. 15;

e-4-1) step S412, comparing the motion amount information from the panoramic video stream and the inertial navigation data in the comparison time window to obtain the similarity, as shown in formula (11), defined as filter a:

∑Sim(dpi,dti) (11)

where dpi denotes a moving amount obtained based on the inertial navigation data, and dti denotes a moving amount obtained based on the panoramic video stream. The filter a filters out dissimilar shift amounts, e.g., the variance of two shift amounts is large, etc. For example, similarity sim is previously determined, and similarity is generally a value less than 1, with closer to 1 representing more similarity. Therefore, the filtering can be directly removing the minimum value of sim, or setting a certain threshold range and filtering the value smaller than the threshold.

e-4-2) step S422, in the comparison time window, comparing the steering information from the panoramic video stream and the inertial navigation data to obtain the similarity, as shown in formula (12), defined as filter B:

∑Sim(tp′i,tti) (12)

where tp' i represents steering information derived based on inertial navigation data, and tti represents steering information derived based on a panoramic video stream. The filter B filters out the dissimilar steering information, for example, the variance of two steering information is large, etc. For example, similarity sim is previously determined, and similarity is generally a value less than 1, with closer to 1 representing more similarity. Therefore, the filtering can be directly removing the minimum value of sim, or setting a certain threshold range and filtering the value smaller than the threshold.

Thus, in step S43, in conjunction with filters A and B, impossible tracks are filtered out.

e-4-3) step S44, filtering out impossible tracks by combining historical information of historical comparison time windows, and defining as a filter C:

in combination with the historical window, the synchronization time window is expanded, that is, the length of the comparison time window of the comparison track is expanded, for example, the comparison in 1s is changed into the comparison in 3s, and the comparison track is also increased. Filtering is then performed based on the comparison of the shift amount and the steering, again based on filter a and filter B as filter C.

e-5) finally, step S45, filtering out all impossible trajectories through filter a, filter B, filter C, obtaining the most likely trajectory and the detected persons corresponding to the two trajectories, so as to identify the persons detected in the panoramic video stream as the identity of the persons corresponding to the inertial navigation data of the mobile device.

Of course, the above mentioned filters a, B, C are only examples, and virtually any way of filtering out impossible tracks is feasible and included within the scope of the present application.

Fig. 16 shows a block diagram of an image processing apparatus 1600 according to an embodiment of the invention.

The image processing apparatus 1600 includes: a receiver 1601 configured to receive inertial navigation data of a mobile device carried by a first moving body and a plurality of panoramic image frames photographed by a panoramic camera in one or more synchronized time windows; a first calculator 1602 configured to obtain first information related to a movement trajectory of a mobile device carried by the first mobile body based on inertial navigation data of the mobile device, the first information including a movement amount and steering information of the movement trajectory of the mobile device in the one synchronized time window; a second calculator 1603 configured to obtain, based on the panoramic image frame, second information related to a movement trajectory of one or more to-be-identified moving bodies detected in the panoramic image frame, the second information including a movement amount and steering information of the movement trajectory of the one or more to-be-identified moving bodies in the one synchronized time window; a comparator 1604 configured to compare the similarity of the first information and the second information to match a moving object to be recognized in the panoramic image frame with the first moving object carrying the mobile device.

In one embodiment, the first calculator 1602 may be configured to: obtaining Pedestrian Dead Reckoning (PDR) data indicating gait, step size, and steering information of a movement trajectory of the mobile device based on the inertial navigation data.

In one embodiment, the first calculator 1602 may be configured to: analyzing a gait of a first mobile body based on the inertial navigation data; filtering out a part of impossible acceleration values and gyroscope values to obtain alternative acceleration values and alternative gyroscope values based on the acceleration values, the gyroscope values, the acceleration value threshold values, the gyroscope value threshold values and the gait of the first mobile body in the inertial navigation data; obtaining a step size based on the candidate acceleration value and the velocity at the beginning in a synchronization time window; filtering out impossible step sizes based on a step size threshold to obtain alternative movement amounts; obtaining a rotation direction magnitude value as the direction information based on the alternative gyroscope value.

In one embodiment, the second calculator 1603 may be configured to: obtaining all coordinate positions of the detected moving bodies to be identified in the panoramic image frames in the synchronous time window; connecting all coordinate positions of the plurality of moving bodies to be recognized in the panoramic image frames at the beginning and the end in the synchronous time window to obtain all possible tracks; and filtering out impossible tracks from all the possible tracks to obtain alternative tracks based on a displacement threshold value of the moving body to be recognized within a preset time length.

In one embodiment, the second calculator 1603 may be configured to: the step of obtaining second information related to a movement locus of a moving body to be recognized in the panoramic image frame based on the panoramic image frame further includes: obtaining actual distances of the plurality of moving bodies to be recognized from the center of a panoramic image frame based on the panoramic image; obtaining rotation angles of the plurality of moving bodies to be identified relative to the panoramic camera based on the alternative tracks; based on the distances and the rotation angles, movement amount candidates of the plurality of moving bodies to be recognized are obtained.

In one embodiment, the second calculator 1603 may be further configured to: judging the steering information of the moving body to be recognized based on the size difference of the actual distance between the moving body to be recognized and the center of the panoramic image frame in the two panoramic image frames at the beginning and the end of a synchronous time window; and if the size difference is zero, judging the steering information of the moving body to be recognized in the synchronous time window based on the size difference of the position coordinates of the moving body to be recognized in the two panoramic image frames at the beginning and the end of the synchronous time window.

In one embodiment, the comparator 1604 may be configured to: determining a comparison time window having a length that is a multiple of a length of the synchronization time window; in the comparison time window, comparing the movement amount in the first information with all the alternative movement amounts in the second information to obtain distance similarity; in the comparison time window, comparing the steering information in the first information with the steering information in the second information to obtain steering similarity; and judging which of the one or more moving bodies to be identified is matched with a first moving body carrying the mobile equipment based on the distance similarity and the steering similarity.

In this way, the similarity of the tracks of the plurality of moving bodies to be identified detected in the panoramic video stream and the first moving body carrying the mobile device for generating the inertial navigation data can be processed and compared, and the moving body (for example, a user) identity authentication and binding are carried out on the closest track, so that an identity information authentication result is obtained. For example, if the degree of similarity between the trajectory of one moving body to be recognized among the trajectories of a plurality of moving bodies to be recognized detected from the panoramic video stream and the trajectory of a certain moving body carrying the mobile device that generates inertial navigation data is the greatest, the moving body to be recognized may be regarded as the first moving body carrying the mobile device.

A mobile device 1700 shown in fig. 17 includes: a data collector 1701 configured to gather inertial navigation data from a mobile device; a communicator 1702 configured to transmit the collected inertial navigation data to an image processing apparatus 1600, wherein the image processing apparatus 1600 receives inertial navigation data of the mobile device carried by a first moving body and a plurality of panoramic image frames photographed by a panoramic camera in one synchronized time window; obtaining first information related to a movement track of the mobile device based on inertial navigation data of the mobile device carried by the first moving body, wherein the first information comprises movement amount and steering information of the movement track of the mobile device in the synchronous time window; obtaining second information related to the moving track of one or more moving bodies to be recognized detected in the panoramic image frame based on the panoramic image frame, wherein the second information comprises the moving amount and steering information of the moving track of the one or more moving bodies to be recognized in the one synchronous time window; and comparing the similarity of the first information and the second information to match the moving object to be identified in the panoramic image frame with the first moving object carrying the mobile equipment.

Here, the image processing apparatus 1600 may also have the structure described with reference to fig. 17, which is not described herein again.

The image processing system may include a processor (H1); a memory (H2) coupled to the processor (H1) and having stored therein computer-executable instructions for, when executed by the processor, performing the steps of the method as described herein with various embodiments.

The processor (H1) may include, but is not limited to, for example, one or more processors or microprocessors or the like.

The memory (H2) may include, but is not limited to, for example, random Access Memory (RAM), read Only Memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, a hard disk, a floppy disk, a solid state disk, a removable disk, a CD-ROM, a DVD-ROM, a Blu-ray disk, and the like.

The computer system may include, among other things, a data bus (H3), an input/output (I/O) bus (H4), a display (H5), and an input/output device (H6) (e.g., keyboard, mouse, speakers, etc.).

The processor (H1) may communicate with external devices (H5, H6, etc.) via a wired or wireless network (not shown) through the I/O bus (H4).

The memory (H2) may also store at least one computer executable instruction for the respective functions and/or steps of the method in the embodiments described in the present technology when executed by the processor (H1).

Of course, the above-mentioned embodiments are merely examples and not limitations, and those skilled in the art can combine and combine some steps and apparatuses from the above-mentioned separately described embodiments to achieve the effects of the present invention according to the concepts of the present invention, and such combined and combined embodiments are also included in the present invention, and such combined and combined embodiments are not necessarily described herein.

It is noted that advantages, effects, and the like, which are mentioned in the present disclosure, are only examples and not limitations, and they are not to be considered essential to various embodiments of the present invention. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the invention is not limited to the specific details described above.

The block diagrams of devices, apparatuses, devices, systems involved in the present disclosure are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by one skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably herein. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

The flowchart of steps in the present disclosure and the above description of methods are merely illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order presented. As will be appreciated by those of skill in the art, the order of the steps in the above embodiments may be performed in any order. Words such as "thereafter," "then," "next," etc. are not intended to limit the order of the steps; these words are simply used to guide the reader through the description of the methods. Furthermore, any reference to an element in the singular, for example, using the articles "a," "an," or "the" is not to be construed as limiting the element to the singular.

In addition, the steps and devices in the embodiments are not limited to be implemented in a certain embodiment, and in fact, some steps and devices related to the embodiments may be combined according to the concept of the present invention to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

The individual operations of the methods described above can be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and/or software components and/or modules including, but not limited to, a hardware circuit, an Application Specific Integrated Circuit (ASIC), or a processor.

The various illustrative logical blocks, modules, and circuits described may be implemented or described with a general purpose processor, a Digital Signal Processor (DSP), an ASIC, a field programmable gate array signal (FPGA) or other Programmable Logic Device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The steps of a method or algorithm described in connection with the disclosure herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may reside in any form of tangible storage medium. Some examples of storage media that may be used include Random Access Memory (RAM), read Only Memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, and the like. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. A software module may be a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media.

The methods disclosed herein comprise one or more acts for implementing the described methods. The methods and/or acts may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and/or use of specific actions may be modified without departing from the scope of the claims.

The above-described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions on a tangible computer-readable medium. A storage media may be any available tangible media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disk (disk) and disc (disc) includes Compact Disc (CD), laser disc, optical disc, digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

Accordingly, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions stored (and/or encoded) thereon that are executable by one or more processors to perform the operations described herein. The computer program product may include packaged material.

Software or instructions may also be transmitted over a transmission medium. For example, the software may be transmitted from a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, or microwave.

Further, modules and/or other suitable means for carrying out the methods and techniques described herein may be downloaded and/or otherwise obtained by a user terminal and/or base station as appropriate. For example, such a device may be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, the various methods described herein can be provided via storage means (e.g., RAM, ROM, a physical storage medium such as a CD or floppy disk) so that the user terminal and/or base station can obtain the various methods when coupled to or providing storage means to the device. Moreover, any other suitable technique for providing the methods and techniques described herein to a device may be utilized.

Other examples and implementations are within the scope and spirit of the disclosure and appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard-wired, or any combination of these. Features implementing functions may also be physically located at various locations, including being distributed such that portions of functions are implemented at different physical locations. Also, as used herein, including in the claims, "or" as used in a list of items beginning with "at least one" indicates a separate list, such that, for example, a list of "at least one of a, B, or C" means a or B or C, or AB or AC or BC, or ABC (i.e., a and B and C). Furthermore, the word "exemplary" does not mean that the described example is preferred or better than other examples.

Various changes, substitutions, and alterations to the techniques described herein may be made without departing from the techniques of the teachings as defined by the appended claims. Moreover, the scope of the claims of the present disclosure is not limited to the particular aspects of the process, machine, manufacture, composition of matter, means, methods and acts described above. Processes, machines, manufacture, compositions of matter, means, methods, or acts, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein may be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or acts.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the invention to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims

1. An image processing method comprising the steps of:

receiving inertial navigation data of a mobile device carried by a first moving body and a plurality of panoramic image frames shot by a panoramic camera in a synchronous time window;

obtaining first information related to a moving track of the mobile device based on inertial navigation data of the mobile device carried by the first moving body, wherein the first information comprises moving amount and steering information of the moving track of the mobile device in one synchronous time window, and steering is a rotating direction in one synchronous time window;

obtaining second information related to the moving track of one or more moving bodies to be recognized detected in the panoramic image frame based on the panoramic image frame, wherein the second information comprises the moving amount and steering information of the moving track of the one or more moving bodies to be recognized in the one synchronous time window;

comparing the similarity of the first information and the second information to match a moving object to be identified in a panoramic image frame with a first moving object carrying the mobile device;

wherein the step of obtaining second information related to a movement locus of a moving body to be recognized in the panoramic image frame based on the panoramic image frame further comprises: judging the steering information of the moving body to be recognized based on the size difference of the actual distance between the moving body to be recognized and the center of the panoramic image frame in the two panoramic image frames at the beginning and the end of a synchronous time window; and if the size difference is zero, judging the steering information of the moving body to be recognized in the synchronous time window based on the size difference of the position coordinates of the moving body to be recognized in the two panoramic image frames at the beginning and the end of the synchronous time window.

2. The method according to claim 1, wherein the step of obtaining first information related to a movement trajectory of a mobile device carried by the first moving body based on inertial navigation data of the mobile device comprises:

obtaining Pedestrian Dead Reckoning (PDR) data indicative of gait, step size, and steering information of a movement trajectory of the mobile device based on the inertial navigation data.

3. The method of claim 2, wherein analyzing behavior of the first mobile body based on the inertial navigation data to obtain Pedestrian Dead Reckoning (PDR) data indicative of gait, step size and steering information of a movement trajectory of the mobile device comprises:

analyzing a gait of a first mobile body based on the inertial navigation data;

filtering out a part of impossible acceleration values and gyroscope values to obtain alternative acceleration values and alternative gyroscope values based on the acceleration values, the gyroscope values, the acceleration value threshold values, the gyroscope value threshold values and the gait of the first mobile body in the inertial navigation data;

obtaining a step size based on the candidate acceleration value and the velocity at the beginning in a synchronization time window;

filtering out impossible step sizes based on a step size threshold to obtain alternative movement amounts;

obtaining a rotation direction magnitude value as the direction information based on the alternative gyroscope value.

4. The method according to claim 1, wherein, in a case where there are a plurality of moving bodies to be identified, the step of obtaining second information relating to a trajectory of movement of the moving body to be identified in the panoramic image frame based on the panoramic image frame includes:

obtaining all coordinate positions of the detected moving bodies to be identified in the panoramic image frames in the synchronous time window;

connecting all coordinate positions of the plurality of moving bodies to be recognized in the two panoramic image frames at the beginning and the end in the synchronous time window to obtain all possible tracks;

and filtering out impossible tracks from all the possible tracks to obtain alternative tracks based on a displacement threshold value of the moving body to be recognized within a preset time length.

5. The method according to claim 4, wherein the step of obtaining second information related to a movement locus of a moving body to be recognized in the panoramic image frame based on the panoramic image frame further comprises:

obtaining actual distances of the plurality of moving bodies to be recognized from the center of a panoramic image frame based on the panoramic image;

obtaining rotation angles of the plurality of moving bodies to be identified relative to the panoramic camera based on the alternative tracks;

based on the distances and the rotation angles, movement amount candidates of the plurality of moving bodies to be recognized are obtained.

6. The method of claim 1, wherein the step of comparing the similarity of the first and second information to match the moving object to be identified in the panoramic image frame with the first moving object carrying the mobile device comprises:

determining a comparison time window, the length of the comparison time window being a multiple of the length of the synchronization time window;

in the comparison time window, comparing the movement amount in the first information with all the alternative movement amounts in the second information to obtain distance similarity;

in the comparison time window, comparing the steering information in the first information with the steering information in the second information to obtain steering similarity;

and judging which of the one or more moving bodies to be identified is matched with a first moving body carrying the mobile equipment based on the distance similarity and the steering similarity.

7. A mobile device, comprising:

a data collector configured to gather inertial navigation data from a mobile device;

a communicator configured to transmit the collected inertial navigation data to an image processing apparatus,

the image processing device receives inertial navigation data of the mobile equipment carried by a first moving body and a plurality of panoramic image frames shot by a panoramic camera in a synchronous time window; obtaining first information related to a moving track of the mobile device based on inertial navigation data of the mobile device carried by the first moving body, wherein the first information comprises moving amount and steering information of the moving track of the mobile device in one synchronous time window, and steering is a rotating direction in one synchronous time window; obtaining second information related to the moving track of one or more moving bodies to be recognized detected in the panoramic image frame based on the panoramic image frame, wherein the second information comprises the moving amount and steering information of the moving track of the one or more moving bodies to be recognized in the one synchronous time window; comparing the similarity of the first information and the second information to match a moving body to be identified in a panoramic image frame with a first moving body carrying the mobile device;

8. An image processing apparatus comprising:

a receiver configured to receive inertial navigation data of a mobile device carried by a first moving body and a plurality of panoramic image frames photographed by a panoramic camera in one or more synchronized time windows;

a first calculator configured to obtain first information related to a movement trajectory of a mobile device carried by the first mobile body based on inertial navigation data of the mobile device, the first information including a movement amount of the movement trajectory of the mobile device in the one synchronous time window and steering information, where steering is a rotation direction within the one synchronous time window;

a second calculator configured to obtain second information related to a movement trajectory of one or more to-be-recognized moving bodies detected in the panoramic image frame based on the panoramic image frame, the second information including a movement amount and steering information of the movement trajectory of the one or more to-be-recognized moving bodies in the one synchronized time window;

a comparator configured to compare similarities of the first information and the second information to match a moving object to be identified in a panoramic image frame with a first moving object carrying the mobile device;

wherein the second calculator judges the turn information of the moving body to be recognized based on a difference in size between two panoramic image frames at the start and end of one synchronized time window of an actual distance of the moving body to be recognized from the center of the panoramic image frames; and if the size difference is zero, judging the steering information of the moving body to be recognized in the synchronous time window based on the size difference of the position coordinates of the moving body to be recognized in the two panoramic image frames at the beginning and the end of the synchronous time window.

9. An image processing system comprising:

one or more processors;

one or more memories having one or more computer-executable programs stored therein that, when executed by the one or more processors, perform the steps of:

receiving inertial navigation data of a mobile device carried by a first moving body and a plurality of panoramic image frames shot by a panoramic camera in one or more synchronous time windows;

comparing the similarity of the first information and the second information to match a moving body to be identified in a panoramic image frame with a first moving body carrying the mobile device;

wherein the step of obtaining second information related to a movement trajectory of a moving body to be recognized in the panoramic image frame based on the panoramic image frame further comprises: judging the steering information of the moving body to be recognized based on the size difference of the actual distance between the moving body to be recognized and the center of the panoramic image frame in the two panoramic image frames at the beginning and the end of a synchronous time window; and if the size difference is zero, judging the steering information of the moving body to be recognized in the synchronous time window based on the size difference of the position coordinates of the moving body to be recognized in the two panoramic image frames at the beginning and the end of the synchronous time window.