HK40009591A - Co-localisation - Google Patents

Description

Joint positioning

Technical Field

The present invention relates to a method for co-localization (co-localization ) of multiple devices. More specifically, the present invention relates to the use of both local and remote resources to provide individual (individual, own, personal) devices with substantially real-time capabilities to determine the relative position and/or pose of the or each individual device and other devices in three-dimensional space.

Background

Robust capabilities of devices to determine relative positions or poses of other devices within three-dimensional space, for example with accuracy to within one centimeter and with substantially real-time speed (e.g., at a rate of 30 times per second), may be critical for many robotic, augmented reality, and virtual reality applications.

For example, accuracy on a substantially centimeter level enables multi-device augmented reality applications in which each device knows the corresponding three-dimensional locations of the other devices. With this accuracy, the application can display augmented content at the relevant pixel locations, enabling a substantially seamless augmented reality user experience. This capability may also allow multiple robots, such as autonomous cars, to know and collaborate with each other's location, for example, in traffic situations.

This level of accuracy cannot be achieved by current Global Navigation Satellite System (GNSS) technology, especially in challenging environments. This is generally because GNSS is hindered by difficulties presented by environmental conditions experienced in at least some, if not most, locations (atmospheric effects, signals reflected by buildings in dense urban environments, visibility of the sky, etc.). In particular, obtaining satellite reference locations within buildings and dense urban environments is often very inaccurate (i.e., there is a potential error of about 10 meters from the true device location).

Furthermore, in order for a positioning function with the above-mentioned level of accuracy to be usable in robotic and augmented reality applications, the positioning function must be sufficiently robust. In order for the positioning function to be robust enough, the positioning function should continue to operate under all weather conditions, in varying environments of large scale indoors and outdoors, and substantially in real time.

Classical single-robot and multi-robot SLAM (simultaneous localization and mapping) solutions are not robust enough and are generally not suitable for large-scale use and for situations where the degree of environmental changes is observed in the real world. Thus, these systems are not designed for challenging situations where a large number of devices communicate with each other over a bandwidth limited channel. Most implementations are not scaled properly and are often designed for a predetermined set of environmental conditions and within a map of a predetermined size in order to avoid the need for large amounts of storage, processing power, or communication bandwidth.

Disclosure of Invention

Aspects and/or embodiments seek to provide a distributed positioning and/or mapping system that is capable of delivering substantially highly accurate real-time positioning at one or more devices.

According to a first aspect, there is provided a method of determining the relative position of a device having one or more sensors, comprising the steps of: receiving positioning requests from a plurality of devices, the positioning requests including at least a portion of data from one or more sensors; transmitting positioning data to at least one of the plurality of devices in response to the positioning request; receiving pose data from one or more of a plurality of devices; transmitting the pose data to at least one of the plurality of devices; wherein one or more of the plurality of devices are operable to determine a relative position of at least one or more of the plurality of devices from the received positioning data and pose data.

According to a second aspect, there is provided a method of determining the position of a device having one or more sensors relative to one or more other devices, comprising the steps of: sending a location request to a server system, the location request including at least a portion of data from one or more sensors; receiving positioning data from the server system in response to the positioning request; sending the pose data to a server system; receiving pose data for at least one other device from a server system; a position of the device relative to the at least one other device is determined from the received positioning data and pose data.

According to a third aspect, there is provided a system for determining the position of devices relative to each other, the devices having one or more sensors, wherein the system comprises: a plurality of client devices including at least one sensor that generates sensor data and a client location system; a server system comprising a global map and a pose manager containing pose data for at least one client device; wherein the client positioning system is configured to send positioning information to the server system and receive positioning information from the server system; wherein the at least one client device is configured to send pose data to and receive pose data from the pose manager; and wherein at least one client device is configured to combine the client device pose data with pose data of other client devices to determine a relative position of the client device and the other client devices.

In the case where devices or client devices provide both sensor data and pose data when making a positioning request to the server system, this may allow the server system to provide more accurate positioning data and/or may allow one or each of the (client) devices to determine the relative position and/or pose of one or more other (client) devices. The server system may use a global map (map ), sometimes referred to as a map, a reference map, or a primary map, to determine a location of the (client) device and send the location to the (client) device in location data.

Optionally, the step of combining pose data is performed, and wherein the step of determining the position of the device comprises determining the position of the device using the combined pose data.

Using the combined pose data may enable a more accurate determination of the position of the device.

Optionally, the location request is sent to a server system.

By sending a location request to the server system, the client device can obtain a global reference location from the server system, rather than relying on a potentially inaccurate local determination of the location.

Optionally, pose data for individual client devices is stored on a pose manager, and wherein the pose manager receives pose data for a client device and transmits the pose data for that client device to other client devices.

By storing pose data from multiple client devices in one location, some or all of the collected pose data may be sent to one or more of the client devices to enable the client devices to determine the relative location and/or pose of one or more of the other client devices.

Optionally, a further step of providing the relative position information of the client device to the application is performed. Optionally, the application is an augmented reality engine. Optionally, the client device has a display that displays the augmented reality data.

By providing the relative location information to the application, the client device may display the relative location information to the user, for example by showing the relative location information in or on a map. The relative location information may be displayed by the client device using various applications and/or mechanisms, for example, on the device display and/or any heads-up display (HUD) or Augmented Reality (AR) glasses, etc.

Optionally, the positioning request further comprises an estimated position. Optionally, the method further comprises the step of estimating a location of the device based on data from the one or more sensors, and wherein the step of determining the location of the device comprises determining the location of the device using the estimated location.

The position of the device may be estimated using various methods, including using local area sensors such as satellite positioning to estimate the position of the device before using the estimated position and other data to determine the position for the device.

Optionally, the step of determining the location of the device further comprises using at least a portion of the data from the one or more sensors and the received positioning data.

Using a portion of the data from one or more sensors in determining the location of the device allows the location to be determined using the sensor data. At least a portion of the data may include visual range data and/or some or all of the raw data associated with any visual range.

Optionally, the step of determining the location of the device comprises any of: determining a location of the device at a point in time when the location request is sent to the server system; or the current location of the device is determined taking into account the point in time at which the positioning request was sent to the server system.

Considering the point in time at which the positioning request is sent to the server system may allow the server system to compensate for movement of the device since the positioning request was sent to the server system, and/or may allow the server system to determine the current position of the device in consideration of the time difference since the positioning request was sent by the client device to the server system.

Optionally, the one or more sensors comprise at least one visual data sensor. Optionally, the at least one visual data sensor comprises any one or a combination of: an image camera; a video camera; a monocular camera; a depth camera; a stereoscopic image camera; a high dynamic range camera; a light detection and ranging sensor; radio detection and ranging sensors. Optionally, at least a portion of the data from the one or more sensors comprises visual data from at least one visual data sensor.

Sensor data including visual data allows the device to send visual data to the server along with any positioning requests, which may allow the server to use the visual data to determine the position and/or pose of the device. Various visual data may be used for this purpose.

Optionally, the one or more sensors comprise an inertial measurement unit.

Sensor data including inertial measurement data allows the device to send this data to the server along with any positioning requests, which may allow the server to use the inertial measurement data to determine the position and/or pose of the device.

Optionally, the step of estimating the position further comprises using previous positioning data. Optionally, the previous positioning data comprises any of: one or more previous location estimates; one or more previous location requests; one or more of previous said positioning data received from a server system in response to a previous said positioning request; as well as previous location determinations.

Using previous position estimates may allow evaluation of possible future and current positions.

Optionally, the step of estimating the position is performed using any of: local pose; or a local reference frame. Optionally, the positioning data received from the server system in response to the positioning request uses any of: global pose; or a global frame of reference.

A location may be estimated using a local pose or local reference frame based on cross-referencing the local pose or local reference frame with data from one or more devices and/or a global pose and/or a global map and/or a global reference frame stored on a server.

Optionally, the method further comprises the steps of: determining transformation or relation between the global pose and the local pose; alternatively, a transformation or relationship of the global reference frame to the local reference frame is determined.

Understanding the relationship between or forming a transformation between the global pose and the local pose of the device and/or the global reference frame and the local reference frame of the device may allow the server to more accurately locate the device.

Optionally, the apparatus comprises any of the following: a mobile phone, a robot, an augmented reality or virtual reality headset, or a navigation device, an autonomous vehicle, a drone, or other autonomous vehicle.

The device may take many different forms, and aspects and/or embodiments may be used to enable positioning of the device in each of these forms. As an example, a mobile phone may be installed in a standard road vehicle as a way to retrofit an augmented reality application to any vehicle.

Optionally, the step of determining the position of the device is accurate to within a few centimeters, optionally to within one centimeter.

With an accuracy within a few centimeters, aspects and/or embodiments may provide suitably accurate positioning to enable better use of existing technology capabilities as well as new technology capabilities.

Optionally, the method is continuous.

Providing a continuous solution enables a solution that critical systems can rely on, allowing better use of existing technical capabilities as well as new technical capabilities.

Alternatively, determining the location of the device may be performed inside a building or in a dense urban area.

Providing a method by which locations inside and outside a building and/or in dense urban areas may be determined may allow consistently accurate positioning in a wide variety of environments, such as, for example, tunnels, parking lots, or other environments where reliable GNSS based positioning is not accessible.

Optionally, the sensor data comprises data on environmental conditions, such as data on weather conditions.

In the case of systems that use, for example, visual range or other techniques, it may be helpful to understand and have relevant reference data for different weather conditions to allow positioning in different weather conditions.

Optionally, the server system comprises global map data; optionally wherein the global map data comprises any of: one or more sets of local map data and/or one or more sets of global map data and/or one or more global maps for the interconnected local maps.

By providing a global map, the global map may be used as a reference data source from which to locate a device that may be determined for sending a location request or to determine the location of the device.

Alternatively, the server system may be any of: a single server, a distributed server system, a cloud system, a physical system, or a virtual system. Alternatively, the global map may be distributed among multiple server systems. Optionally, the global map may be hosted in a peer-to-peer arrangement on any of: one or more devices acting as a server system, or a mixture of one or more server systems and one or more devices. Optionally, the device and the server system communicate over any of a mobile data network or a wireless data network.

Various hardware and/or software configurations and/or implementations may be used to provide flexibility and/or scale determination for the system.

Optionally, the apparatus has a pitch system. Optionally, the range system processes the sensor data to determine a relative positional difference between successive sensor data. Optionally, the position estimate has between three and six degrees of freedom. Optionally, the range system performs fully simultaneous localization and mapping and/or loop closure and/or graph optimization. Optionally, the range system outputs the estimated position.

Providing a range system may allow more accurate position estimates to be determined by the device and/or may allow richer sensor data and/or metadata to be collected for submission to the server along with the positioning request.

Optionally, the positioning data responsive to the positioning request is transmitted to two or more of the plurality of devices. Optionally, pose data is received from two or more of the plurality of devices.

Providing positioning data to and/or receiving pose data from two or more devices may allow relative pose and/or positioning to be determined.

Optionally, the estimated position is relative to a local origin. Optionally, the previous positioning data is stored with a timestamp. Optionally, the location of the device is determined in a global coordinate system. Optionally, the positioning request is in a local coordinate system. Optionally, the method further comprises the step of determining a transformation between the global coordinate system and the local coordinate system. Optionally, the positioning data is in a global coordinate system.

Optionally, the pose data further comprises velocity data.

In addition to transmitting the estimated pose of the device to and from the server, the pose data may also include a velocity estimate. This may allow the location of the device to be predicted within a short time frame and mitigate time lags in the location of the device caused by latency to the server (i.e., showing the current location of the device rather than the location at the time data is collected or a location request is transmitted). Such information may be estimated by differentiating pose data or obtained directly from implemented accelerometer sensors.

Optionally, the pose data further comprises an estimate of uncertainty.

In addition to transmitting pose data or estimated device poses to and from the server, an estimate of uncertainty of the estimated poses may be transmitted to the server along with the pose data. Modifications may be made in the transmission data and associated processing to enhance the process to support an estimation of the accuracy of the calculated pose.

Optionally, one or more of the plurality of client devices together provide at least a portion of the functionality of the server system and/or the client devices communicate in a peer-to-peer arrangement.

Devices that are capable of sharing the same low latency network connection, such as bluetooth or connected to the same Wi-Fi network, may have their own version of the server module and may additionally exchange information about their pose and optionally map fragments (fragmentation maps) directly in the case of or bypassing communication with the server. This may improve the performance of the system, reduce communication lag, and make the system robust against the effects of central server unreachability or failure.

Optionally, the sensor data is used to create or update a global map. Optionally, creating or updating the global map comprises any of: creating a new node in the global map; updating or modifying existing nodes in the global map; deleting nodes in the global map; optimizing a global map; or perform global loop closure to link the locations of different positioning nodes.

Updating the global map or the global master map with sensor data allows the global map to adapt over time through AR usage and improve continuously. This includes global maps that are, for example, larger and larger in size and richer in data for different environmental conditions.

Drawings

Embodiments will now be described, by way of example only, with reference to the accompanying drawings having the same reference numerals, in which:

FIG. 1 shows an overview of a system according to an embodiment;

FIG. 2 illustrates a plurality of client devices according to an embodiment; and

FIG. 3 illustrates a flow chart showing system operation according to an embodiment.

Detailed Description

An example embodiment will now be described with reference to fig. 1.

In the embodiment of fig. 1, the system 1 comprises one server system 2 communicating with at least one client device 3.

The client device 3 according to this embodiment will now be described in more detail.

(in use, multiple client devices 3 will typically be deployed.)

In this embodiment, the client device 3 is configured to run on client hardware 3 equipped with at least one sensor 31. For example, the client hardware 3 may be a device such as a mobile phone, an augmented reality headset, a robot, a drone, or an autonomous car driving system. Typically, at least one, and in many cases a plurality of client devices 3 will be deployed.

The sensor 31 may for example comprise: a camera 31, which may be an image sensor, a depth camera, a light detection and ranging ("LIDAR"), or a video sensor, an inertial measurement unit ("IMU") 32, and/or a satellite positioning system, such as a global navigation satellite system ("GNSS"), 33. The data from the sensors 31 may be combined or mixed by a combiner 34. It should be appreciated that any suitable sensor technology may be used as one or more of the types of sensors 31 listed above.

The role of the or each client device 3 is: processing client device sensor data acquired from the sensors 31, communicating with the server system 2 to maintain a real-time global position estimate for the or each client device 3, providing individual pose data to the server system 2, receiving pose data for other client devices 3, and combining the individual pose data with pose data for other client devices 3 to determine relative positions of the other client devices 3.

In addition, the or each client device 3 is configured to maintain high frequency communication with the server pose manager 23 in order to transmit its own pose information and receive the locations of other client devices 3. Combining the client device's 3 own pose with the poses of other client devices 3 results in an accurate real-time estimate of the pose of the client device 3 to be used in AR or robotic applications.

The or each client device 3 contains a client location system 4 which acquires data from the sensor 31 and processes the data from the sensor 31. The client positioning system 4 communicates with the server system 2 to maintain an accurate global pose 5.

Furthermore, the or each client device 3 of the embodiment is configured to process sensor data, and the or each client device 3 comprises a client positioning system 4. The client positioning system 4 may take the form of a local visual range ("VO") system, or may be a simultaneous positioning and mapping ("SLAM") system. The client positioning system 4 is configured to process sensor data from the sensors 31 of the client device 3 to determine the movement and location of the client device in the local map of the environment. In addition, client positioning system 4 generates positioning requests, calculates and reports device locations, receives one or more absolute poses for other devices, and calculates pixel coordinates for other devices.

The sensor data is used to determine the movement and location of the client device 3 in a local map of the environment. In view of the local area map and sensor 31 data, the client positioning system 4 can determine the location of the client device 3. This can be done using standard positioning methods such as triangulation or shape matching at a relatively high rate and whenever such a reading is available (and typically at 30 Hz). Communication with the server system in the form of map downloads or synchronization typically occurs at a slower rate (e.g., at a rate of once per second or once every few seconds).

The local map may be constructed using sensors or by directly downloading a portion of the global map from the server system 2 (explained in more detail later). If the map is created from sensor data, an additional synchronization step of the local and global maps will be required to determine the location of the local map relative to the global map.

This determination may be accomplished in several ways, based on the sensor 31 used. For example, in the case of a camera sensor, camera movement may be determined by matching and triangulating observed image features or pixel intensities between successive measurements to perform a visual range. A grid matching method such as iterative closest point optimization may be used to achieve similar pose estimation in an active sensor 31 such as a depth camera or LIDAR sensor. Furthermore, several such measurements from different sensors 31 and modalities may be integrated into one pose estimation using methods such as kalman filtering to compensate for the deficiencies of the individual sensors 31 and achieve higher robustness and accuracy.

Based on the sensor 31 data and/or relative position differences, the measurements are then accumulated to provide a relatively real-time position estimate. In the most general case, the position estimate takes the form of a pose with six degrees of freedom (three-dimensional rotation and three-dimensional translation) and with additional information such as velocity and/or acceleration. In case of an embodiment involving e.g. planar automatic scenes, the position estimation can be reasonably reduced to three degrees of freedom (rotation and two-dimensional translation), together with velocity and/or acceleration. In some embodiments, only one degree of freedom may be used.

The client device 3 also includes a global pose record 30 that communicates with the server system and receives positioning responses from the server system. The global pose record 30 is in communication with a global pose estimator. The global pose estimator is also in communication with the local pose record and range system. The global pose estimator outputs an estimated global position of the client device 3. The range system and local pose records communicate with the sensors 31. The global pose estimator outputs an estimated global position of the client device 3, and the estimated global position may be combined with the combined sensor data to be sent to the server system 2 as a positioning request.

The role of the server system 2 in this embodiment is to maintain and update a consistent global master map 22 and to respond to global positioning requests 39 from client devices 3 using global master map 22 data stored on the server system 3. In this embodiment, which typically communicates over a bandwidth-limited communication channel, the global positioning requests 39 from each client device are expected to occur at a low frequency in order to minimize bandwidth usage and/or utilize available bandwidth in an efficient manner. The positioning response 26 is sent by the server system 2 to the client device 3 that sent the positioning request 39.

Returning to the more general discussion of the client device 3, errors may be introduced when performing the ranging. This is because the range typically accumulates error over time if based only on local sensor data and estimates. The effect of this error is known as "drift". To mitigate this effect, the client device 3 may implement a SLAM system as discussed above. The system uses a loop closure and map optimization program and produces a much more accurate position output.

The implementation of such a system will depend on the type of sensor 31 used, such as, for example, a monocular or stereo camera, a depth camera, or one or more laser sensors.

The server system 2 of this embodiment will now be described in more detail below.

The server system 2 includes: a global map 22 maintaining a localization map of the environment, a localizer 21 accepting a localization request 41 and responding 42 to the localization request, and a pose manager 23 synchronizing and transmitting pose information 5 of individual client devices 3.

In this embodiment, the server system 2 runs on and is implemented using a cloud infrastructure, but in other embodiments, the server system 2 may have various physical and/or virtual configurations. In other embodiments, for example, there may be one or more servers and/or server systems, and where there is more than one server and/or server system, these servers and/or server systems may be configured to act as a single server or server system or as multiple independent servers or server systems, and may or may not be in direct communication with each other.

A global map 22 is maintained at the server system 2, the global map 22 having or using a global system, i.e. providing coordinates of a global reference system. Each client device 3 may have its own local frame, i.e. local frame of reference. However, in some embodiments, the global master map may include a plurality of local and/or global maps, each having their own respective local and/or global frame of reference.

Alternatively, devices that can share the same low latency network connection such as bluetooth or connect to the same Wi-Fi network may have their own version of the server module and may exchange information about their pose and optionally a map of the parcel, for example in a peer-to-peer arrangement with or bypassing communications with the server. This may improve the performance of the system, reduce communication lag, and make the system robust against the effects of central server unreachability or failure.

The client device 3 and/or the server system 2 may not be able to relate the local reference frames of multiple client devices 3 to each other, and therefore these local reference frames may need to exist as separate local maps and/or global maps within the global master map. For example, in case the client device 3 is operating indoors, e.g. in a factory, and does not need to leave the factory, it is often not possible to relate it to an outdoor local area map or other local and/or global maps related to other client devices 3, e.g. with GNSS coordinates.

In this embodiment, the global map 22 is composed of the data necessary to perform a global localization of the device given its sensor data. For example, in the case of a visual positioning system, a map may consist of 3D locations of visual features in space, such as image angles, scale-invariant feature transforms ("SIFTs"), AKAZE local feature matching, or binary robust scale-invariant keypoint ("BRISK") analysis that may be used to triangulate the position of a camera when capturing images using established methods. In the case of a LIDAR or depth camera based sensor, the map may contain a 3D point cloud to perform shape matching based localization, or a combination of visual and shape based methods. Such maps may be pre-surveyed environments using different sensors or may be constructed in a distributed manner.

The server system 2 comprises a locator 20 responsive to a location request from the client device 3. The locator 20 at the server system 2 responds 42 to the location request 41 of the or each client device 3 with an estimate of the "global position" of the or each client device 3 at the time the location request 41 was issued, which estimate is sent by the server system 2 as a location response 42.

The localization request 41 from the or each client device 3 contains a portion of the sensor data from the one or more sensors 31 necessary to perform a global localization using the global map. The positioning request 41 may also contain additional data necessary for positioning, such as its estimated position or the result of a previous positioning request or a partial local area map constructed by the local SLAM system.

The positioning request 41 is sent to the positioner or (sub-) process 20 at the server system 2. At the same time, the current estimate of the device's position in the local coordinate system resulting from the measure distance is added to the client local pose record database. The locator 20 at the server system 2 responds to the location request 41 of the or each client device 3 with an estimate of the global pose of the or each client device 3 at the time the location request 41 was issued, which is sent by the server system 2 as a location response 41. The positioning response 41, when received by the or each client device 3, is then stored in the global pose record of the or each client device 3. As soon as at least one positioning request 41 is successfully responded to, the relative and global poses of these requests are retrieved from the local and global pose records and compared to provide an estimate of the local origin pose of the client device 3 in the global map. This estimate is then combined with subsequent high frequency device pose/position estimates from the range system in the local coordinate system to provide a high frequency device pose or position in the global coordinate system.

In some implementations, the information stored in the local and/or global pose records is a list of local and/or global positions of the device. Each location is associated with a particular time and a unique timestamp or ID. In some embodiments, each location may alternatively or additionally be associated with GPS data, which may include time-stamped data. The time may be the time measured by a particular sensor and a timestamp or ID may be used to cross-reference local and global recordings. The local and global poses of one or more devices are correlated with the current local device pose to give the current global device pose. In other embodiments, additional or alternative information may be stored in the local or global pose records.

The locator 20 performs the 6 degree of freedom location of the device at the time of the request using the data from the location request and transmits this information back to the client device. In addition to the 6 degree of freedom pose, the localization response may contain additional information necessary to perform or improve the performance of the client local SLAM system, such as a portion of a global map around the device location. This communication occurs infrequently, usually once every second or every few seconds, due to the amount of data required.

The server pose manager receives such information and transmits the information back to all relevant client devices whenever the global position of an individual client device can become available from the client vision system (typically at the frequency of client sensor readings, or 30 Hz). In this way, all devices can know the location of the interactive AR and other devices necessary for the robotic application at real-time speed.

The system of some or all of these embodiments may be used in distributed large-scale scenarios with low-cost client hardware, such as mobile phones, and/or with any or all of augmented reality headsets, autonomous cars, drones, and other robots.

The server system 2 and the client device 3 typically communicate with each other over a bandwidth limited communication channel, and in this embodiment the communication channel 4 is a mobile telephone cellular data network, for example. Such limitations do not allow, for example, sensor data to be streamed to a server with high fidelity in real time, and do not allow some established SLAM systems to be used. Therefore, the communication protocol must be as efficient as possible. In other embodiments, other wireless data networks may be used in place of or in addition to the mobile phone cellular data network.

To ensure the accuracy of the relative pose between two devices, broadly speaking, there are four elements in play, which are:

the accuracy of the SLAM system on the or each client device 3;

the accuracy of locating the first client device 3 against the (against, relative) global map;

the accuracy of locating the second client device 3 against the global map; and

accuracy of global map data between the locations of the first client device 3 and the second client device 3.

The accuracy of the SLAM on the or each client device 3 is the amount of "drift" (discussed above) that has accumulated over time since the last fix. The techniques and apparatus described herein seek to mitigate (or mitigate) the "drift" phenomenon.

The accuracy of the localisation of the first and second client devices 3 against the global map can be derived from the performance of the localiser and the quality of the global map at any particular location. The accuracy of the global map data between the locations of the first and second client devices 3 depends on the quality of the underlying (potential) map. Client devices 3 located against the same portion of the global map will have a low level of relative uncertainty between their locations, while devices located against different portions of the global map will have a higher level of relative uncertainty. Various techniques may be used to calculate the relative accuracy of (and between) different portions of the global map, one such technique calculating the shortest path in the associated pose graph of the base map.

Considering all four elements presented above, the SLAM system, pose estimator, locator and pose manager on the or each client device 3 in fig. 4 are required to cooperate to perform the respective segment calculations and appropriate enhancements of the transmitted data to support the calculations. This includes passing information about the relative accuracy of the pose estimation from the locators and SLAM systems on the or each client device 3 in a particular form (e.g. as a covariance matrix), estimating the accuracy of the global map between all pairs of located client devices 3 (or a subset) in the pose manager, and combining all this information together in the pose estimator.

The output of the client positioning system 4 may provide a substantially high quality estimate (or estimates) of the device's location relative to some arbitrary local origin, which is typically the device's location at system start-up or beginning.

In order to obtain a global position fix (i.e., a substantially highly accurate position in the global map), the relative position in the global coordinate map with respect to the local origin must be estimated. To achieve this, the client device 3 periodically performs a global "positioning request" 41 to the server system 2. Summaries of recent sensor inputs in the form of, for example, image or video data, depth maps, features, relationships with previous positioning requests, etc. are aggregated to create a positioning request 41. Sometimes, for a particular train, data will only be available from one, typically high frequency, sensor, such as the IMU, and therefore only that data is transmitted in the location request 41. For a typical system, data may be available from multiple sensors, such as images from vision sensors and IMU data, which may be transmitted in a location request 41. Since the bandwidth between the client device 2 and the server system 3 is limited, the positioning request 41 typically has a much smaller data size and is performed at a much lower frequency than the relevant equivalent raw sensor data, but given sufficient bandwidth, the raw sensor data may optionally be streamed to the server system as a continuous positioning request (and similarly, then the positioning response from the server may be intermittent or continuous).

The positioning request 41 is sent to a locator module or (sub) process 21 at the server system 2. The locator 20 at the server system 2 responds to the location request 41 of the client device 3 with an estimate of the "global position" of the device 3 at the time the location request 41 was issued, which is sent by the server system 2 as a location response 42.

This estimate is then combined with subsequent high frequency device pose/position estimates from the client positioning system 4 in the local coordinate system to provide a high frequency device pose, or position, i.e. the global position 5 of the device, in the global coordinate system.

In this embodiment, the transmission of pose data to and from the server is handled by the client pose manager for the client device.

In order to maintain real-time interaction rates with other client devices 3, their relative positions with respect to the other client devices 3 must be known. This is achieved by reporting the device position to the server pose manager 23 whenever a new global position for the device 5 is available.

The server pose manager 23 collects global pose data 5 for individual client devices 3 and redistributes this data to all client devices 3 involved in the system. Thus, all client devices 3 have access to the latest (and most current) global position 5 of each other client device 3 and can thus determine their relative position. Furthermore, since the global position information 5 is typically very small (for 6 degrees of freedom poses), it can communicate with the server at a very high frequency, ensuring that the latest global position 5 of each client device 3 is available for each client device 3.

Given the up-to-date nature of the global position 5 of each client device 3, the relative position of the client device 3 may be used in the application. In an embodiment, the global position 5 of a particular client device 3 and the global positions 5 of other client devices 3 (which become the relative positions of the other client devices 3) are sent to the AR engine 60 to enable information about the relative positions of the other client devices 3 to be projected onto the display 61 using the AR engine 60. This may use projection geometry, or the enhanced object or marker may be rendered or otherwise represented against an appropriate person or object. In the case of AR applications, this is shown in fig. 2.

The operation of the system 1 shown in the flow chart of fig. 3 will now be described in further detail.

In step 301, the client device 3 obtains sensor data 34, e.g. image or video data from a camera, from the on-board sensors 31, 32, 33. In step 302, the client device 3 requests its global device location from the locator 21. In step 303, the client device 3 receives its global device location from the locator 21. In step 304, the client device 3 combines the sensor data 34 with the positioning response 42 to determine its global device pose 5. In step 305, the client device 3 sends its global device pose 5 to the pose manager 23. In step 306, the client device 3 receives the global device pose 52 of the other device from the pose manager 23. In step 307, the client device combines its individual global pose 5 with the device pose 52 of any other device and determines their relative position 53. In step 308, the client device 3 uses the relative location information 53 in the application 60.

In some implementations, some or all of the client devices do not estimate their locations and/or do not transmit the estimated locations along with the sensor data to the server in a positioning request. In general, this will be the case for the following devices: these devices have temporarily invalid or malfunctioning range systems, or limited functionality or hardware, or are generating different inaccurate position estimates.

The term "global map" throughout the specification may be replaced with "map", "reference map", "base map", or "primary map", respectively. The term "global map" is intended to define the primary or primary map data for reference targets, typically stored at a server. A map referred to throughout the specification as a "local map" is a map that is typically stored and/or generated on a device (such as a mobile device), and the term may be replaced with a "city map", a "local map", or a "portion and/or subset of a global map".

Any system feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus functional features may alternatively be expressed in terms of their corresponding structures.

Any features of one aspect of the invention may be applied to other aspects of the invention in any suitable combination. In particular, method aspects may be applied to system aspects and vice versa. Furthermore, any, some, and/or all features of one aspect may be applied to any, some, and/or all features of any other aspect, in any suitable combination.

It is also to be understood that particular combinations of the various features described and defined in any aspect of the invention may be implemented and/or supplied and/or used independently.

Claims (56)

1. A method of determining the relative position of a device having one or more sensors, comprising the steps of:

receiving positioning requests from a plurality of devices, the positioning requests including at least a portion of data from the one or more sensors;

transmitting positioning data to at least one of the plurality of devices in response to the positioning request;

receiving pose data from one or more of the plurality of devices;

transmitting pose data to each of the plurality of devices;

wherein each device of the plurality of devices is operable to determine the relative position of the plurality of devices from the received positioning data and pose data.

2. A method of determining the position of a device having one or more sensors relative to one or more other devices, comprising the steps of:

sending a location request to a server system, the location request including at least a portion of data from the one or more sensors;

receiving positioning data from the server system in response to the positioning request;

sending the pose data to a server system;

receiving pose data of other devices from the server system;

determining a position of the device relative to the other devices from the received positioning data and pose data.

3. A system for determining the position of devices having one or more sensors relative to each other, the system comprising:

a plurality of client devices comprising:

at least one sensor generating sensor data; and

a client positioning system;

a server system, comprising:

a global map; and

a pose manager containing pose data for a plurality of client devices;

wherein the client positioning system is configured to:

sending and receiving positioning information to and from the server system;

wherein the client device is configured to:

sending pose data to and receiving pose data from the pose manager;

and the number of the first and second electrodes,

wherein the client device is configured to:

combining the pose data of the client device with the pose data of other client devices to determine a relative position of the client device and the other client devices.

4. The method of any preceding claim, further comprising combining pose data, and wherein determining the position of the device further comprises using the combined pose data to determine the position of the device.

5. A method according to any preceding claim, wherein the location request is sent to a server system.

6. The method of any preceding claim, wherein the pose data of individual client devices is stored on a pose manager, and wherein the pose manager receives the pose data of the client devices and transmits the pose data of the client devices to other client devices.

7. The method of any preceding claim, further comprising the steps of: providing the relative location information of the client device to an application.

8. The method of claim 7, wherein the application is an augmented reality engine.

9. The method of claim 8, wherein the client device has a display that displays augmented reality data.

10. A method as claimed in any preceding claim, further comprising the step of estimating the location of the device based on data from the one or more sensors, and wherein the step of determining the location of the device comprises determining the location of the device using the estimated location.

11. A method as claimed in any preceding claim, wherein the step of determining the position of the device further comprises using at least a portion of the data from the one or more sensors and the received positioning data.

12. The method of any preceding claim, wherein the positioning request further comprises an estimated location.

13. A method as claimed in any preceding claim, wherein the step of determining the location of the device comprises any of: determining a location of the device at a point in time when the location request is sent to the server system; alternatively, the current location of the device is determined taking into account the point in time at which the positioning request is sent to the server system.

14. The method of any preceding claim, wherein the one or more sensors comprise at least one visual data sensor.

15. The method according to the preceding claim, wherein the at least one visual data sensor comprises any or a combination of: an image camera; a video camera; a monocular camera; a depth camera; a stereoscopic image camera; a high dynamic range camera; a light detection and ranging sensor; radio detection and ranging sensors.

16. The method of claim 14 or 15, wherein the at least a portion of the data from the one or more sensors comprises visual data from the at least one visual data sensor.

17. A method according to any preceding claim, wherein the one or more sensors comprise an inertial measurement unit.

18. A method according to any preceding claim, wherein the step of estimating a position further comprises using previous positioning data.

19. The method of the preceding claim, wherein the previous positioning data comprises any of: one or more previous location estimates; one or more previous location requests; one or more of previous said positioning data received from said server system in response to a previous said positioning request; and, previous location determinations.

20. A method as claimed in any preceding claim, wherein the step of estimating a position is performed using any of: local pose; or a local reference frame.

21. A method as claimed in any preceding claim, wherein the positioning data received from the server system in response to the positioning request uses any of: global pose; or a global frame of reference.

22. Method according to the preceding claim, further comprising the step of: determining transformation or relation between the global pose and the local pose; alternatively, a transformation or relationship of the global reference frame to the local reference frame is determined.

23. A method according to any preceding claim, wherein the apparatus comprises any of: a mobile phone, a robot, an augmented reality or virtual reality headset, or a navigation device, an autonomous vehicle, a drone, or other autonomous vehicle.

24. A method as claimed in any preceding claim, wherein the step of determining the position of the device is accurate to within a few centimetres, optionally within one centimetre.

25. The method of any preceding claim, wherein the method is continuous.

26. The method of any preceding claim, wherein determining the location of the device can be performed inside a building or in a dense urban area.

27. A method according to any preceding claim, wherein the sensor data comprises data relating to environmental conditions, such as data relating to weather conditions.

28. A method according to any preceding claim, wherein the server system comprises global map data; optionally, wherein the global map data comprises any of: one or more sets of local map data and/or one or more sets of global map data and/or one or more global maps having interconnected local maps.

29. A method according to any preceding claim, wherein the server system can be any of: a single server, a distributed server system, a cloud system, a physical system, or a virtual system.

30. The method according to the preceding claim, wherein the global map is distributable between a plurality of server systems.

31. The method of any of claims 28, 29 or 30, wherein the global map can be hosted in a peer-to-peer arrangement on any of: one or more devices acting as a server system; or a mixture of one or more server systems and one or more devices.

32. A method according to any preceding claim, wherein the device and server system communicate by any of: a mobile data network; or, a wireless data network.

33. The method of any preceding claim, wherein the apparatus has a range system.

34. The method of the preceding claim, wherein the range system processes sensor data to determine a relative positional difference between successive sensor data.

35. The method according to the preceding claim, wherein the position estimate has between three and six degrees of freedom.

36. The method according to any of the two preceding claims, wherein the range system performs fully simultaneous localization and mapping and/or loop closure and/or graph optimization.

37. The method of any one of claims 33 to 36, wherein the range system outputs the estimated position.

38. A method as claimed in any preceding claim, wherein the positioning data is transmitted to two or more of the plurality of devices in response to the positioning request.

39. A method as claimed in any preceding claim, wherein the pose data is received from two or more of the plurality of devices.

40. A method as claimed in any preceding claim, wherein the method determines a relative position and/or a relative pose.

41. The method of any preceding claim, wherein the at least one of the plurality of devices comprises each device.

42. The method of any preceding claim, wherein each client device comprises: at least one sensor generating sensor data; and/or, a client location system.

43. The method of any preceding claim, wherein the pose data further comprises velocity data.

44. A method as claimed in any preceding claim, wherein the pose data further comprises an estimate of uncertainty.

45. A method as claimed in any preceding claim, wherein one or more of the plurality of client devices provide at least part of the functionality of the server system and/or the client devices communicate in a peer-to-peer arrangement.

46. The method of any preceding claim, wherein the estimated position is relative to a local origin.

47. The method of any preceding claim, wherein the previous positioning data is stored with a timestamp.

48. A method according to any preceding claim, wherein the location of the device is determined in a global coordinate system.

49. The method of any preceding claim, wherein the location request is in a local coordinate system.

50. Method according to the preceding claim, further comprising the step of: a transformation between the global coordinate system and the local coordinate system is determined.

51. The method of any preceding claim, wherein the positioning data is in a global coordinate system.

52. The method of any of claims 3, 28 to 31, further comprising: the global map is created or updated using sensor data.

53. The method of claim 52, wherein creating or updating the global map comprises any of: creating a new node in the global map; updating or modifying existing nodes in the global map; deleting nodes in the global map; optimizing the global map; alternatively, a global loop closure is performed to link the positions of the different positioning nodes.

54. A system, the system comprising: at least one server system operable to perform the method of any preceding method claim; and at least one client device.

55. A computer program product operable to perform the method and/or system of any preceding claim.

56. An apparatus configured to provide the method of any preceding claim.