DE202014011010U1 - Target tracking systems - Google Patents

Abstract

A system for controlling an unmanned aerial vehicle (101), UAV, the system comprising: one or more receivers configured to command one or more navigation commands from a remote controller (1) to move the UAV along a flight path, and (2) to receive destination information (116) of a target to be tracked by an imaging device (104) connected to the UAV (101) via a carrier (102), the carrier being configured to allow the carrier to be tracked Imaging means (104) relative to the UAV (101), and one or more processors configured to select between (1) an automatic tracking mode and (2) a manual tracking mode, wherein in the automatic tracking mode one or more of the following a plurality of processors are configured to adjust at least one of the UAV (101) or the carrier (102) while the UAV travels along the trajectory e In response to the one or more navigation commands, the remote controller moves to track the destination in accordance with the destination information to maintain the destination remaining within an image captured by the imaging device.

Description

GENERAL PRIOR ART

Aircraft, such as unmanned aerial vehicles (UAVs), may be used to perform surveillance, reconnaissance and reconnaissance tasks for military and civil applications. Such aircraft may carry a payload configured to perform a specific function, such as taking pictures of the surrounding environment.

In some circumstances, it may be desirable for aircraft to have a specific destination. For small size aircraft, such tracking is conventionally accomplished via control commands from a user-operated remote control terminal or device. Such manual tracking control may become difficult in certain circumstances, such as when the moving object or target is moving fast or when the moving object is at least partially hidden from the user's view. Moreover, the attention required for such manual tracking requires an associated user controlling a camera on board the aircraft, separate from a user controlling the navigation of the aircraft, thereby increasing the cost of aerial photography and other applications of the aircraft.

BRIEF SUMMARY OF THE INVENTION

In some circumstances, it may be desirable for aircraft to have a specific destination. Thus, there is a need for improved UAV tracking methods and systems that ensure automatic or semi-automatic tracking of a target, thereby relieving operators of the aircraft of manually tracking the targets. The present invention provides systems, methods, and devices associated with target tracking by UAV. The UAV may be configured to receive destination information regarding a destination to be followed by an imager connected to the UAV from a control station. The target information may be used by the UAV to automatically track the target so as to maintain a predetermined position and / or size of the target within one or more images captured by the imager. Any description of the tracking may include visual tracking by the imaging device. The control station may be configured to display images from the imaging device as well as to allow user input regarding the destination information.

According to one aspect of the present invention, a method for controlling an unmanned aerial vehicle (UAV) is provided. The method includes receiving, from a remote user, one or more navigation commands to move the UAV along a trajectory, receiving, from the remote user, destination information of a target to be tracked by an imager on the UAV, and tracking the target corresponding to the target information by automatically adjusting at least one of the UAV and the imaging device while the UAV moves along the trajectory corresponding to the one or more navigation commands from the remote user.

According to another aspect of the present invention, an unmanned aerial vehicle (UAV) with tracking capabilities is provided. The UAV comprises: one or more receivers configured, singly or in common, to command from a remote user (1) one or more navigation commands to move the UAV along a trajectory; and (2) target information by an imaging device receive on the UAV to be tracked destination, and one or more processors configured, singly or in common, to track the destination according to the destination information by automatically adjusting at least one of the UAV and the imager while the UAV is traveling along the trajectory corresponding to the one or more navigation commands from the remote user moves.

According to another aspect of the present invention, a system for controlling an unmanned aerial vehicle (UAV) is provided. The system includes: one or more receivers configured, singly or in common, to command from a remote user (1) one or more navigation commands to move the UAV along a trajectory, and (2) target information by an imaging device receive on the UAV to be tracked destination, and one or more processors configured, singly or in common, to track the destination according to the destination information by automatically adjusting at least one of the UAV and the imager while the UAV is traveling along the trajectory corresponding to the one or more navigation commands from the remote user moves.

In some embodiments, the imaging device includes a camera or a camera recorder.

In some embodiments, the one or more navigation commands are configured to control a speed, position, or location of the UAV.

In some embodiments, the target is immobile relative to a reference object.

In some embodiments, the target moves relative to a reference object.

In some embodiments, the destination information includes initial destination information.

In some embodiments, the initial target information includes an initial position or an initial size of the target within an image captured by the imaging device.

In some embodiments, the destination information includes destination type information.

In some embodiments, tracking the target in accordance with the target information further includes identifying, based on the target species information, the target to be tracked from within one or more images captured by the imaging device using an image recognition algorithm.

In some embodiments, the destination type information includes color, texture, or pattern information.

In some embodiments, the destination information includes expected destination information.

In some embodiments, the expected target information includes an expected position or size of the target within an image captured by the imaging device.

In some embodiments, the expected size of the target is the same as an initial size of the target.

In some embodiments, the expected position of the target is the same as an initial position of the target.

In some embodiments, following the target in accordance with the target information includes tracking, within a predetermined tolerance level, the expected position, or the expected size of the target within one or more images captured by the imaging device.

In some embodiments, the imaging device is connected to the UAV via a carrier that is configured to allow the imaging device to move relative to the UAV.

In some embodiments, the carrier is configured to allow the imager to rotate about at least two axes relative to the UAV.

In some embodiments, tracking the target according to the destination information includes automatically adjusting at least one of the UAV, the carrier, or the imaging device as the UAV moves along the trajectory corresponding to the one or more navigation commands from the remote user.

In some embodiments, the target information includes expected target information, and tracking the target according to the target information includes determining the current target information of the target based on one or more images captured by the imaging device, detecting a deviation of the current target information from the expected one Destination information and calculating a setting on the UAV, the carrier or the imaging device, so as to substantially correct the deviation.

In some embodiments, the deviation is associated with a change in the position of the target, and the calculated setting is associated with an angular velocity for the UAV.

In some embodiments, the angular velocity is related to a yaw axis of the UAV.

In some embodiments, the angular velocity is related to a pitch axis of the UAV.

In some embodiments, the deviation is associated with a change in the position of the target, and the calculated setting is associated with an angular velocity for the imaging device relative to the UAV.

In some embodiments, the calculated adjustment is used to generate control signals for the wearer so as to cause the imaging device to move relative to the UAV.

In some embodiments, the deviation is a change in the size of the target and the setting is linked to a linear velocity for the UAV.

In some embodiments, the deviation is associated with a change in the size of the target, and the adjustment is associated with one or more parameters of the imaging device.

In some embodiments, the one or more parameters of the imager include focal length, zoom, or focus.

In some embodiments, the calculated adjustment is limited to a predetermined range.

In some embodiments, the predetermined range corresponds to a predetermined range of the control lever amount of a control system.

In some embodiments, the control system includes a flight control system for the UAV or a control system for the carrier.

In some embodiments, a warning signal is provided if the calculated setting falls outside the predetermined range.

In some embodiments, tracking the target includes comparing the calculated setting to a predetermined maximum threshold and providing the predetermined maximum threshold if the calculated setting exceeds the predetermined maximum threshold.

In some embodiments, the predetermined maximum threshold includes a maximum angular velocity or a maximum linear velocity for the UAV or imager.

In some embodiments, tracking the target includes comparing the calculated setting to a predetermined minimum threshold and providing the predetermined minimum threshold if the calculated setting is less than the predetermined minimum threshold.

In some embodiments, the predetermined minimum threshold includes a minimum angular velocity or a minimum linear velocity for the UAV or imager.

In some embodiments, the destination information is received from a remote control device accessible to the remote user.

In some embodiments, the one or more navigation commands are received by the same remote control device.

In some embodiments, the one or more navigation commands are received from a different remote control device.

In some embodiments, the remote control device is configured to receive a user input from a touch screen, a joystick, a keyboard, a mouse, or a stylus.

In some embodiments, the remote control device is configured to receive a user input from a portable device.

In some embodiments, the remote control device is configured to: receive one or more images captured by the imaging device from the UAV, display the one or more images, receive a user selection of a target from within a displayed image, target the destination on the UAV Base the user selection of the target to generate and transmit the target information to the UAV.

In some embodiments, the remote control device is further configured to generate the one or more navigation commands based on user input and to communicate the one or more navigation commands to the UAV.

In some embodiments, the remote control device is further configured to receive tracking information regarding the destination and display the one or more images with the tracking information.

In one aspect of the present invention, an unmanned aerial vehicle (UAV) with tracking capabilities is provided. The UAV comprises: one or more receivers, individually or in common, configured to receive, from a remote user, user-specified destination information of a destination to be tracked by an imager on the UAV, the user-specified destination information being a predetermined position or a predetermined location Size of the target within an image captured by the imager wherein the imaging device is connected to the UAV via a carrier configured to allow the imaging device to move in relation to the UAV, and one or more processors configured to be individually or collectively configured therefor Detecting deviation from the predetermined position or the predetermined size of the target based on one or more images taken by the imaging device and generating commands to automatically adjust the UAV, the carrier or the imaging device so as to detect the detected deviation from the predetermined position or substantially correct the predetermined size of the target.

According to another aspect of the present invention, a system for controlling an unmanned aerial vehicle (UAV) is provided. The system includes: one or more receivers, individually or collectively, configured to receive, from a remote user, user-specified destination information of a destination to be tracked by an imager on the UAV, the user-specified destination information being a predetermined location or a predetermined location Size of the target within an image captured by the imaging device, the imaging device being connected to the UAV via a carrier configured to allow the imaging device to move relative to the UAV, and one or more processors that , individually or collectively, configured to detect a deviation from the predetermined position or size of the target based on one or more images taken by the imaging device and generate commands to cause the UAV, de n carrier or the imaging device so as to substantially correct the detected deviation from the predetermined position or the predetermined size of the target.

According to another aspect of the present invention, a method for controlling an unmanned aerial vehicle (UAV) is provided. The method includes: receiving, from a remote user, user-specified destination information of a target to be tracked by an imager on the UAV, the user specified destination information including a predetermined position or a predetermined size of the target within an image captured by the imager, wherein the Imaging device is connected to the UAV via a carrier configured to allow the imaging device to move in relation to the UAV, detected by a processor aboard the UAV, a deviation from the predetermined position or the predetermined magnitude of the target on the basis of one or more images taken by the imaging device and automatically adjusting the UAV, the carrier or the imaging device so as to determine the detected deviation from the predetermined position or the predetermined one Essentially correct the size of the target.

In some embodiments, the imaging device includes a camera or a camera recorder.

In some embodiments, the method further comprises receiving, from the remote user, one or more commands configured to control a speed, position, or location of the UAV.

In some embodiments, the method further comprises receiving, from the remote user, one or more commands configured to control a speed, position or attitude of the wearer.

In some embodiments, the method further comprises receiving, from the remote user, one or more commands configured to control one or more operating parameters of the imaging device.

In some embodiments, the one or more operating parameters of the imager include focal length, zoom level, imaging mode, image resolution, focus, depth of field, exposure, intensity, or field of view.

In some embodiments, the carrier is configured to allow the imager to rotate about at least one axis relative to the UAV.

In some embodiments, the carrier is configured to allow the imager to rotate about at least two axes relative to the UAV.

In some embodiments, the destination information of the destination further includes destination type information.

In some embodiments, the destination type information includes a color or texture of the destination.

In some embodiments, the predetermined position of the target includes an initial position or an expected position of the target.

In some embodiments, the predetermined size of the target includes an initial size or an expected size of the target.

In some embodiments, detecting the deviation from the predetermined position or size of the target includes comparing a position or size of the target within the one or more images captured by the imager with the predetermined position or size.

In some embodiments, adjusting the UAV, the carrier, or the imaging device includes calculating a setting on the UAV, the carrier, or the imager so as to substantially correct the deviation.

In some embodiments, the deviation is associated with a change in the position of the target, and the calculated setting is associated with an angular velocity for the UAV.

In some embodiments, the angular velocity is related to a yaw axis of the UAV.

In some embodiments, the angular velocity is related to a pitch axis of the UAV.

In some embodiments, the deviation is associated with a change in the position of the target, and the calculated setting is associated with an angular velocity for the imaging device relative to the UAV.

In some embodiments, the calculated adjustment is used to generate control signals for the wearer so as to cause the imaging device to move relative to the UAV.

In some embodiments, the angular velocity is related to a yaw axis of the imager.

In some embodiments, the angular velocity is related to a pitch axis of the imager.

In some embodiments, the deviation is associated with a change in the size of the target, and the adjustment is associated with a linear velocity for the UAV.

In some embodiments, the deviation is associated with a change in the size of the target, and the adjustment is associated with one or more parameters of the imaging device.

In some embodiments, the one or more parameters of the imager include focal length, zoom level, imaging mode, image resolution, focus, depth of field, exposure, intensity, or field of view.

In some embodiments, the calculated adjustment is limited to a predetermined range.

In some embodiments, the predetermined range corresponds to a predetermined range of the control lever amount of a control system.

In some embodiments, the control system includes a navigation control system for the UAV or a control system for the wearer.

In some embodiments, the method further comprises providing a warning signal if the calculated adjustment falls outside the predetermined range.

In some embodiments, the warning signal is used to provide an audio or visual signal.

In some embodiments, the warning signal is used to provide a kinetic signal.

In some embodiments, the method further comprises communicating, in substantially real time, images taken by the imaging device to a remote user device accessible to the remote user.

In some embodiments, the remote user device includes a display for displaying the images captured by the imaging device.

In some embodiments, the remote user device includes an input device for providing the destination information.

In some embodiments, the input device includes a touch screen, a Joystick, a keyboard, a mouse or a pen.

In some embodiments, the input device includes a portable device.

In some embodiments, the destination information is provided based on the transmitted images.

In some embodiments, the method further comprises providing, in real time, tracking information of the destination to the remote user equipment.

In some embodiments, the remote user equipment is configured to: receive a user selection of the destination from within one or more images displayed on the remote user equipment, and generate the destination information of the destination based on the user selection of the destination.

According to another aspect of the present invention, a method for controlling an unmanned aerial vehicle (UAV) is provided. The method includes: displaying, in substantially real-time, via a display, one or more images captured by an imager connected to the UAV, receiving, via an input device, a user selection of a target from within the one or more substantially Real-time displayed images, generating target information of the target based at least in part on the user selection of the target, and providing the target information to the UAV so as to allow the UAV to autonomously track the target according to the target information.

According to another aspect of the present invention, a system for controlling an unmanned aerial vehicle (UAV) is provided. The system includes: a display configured to display substantially in real time one or more images taken by an imager connected to the UAV, an input device configured to provide user selection of a target from within at least one of the one or to receive the plurality of images displayed on the display, one or more processors that, individually or collectively, are configured to generate destination information of the destination based at least in part on the user selection of the destination, and a sender configured thereto to provide the destination information to the UAV so as to allow the UAV to autonomously track the destination according to the destination information.

According to another aspect of the present invention, an apparatus for controlling an unmanned aerial vehicle (UAV) is provided. The apparatus comprises: a display configured to display substantially in real time one or more images taken by an imager connected to the UAV, an input device configured to provide user selection of a target from within at least one of the one or to receive the plurality of images displayed on the display, one or more processors that, individually or collectively, are configured to generate destination information of the destination based at least in part on the user selection of the destination, and a sender configured thereto to provide the destination information to the UAV so as to allow the UAV to autonomously track the destination according to the destination information.

In some embodiments, the destination information includes initial destination information.

In some embodiments, the initial target information includes an initial position or an initial size of the target within an image captured by the imaging device.

In some embodiments, the destination information is generated based on the user selection of the destination.

In some embodiments, the destination information includes destination type information.

In some embodiments, the destination type information includes color, texture, or pattern information.

In some embodiments, the destination type information is generated based on the user selection of the destination.

In some embodiments, the destination information includes expected destination information.

In some embodiments, the expected destination information is generated based on the user selection of the destination.

In some embodiments, the expected target information includes an expected position or size of the target within an image captured by the imaging device.

In some embodiments, the destination information includes unintended destination information.

In some embodiments, the input device includes a touch screen, a joystick, a keyboard, a mouse, a pen, or a portable device.

In some embodiments, the user selection of the target is achieved by a user selecting an area of the at least one of the one or more images displayed on the display, the selected area corresponding to the destination.

In some embodiments, user selection of the target is achieved by a user directly touching an area of the at least one of the one or more images displayed on the display, the touched area corresponding to the destination.

In some embodiments, the user selects the area using a pen, a mouse, a keyboard, or a portable device.

In some embodiments, selecting the area includes touching, swiping, circling, or clicking in the area.

In some embodiments, the one or more processors, individually or in common, are further configured to display on the display, in response to the user selection of the destination, the selected destination with a selection indicator, the selection indicator indicating that the destination has been selected by the user is.

In some embodiments, the one or more processors, individually or in common, are further configured to receive tracking information regarding the destination and, based on the tracking information, to display the selected destination with a tracking indicator within one or more subsequent images captured by the imaging device wherein the tracking indicator indicates, in substantially real-time, that the target is being tracked by the UAV in accordance with the target information.

According to another aspect of the present invention, a method for controlling an unmanned aerial vehicle (UAV) is provided. The method includes: receiving, in substantially real-time, one or more images captured by an imager connected to the UAV, receiving user-specified destination information of a target that has a predetermined position or a predetermined size of the target within an image of the one or more images providing the user-specified destination information to the UAV and displaying the one or more images and a tracking pointer associated with the destination within the one or more images, wherein the tracking indicator indicates that the target is substantially through the UAV is tracked in real time according to the user-specified destination information.

According to another aspect of the present invention, an apparatus for controlling an unmanned aerial vehicle (UAV) is provided. The apparatus includes: a receiver configured to receive, in substantially real-time, one or more images captured by an imager connected to the UAV, an input device configured to provide user-specified destination information of a target having a predetermined position or include a predetermined size of the destination within an image of the one or more images, receive a sender to provide the user-specified destination information for the UAV, and a display configured to store the one or more images and an image with the goal of displaying a tracking indicator associated with the target within the one or more images, wherein the tracking indicator indicates that the target is being tracked by the UAV in substantially real time according to the user specified target information.

According to another aspect of the present invention, a system for controlling an unmanned aerial vehicle (UAV) is provided. The system includes: a receiver configured to receive, in substantially real-time, one or more images captured by an imager connected to the UAV, an input device configured to provide user-specified destination information of a target having a predetermined position or include a predetermined size of the destination within an image of the one or more images, receive a sender to provide the user-specified destination information for the UAV, and a display configured to store the one or more images and an image with the goal of displaying a tracking indicator associated with the target within the one or more images, wherein the tracking indicator indicates that the target is being tracked by the UAV in substantially real time according to the user specified target information.

In some embodiments, the input device is further configured to receive one or more commands configured for speed, position, alignment or position of the UAV or to control one or more operating parameters of the imager.

In some embodiments, the imaging device is connected to the UAV via a carrier configured to allow the imaging device to rotate along at least one axis relative to the UAV, and wherein the input device is further configured to execute one or more commands receive, which are adapted to control a speed, position, orientation or position of the carrier.

In some embodiments, the carrier is configured to allow the imager to rotate about at least two axes relative to the UAV.

In some embodiments, the target is tracked by the imaging device in accordance with the target information via automatic adjustment to at least one of the UAV, the carrier, or the imaging device.

In some embodiments, a second input device is included and configured to receive one or more commands configured to control a speed, position, orientation, or location of the UAV or one or more operating parameters of the imager.

In some embodiments, the imaging device is connected to the UAV via a carrier configured to allow the imaging device to rotate along at least one axis relative to the UAV, and the system further includes a second input device configured therefor is to receive one or more commands adapted to control a speed, position, orientation or attitude of the wearer.

In some embodiments, the one or more operating parameters of the imager include focal length, zoom level, imaging mode, image resolution, focus, depth of field, exposure, intensity, or field of view.

In some embodiments, the target is tracked by the imaging device in accordance with the target information via automatic adjustment to at least one of the UAV, the carrier, or the imaging device.

In some embodiments, the predetermined position includes an initial position of the target.

In some embodiments, the predetermined size includes an initial size of the target.

In some embodiments, the predetermined position includes an expected position of the target.

In some embodiments, the predetermined size includes an expected size of the target.

In some embodiments, the destination information further includes destination type information.

In some embodiments, the destination information is generated based on user selection of the destination.

In some embodiments, the tracking indicator includes a geometric shape, a check mark, or an arrow.

In some embodiments, the geometric shape includes a circle, a rectangle, or a triangle.

In some embodiments, the target is tracked by the imaging device in accordance with the target information via automatic adjustment to at least one of the UAV or the imaging device.

It should be understood that various aspects of the invention may be perceived individually, in common or in combination with one another. Various aspects of the invention described herein may be applied to any of the particular applications set forth below or to any other types of moving objects. Any description herein of aircraft, such as unmanned aerial vehicles, may apply and be used for any movable object, such as any vehicle. In addition, the systems, devices, and methods disclosed herein in the context of air movement (eg, flight) may also be applied in the context of other types of movement, such as ground or water movement, underwater movement, or space movement.

Additional aspects and advantages of the present disclosure will be readily apparent to those skilled in the art from the The following detailed description will be readily apparent, with only exemplary embodiments of the present disclosure shown and described, simply to illustrate the best mode contemplated for carrying out the present disclosure. As will be appreciated, the present disclosure is capable of other and different embodiments, and its various details are capable of modification in various obvious respects, all without departing from the disclosure. Accordingly, the drawings and description are to be considered illustrative rather than limiting.

INCLUSION BY REFERENCE

All publications, patents and patent applications mentioned in this specification are hereby incorporated by reference to the same extent as if each individual publication, patent and patent application were specifically and individually indicated as being incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description, which sets forth illustrative embodiments in which the principles of the invention are used, and the accompanying drawings, of which:

1 an exemplary tracking system, according to embodiments illustrated,

2 illustrate an exemplary flow of image related data between constituents in a tracking system, according to embodiments;

3 illustrate an exemplary flow of control data between constituents in a tracking system, according to embodiments;

4 illustrate an example process for implementing goal tracking, according to embodiments;

5 10 illustrates an exemplary configuration of a moving object, a carrier, and a payload, according to embodiments;

6 an exemplary tracking method for maintaining an expected position of a target, according to embodiments, illustrated,

7 an exemplary tracking method for maintaining an expected size of a target, according to embodiments, illustrated,

8th Figure 11 illustrates another example process for implementing target tracking, according to embodiments;

9 an exemplary process for controlling a moving object to navigate and track, according to embodiments, illustrated,

10 an exemplary process for selecting a target, according to embodiments illustrated,

11 an exemplary process for viewing a tracked goal, according to embodiments, illustrated,

12 an exemplary control station for controlling a moving object, according to embodiments, illustrated,

13A FIGS. 1A to C illustrate exemplary methods for selecting a target using a user interface, according to embodiments;

14 a UAV, according to embodiments illustrated,

15 a movable object including a carrier and a payload, according to embodiments illustrated,

16 an exemplary system for tracking a moving object, according to embodiments illustrated,

17 an exemplary system for controlling a moving object, according to embodiments illustrated,

18 illustrates an exemplary use case for the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides systems and apparatus associated with tracking Unmanned Aerial Vehicles (UAV). A UAV may be configured to receive destination information regarding a destination to be followed by an imager connected to the UAV from a control station. The target information can be used by the UAV to automatically cause the Imaging device tracks the target so as to maintain a predetermined position and / or size of the target within one or more images captured by the imager. The tracking of the target may be performed while the UAV is being controlled to navigate according to user commands and / or predetermined navigation lanes. The control station may be configured to display images from the imaging device as well as to allow user input regarding the destination information.

Using the methods and systems provided herein, a single user may control both the navigation of a UAV and the tracking of a target substantially simultaneously without the assistance of an additional person. The user can use a user interface of a control station to specify the target to be tracked and / or the type of target to track. Such user-specified destination information may be communicated to the UAV, which may autonomously track the destination, for example using an onboard UAV imaging device. Images captured by the imaging device (eg, images and / or videos) may be communicated to the control station in real time for display, playback, storage, or other purposes. The user can also change or set the target to be tracked in real time using the control station. Advantageously, such autonomous tracking can be performed by the UAV while the user is doing other activities, such as controlling the navigation of the UAV or other activities.

For example, a user may configure the UAV to track it itself when doing a variety of activities, such as walking or cycling. It may specify itself as the tracking target for the UAV using the user interface provided by the control station. For example, it may select itself as the destination of an image displayed on a user interface of the control station, for example, using a touch screen. Once the target information is transmitted to the UAV, the user may be relieved of the low level activities associated with manually tracking a target, such as adjusting the UAV, the carrier, or the imaging device. Instead, it may focus on other activities, such as cycling, as the UAV automatically tracks them in real time based on the provided destination information using the methods provided herein. For example, the location, position, speed, zoom, and other aspects of the UAV and / or the imaging device may be automatically adjusted to ensure that the user maintains a designated position and / or size within the images captured by the imaging device. Images captured during the tracking process (eg, videos or images) may be streamed to the control station in real time or substantially in real time for display, playback, storage, or other purposes. All of the above can be achieved by a person in a relatively simple manner, making it easier for users to accomplish previously difficult-to-accomplish tasks.

Such tracking methods and systems, as described herein, advantageously facilitate the automation of the low-level control portion of the tracking process so as to reduce the effort required and the errors resulting from manual tracking. At the same time, the tracking methods and systems described herein allow users to maintain high-level tracking control, if desired, (eg, by specifying the type of target being tracked).

1 illustrates an exemplary tracking system 100 , according to embodiments. The system 100 closes a moving object 101 and a control station 112 one. The system 100 Can be used to one or more goals 116 to pursue. Although the moving object 101 as an unmanned aerial vehicle (UAV), this mapping is not intended to be limiting, and any suitable type of moving object may be used, as described herein. One skilled in the art would recognize that any of the embodiments described herein in the context of aircraft systems may be applied to any suitable moving object.

In some embodiments, the moving object 101 a carrier 102 and a payload 104 lock in. The carrier 102 it can allow that payload 104 in relation to the moving object 101 emotional. For example, it may be the wearer 102 allow for the payload 104 rotates one, two, three or more axes. Alternatively, it may be the carrier 102 allow for the payload 104 rotates along one, two, three or more axes. The axes for the rotational or displacement movement may be perpendicular to each other or not.

In some embodiments, the payload 104 so rigidly to the moving object 101 coupled or connected to the same, that the payload 104 essentially immobile in relation to the moving object 101 remains. For example, the carrier 102 , the moving object and the payload 104 connects, does not allow the payload 104 relative to the moving one 101 object 101 emotional. Alternatively, the payload 104 directly to the moving object 101 be coupled without requiring a carrier.

In some embodiments, the payload 104 one or more sensors for monitoring or tracking one or more targets 116 lock in. Examples of such a payload may include an image capture device or imaging device (eg, a camera or a camera recorder, an infrared imaging device, an ultraviolet imaging device, or the like), an audio capture device (eg, a parabolic mirror microphone), an infrared imaging device, or the like lock in. Any suitable sensor (s) may enter the payload 104 be included to detect any visual, acoustic, electromagnetic or any other desirable signals. The sensors may provide static sensor data (eg, a photograph) or dynamic sensor data (eg, a video). The sensors can acquire sensor data continuously in real time or at high frequencies.

In various embodiments, this may be due to the moving object 101 pursued goal 116 any natural or artificial objects or structures, such as landscapes (eg mountains, vegetation, valleys, lakes or rivers), buildings, vehicles (eg, airplanes, ships, cars, trucks, buses, vans or motorcycles) , The goal 116 may also include living subjects, such as persons or animals. The goal 116 may be moving or immobile relative to any suitable reference frame. The frame of reference may be a relatively fixed frame of reference (for example, the surrounding environment or the earth). Alternatively, the frame of reference may be a moving frame of reference (eg, a moving vehicle). In various embodiments, the destination may be 116 include a passive goal or an active goal. An active target may be configured to communicate information about the target, such as the GPS position of the target, to the moving object. The information may be communicated via wireless communication from a communication unit of the active destination to a communication unit of the mobile object. Examples of an active destination may include a friendly vehicle, building, troop, or the like. A passive target is not configured to convey information about the target. Examples of a passive target may include a neutral or enemy vehicle, buildings, troop, and the like.

The moving object 101 may be configured to receive control data and the control station 112 may be configured to provide it. The control data can be used to directly or indirectly affect aspects of the moving object 101 to control. In some embodiments, the control data may include navigational commands for controlling navigation parameters of the mobile object, such as the position, velocity, orientation, or location of the mobile object 101 , lock in. The control data can be used to control the flight of a UAV. The control data may include the operation of one or more drive units 106 affecting the flight of the UAV. In other cases, the control data may include commands for controlling individual components of the mobile object 101 lock in. For example, the control data may be information for controlling the operations of the wearer 102 lock in. For example, the control data may be used to control an actuating mechanism of the carrier 102 to control, so an angular and / or linear movement of the payload 104 in relation to the moving object 101 to induce. As another example, the control data may be used to determine the motion of the wearer 102 without controlling the payload. As another example, the control data may be used to specify one or more operating parameters for the payload 104 such as taking still or moving pictures, zooming in or out, turning on or off, switching picture modes, changing the picture resolution, changing the focus, changing the depth of field, changing the shutter speed, changing the light level, changing the angle of view or field of view or the like. In other embodiments, the control data may be used for a sensor system (not shown), a communication system (not shown), and the like of the moving object 101 to control.

In some embodiments, the control data may be from the control station 112 Include destination information. In some cases, the destination information may include characteristics of a specific destination, such as an initial location (eg, coordinates) and / or size of a destination within one or more of the locations by the moving object 101 include captured imaging of captured images. Additionally or alternatively, the destination information may include destination type information, such as characteristics of a type or category of destinations, including Color, texture, pattern, size, shape, dimension, and the like. The destination information may include a data representation of an image of the destination. This may include an image of the target in a field of view. The field of view may be defined or enclosed by the images taken by the imaging device.

The destination information may also include expected destination information. The expected target information specifies the characteristics that are expected to satisfy the tracked target in the images captured by the imager. The expected target information may be used to set the moving object, the carrier, and / or the imaging device so that the tracked target in one or more images will maintain a look-up for the expected target information. For example, the target may be tracked to maintain an expected position and / or size within one or more images captured by the imaging device. For example, the expected position of the tracked target may be near the center of the image or off-center. The expected size of the tracked target may be around a certain number of pixels. The expected destination information may be the same as the initial destination information or not. In various embodiments, the expected destination information may or may not be provided by the control station. For example, the expected destination information in the control logic executed by a processing unit aboard the mobile object may be hard-coded, stored in a data memory at and / or remote from the location of the mobile object, or obtained from other suitable sources.

In some embodiments, the destination information (including specific destination information and destination type information) may be at least in part at the control station 112 be generated. Additionally or alternatively, the destination information may be generated based on data from other sources. For example, the destination type information may be derived based on previous images and / or data extracted from local or remote data stores. The images could previously be sent to the moving object 101 connected imaging device 104 or other facilities. The pictures could be computer generated. Such target type information may be selected by the user and / or automatically provided by presetting the moving object.

The destination information can be determined by the moving object 101 be used to one or more goals 116 to pursue. The tracking and any other associated data processing may be performed, at least in part, by one or more processors on board the mobile object 101 be performed. In some embodiments, the target information may be used to determine the target to be tracked by the moving object 116 to identify. Such identification of the target may be based on the initial target information including the specific characteristics of a particular target (eg, initial coordinates of a target within an image captured by the mobile object) or general characteristics of a type of target (eg, color and / or texture of the target (s) to be tracked). In some cases, the destination identifier may include any suitable image recognition and / or matching algorithms. In some embodiments, the destination identifier includes comparing two or more images to determine, extract, and / or match features contained therein.

Once a target is identified, the expected target information may be used to detect a deviation from expected characteristics of the target, such as expected position and / or size. In some embodiments, current target characteristics or information may be determined based on one or more images taken by the mobile object. The current destination information may be compared with the expected destination information provided by the control station to determine the deviation therefrom. A change in the position of the target can be detected by comparing coordinates of the target (eg, the coordinates of a center of the target) within an image with the coordinates of the expected target position. A change in the size of the target can be detected by comparing the size of the area covered by the target (eg, in pixels) with the expected target size. In some embodiments, a change in size may be detected by detecting an orientation, limitations, or other characteristics of the target.

At least in part, based on the detected deviation, control signals may be generated (for example, by one or more processors aboard the mobile object) effecting a setting that substantially corrects the detected deviation. Therefore, the adjustment may be used to provide one or more expected target characteristics (eg, target position and / or size) within the target essentially maintain moving object-captured images. In some embodiments, the adjustment may be performed substantially in real time because the moving object is executing user-provided navigation commands (eg, hovering or moving) and / or predetermined navigation lanes. The adjustment can also be made substantially in real time because the imaging device captures one or more images. In some embodiments, the adjustment may be generated based on other information, such as sensor data sensed by one or more sensors aboard the mobile object (eg, proximity sensor or GPS sensor). For example, positional information of the tracked target may be detected by a proximity sensor and / or provided by the target itself (eg, GPS position). Such position information, in addition to the detected deviation, may be used to generate the adjustment.

The adjustment may relate to the moving object, the carrier and / or the payload (eg the imaging device). For example, the adjustment may cause the movable object and / or the payload (eg, the imager) to change position, location, orientation, angular and / or linear velocity, or the like. The adjustment may cause the carrier to move the payload (eg, the imager) relative to the moving object, such as one or two, three, or more axes, or along the same. Additionally, the adjustment may include adjustment to the zoom, focus, or other operating parameters of the payload (eg, the imager) itself (eg, zooming in / out).

In some embodiments, the adjustment may be generated based at least in part on the nature of the detected deviation. For example, a deviation from the expected target position may require rotation of the moveable object and / or payload (eg, over the carrier) about one, two, or three axes of rotation. As another example, a deviation from the expected target size may require a displacement movement of the moving object along a suitable axis and / or changes in the zoom of the imaging device (eg, zooming in or out). For example, if the current or actual target size is less than the expected target size, the moving object may need to be moved closer to the target and / or the imager may need to be zoomed in to the target. On the other hand, if the current or actual target size is greater than the expected target size, the moving object may need to be moved farther away from the target and / or the imager may have to be zoomed out of the target.

In various embodiments, the adjustment to substantially correct the deviation from the expected target information may be achieved by controlling one or more controllable objects, such as the moving object, the carrier, the imager, or any combination thereof via control signals. In some embodiments, the controllable objects may be selected to implement an adjustment, and the corresponding control signals may be generated based at least in part on the configurations or set values of the controllable objects. For example, an adjustment involving rotation about two axes (eg, yaw and pitch) can only be achieved by a corresponding rotation of the moveable object about the two axes if the imager is rigidly coupled to the moveable object consequently can not move in relation to the moving object. This may be the case when the imaging device is directly coupled to the mobile object or when the imaging device is coupled to the mobile object via a carrier that does not allow for relative movement between the imaging device and the mobile object. The same biaxial adjustment can be achieved by combining an adjustment on both the moveable object and the support if the support allows the imaging device to rotate about at least one axis relative to the moveable object. In this case, the carrier can be controlled to implement the rotation about one or two of the two axes required for the adjustment, and the movable object can be controlled to implement the rotation about one or two of the two axes. For example, the carrier may include a uniaxial universal joint that allows the imaging device to rotate about one of the two axes required for adjustment while achieving rotation about the remaining axis through the movable object. Alternatively, the same biaxial adjustment may be achieved by the carrier alone if the carrier allows the imager to rotate about two or more axes relative to the moving object. For example, the carrier may include a biaxial or triaxial universal joint.

As another example, an adjustment to correct for a change in the size of the target may be achieved by controlling the zooming in / out of the imaging device (eg, if the imaging device supports the required zoom level) by controlling the Moving the moving object (eg, to get closer or farther away from the target) or by a combination of zooming in / out of the imaging device and moving the moving object. A processor on board the mobile object may determine which object or combination of objects to set. For example, if the imager does not support a zoom level required to maintain the required size of the target within an image, the moveable object may be controlled to move, instead of or in addition to adjusting the zoom level of the imager ,

In some embodiments, the adjustment may be implemented by considering other constraints. For example, in cases where the navigation path of the mobile object is predetermined, the adjustment by the carrier and / or the imaging device can be implemented without affecting the movement of the mobile object. For example, the navigation path of the mobile object may be predetermined if a remote user controls navigation of the mobile object via a control station or if the mobile object (eg, autonomous or semi-autonomous) navigates according to a pre-stored navigation path.

Examples of other limitations may include a maximum and / or minimum limit for rotational angle, angular and / or linear velocity, operating parameters, and the like for the moving object, the carrier, and / or the payload (eg, the imager). Such maximum and / or minimum thresholds may be used to limit the range of adjustment. For example, the angular velocity of the mobile object and / or the imaging device may be capped about a particular axis by a maximum angular velocity that is permissible for the moving object, the carrier, and / or the payload (eg, the imaging device). As another example, the linear velocity of the moving object and / or the imaging device may be capped by a maximum linear velocity that is permissible for the moving object, the carrier, and / or the payload (eg, the imaging device). As another example, the focal length setting of the imaging device may be limited by the maximum and / or minimum focal length of the particular imaging device. In some embodiments, such limits may be predetermined and depend on the particular configuration of the moving object, the carrier, and / or the payload (eg, the imager). In some cases, such configurations (eg, by a manufacturer, administrator, or user) may be configurable.

In some embodiments, the moving object 112 be configured to provide data, and the control station 112 may be configured to receive them, such as by sensors aboard the moving object 101 sensed sensor data and tracking data or information used to indicate characteristics of one or more targets tracked by the moving object. Examples of sensor data can be through a through the moving object 101 supported imaging device, or other data acquired by other sensors. For example, real-time or near real-time video from the moving object 101 and / or the payload 104 (eg the imaging device) to the control station 112 be streamed. The sensor data may also include data collected by Global Positioning System (GPS) sensors, motion sensors, inertial sensors, proximity sensors, or other sensors. Examples of tracking information may include relative or absolute coordinates and / or size of the target within one or more images received by the mobile object, changes in the destination between successive frames, GPS coordinates or other location information of the destination, and the like. In some embodiments, the tracking information may be provided by the control station 112 used to indicate the target as being tracked (eg, via a graphical tracking indicator, such as a box around the target). In various embodiments, the data received by the control station may process raw data (eg, raw sensor data as detected by the sensors) and / or processed data (eg, tracking information such as by one or more processors aboard the mobile object ) lock in.

In some embodiments, the control station 112 at one of the moving object 101 , the carrier 102 and / or the payload 104 remote or distant location. The control station 112 can be arranged on a support platform or attached to the same. Alternatively, the control station 112 be a hand-held or portable device. For example, the control station 112 include a smartphone, a tablet, a laptop, a calculator, glasses, gloves, a helmet, a microphone, or suitable combinations thereof.

The control station 112 may be configured for the movable object 101 display received data via an indicator. The displayed data can be sensor data, such as For example, images (eg, still images and videos) captured by a moving object 101 worn imaging device. The displayed data may also include tracking information displayed separately from the image data or displayed on top of the image data. For example, the display may be configured to display the images, with the target including a tracking indicator such as a box, a circle, or any other geometric shape that encloses, is displayed, or highlighted on the tracked target. In some embodiments, the images and the tracking indicator are displayed in substantially real time when the image data and tracking information is received from the mobile object and / or when the image data is captured. In other embodiments, the display may be provided after a certain delay.

The control station 112 may be configured to receive a user input via an input device. The input device may include a joystick, a keyboard, a mouse, a stylus, a microphone, an image or motion sensor, an inertial sensor, and the like. Any suitable user input may be used to interact with the control station, such as manually entered commands, voice control, gesture control, or position control (eg, via movement, position or tilt of the control station). For example, the control station 112 be configured to allow a user, a state of the moving object, the carrier, the payload or any component thereof by manipulating a joystick, changing an orientation of the control station, the interaction with a graphical user interface using a Keyboard, a mouse, a finger or a stylus or by the use of any other suitable method.

The control station 112 may also be configured to allow a user to enter destination information using any suitable method. In some embodiments, the control station 112 be configured to allow a user to immediately select a destination from one or more images (eg, video or snapshot) that are displayed. For example, the user may select an image by directly touching the screen using a finger or stylus or a selection using a mouse or a joystick. The user can draw around the target, touch the target in the image, or otherwise select the target. Machine vision or other techniques can be used to determine a boundary of the target. Otherwise, user input may define the bound of the target. One or more destinations can be selected at once. In some embodiments, the destination is displayed with a selection indicator to indicate that the user has selected the destination for tracking. In some other embodiments, the dispatching station may be configured to allow a user to enter or select destination type information such as color, texture, shape, dimension, or other characteristics associated with a desired destination. For example, the user may type in the destination type information, select such information using a graphical user interface, or use any other suitable method. In some other embodiments, the destination information may be obtained from sources other than the user, such as a remote or local data store, other computing devices that are operatively connected to, or otherwise in communication with, the control station.

In some embodiments, the control station allows a user to choose between a manual tracking mode and an automatic tracking mode. When the manual tracking mode is selected, a user can specify a specific target to track. For example, the user may manually select a destination from one displayed by the control station. The specific destination information (eg, coordinates and / or size) associated with the selected destination is then provided to the mobile object as the initial destination information of the destination. On the other hand, when the automatic tracking mode is selected, the user does not specify a specific target to be tracked. Instead, the user may specify descriptive information about the type of target to track, for example, the user interface provided by the control station. The mobile object may then use the initial destination information of a specific destination or destination type information to automatically identify the destination to be tracked and subsequently track the identified destination.

In general, providing specific destination information (eg, initial destination information) requires more user control of tracking the destination and less automated processing or calculation (eg, image or destination recognition) by a processing system aboard the mobile object. On the other hand, providing destination type information requires less user control of the tracking process, but more on-board processing system performed calculation. The appropriate allocation of control over the tracing process between the user and the moving object can be adjusted depending on a variety of factors, such as the environment of the moving object, the speed or height of the moving object, user preferences and available computational resources (e.g. , CPU or memory) on board and / or outside the moving object or the like. For example, the user may be given relatively more control when navigating the moving object in a relatively complex environment (eg, with numerous buildings or obstacles or inside) than if the moving object were in a relatively simple environment (e.g. open space or outside). As another example, more control may be assigned to the user when the moveable object is at a lower elevation than when the moveable object is at a higher elevation. As yet another example, more control may be assigned to the moving object if the moving object is equipped with a high speed processor configured to perform complex calculations relatively quickly. In some embodiments, the assignment of control over the tracking process between the user and the moving object may be dynamically adjusted based on the factors described herein.

The user input may be used, at least in part, to generate control data, such as described herein. The control data may be generated by the control station, the moving object, a third device, or any combination thereof. For example, the handling of the user of a joystick or control station, or an interaction with a user interface, may be translated into predetermined control commands for changing a state or parameter of the moving object, the carrier, or the payload. As another example, a selection of the user of a target within an image displayed by the control station may be used to provide initial and / or expected destination information for tracking purposes, such as an initial and / or expected location and / or size of the destination , to create. Alternatively, or additionally, the control data may be generated based on information obtained from non-user sources, such as remote or local data storage, other data processing equipment effectively connected to the control station, or the like.

2 illustrates an exemplary flow 200 image-related data between components in a tracking system, according to embodiments. In some embodiments, image related data includes raw or processed data as well as data extracted or derived from the image data, such as tracking information of targets. As illustrated, raw image data can be captured by an image sensor 204 be recorded. The image sensor 204 may be configured to convert optical signals into electronic signals. The image sensor 204 For example, charge-coupled devices (CCDs), active pixel sensors using complementary metal-oxide-semiconductor (CMOS) or n-channel metal oxide semiconductor (N-type metal oxide) semiconductor devices may be used. oxide-semiconductor - NMOS, live MOS) technologies or any other type of sensors. The image sensor may be connected to a moving object, such as a UAV. For example, the image sensor may be part of an imaging device (eg, a camera) carried by a UAV with or without a carrier. The image sensor and / or imaging device may be configured to capture images, videos, or any other image data having any suitable parameters, such as width, height, aspect ratio, mega pixel number, resolution or quality, and the like. For example, the imager may be configured to capture high resolution or ultra high resolution video (e.g., 720p, 1080p, 1440p, 2000p, 2160p, 2540p, 4000p, 4320p, and so forth).

The through the image sensor 204 Captured image data can be stored in a data storage device 202 get saved. The data storage device 202 It can be based on semiconductor magnetic, optical or any other suitable technology and may include flash memories, USB drives, memory cards, solid-state drives (SSDs), hard disk drives (HDDs), floppy disks, optical disks, magnetic tapes and the like. For example, the data storage device 202 removable storage devices that are detachably connectable to an imaging device, such as memory cards of any suitable formats, such as PC Card, Compact Flash, SmartMedia, Memory Stick, Memory Stick Duo, Memory Stick PRO Duo, Miniature Card, Multimedia Card (MMC), Reduced Size Multimedia Card (RS-MMC), MMCmicro Card (MMCmicro), PS2 Card, Secure Digital (SD) Card, SxS, Universal Flash Storage (UFS), miniSD, microSD, xD-Picture Card, Intelligent Stick (iStick), Serial Flash Modules (SFM), NT Card, XQD Card and the like. The data storage device 202 can also external hard drives, optical drives, tape drives, floppy drives and others include suitable storage devices that can be effectively connected to the imaging device.

The through the image sensor 204 Captured image data may be transmitted through an image transmission module 206 to the control station 212 be transmitted. In some embodiments, the image data may be compressed or otherwise processed before passing through the image transmission module 206 be transmitted. In other cases, the image data may not be compressed or processed before being transmitted. The transmitted image data can be displayed on the control station 212 be displayed so that a user who is the control station 212 operated, viewing the image data and / or on the basis of the image data with the control station 212 can interact.

The through the image sensor 204 Captured image data can by a preprocessing unit 208 be preprocessed. The preprocessing unit 208 may include any hardware, software, or a combination thereof. Examples of the preprocessing unit 208 may include a field programmable gate array (FPGA). The preprocessing unit 208 can be effective on the image sensor 204 be connected to pre-process the raw image data before the image data is processed to extract specific information. Examples of by the preprocessing unit 208 In order to ensure the accuracy of the image coordinate system, resampling may include noise reduction, contrast enhancement, scale space representation, and the like.

The image data, as by the preprocessing unit 208 , can continue through a processing unit 210 which are effectively (for example, via a General Purpose Memory Controller (GPMC) connection) to the preprocessing unit 208 connected. The processing unit 210 may include one or more ARM processors. The processing unit 210 may be configured to perform any suitable embodiments of the methods described herein. Examples of by the processing unit 210 Tasks performed may include feature extraction at any suitable level of complexity, image segmentation, data validation, image recognition, image registration, image matching, and the like. In some embodiments, the processing unit generates 210 Tracking information regarding a target being tracked by the moving object. The tracking information may be generated based on image processing by a processing unit of the mobile object and / or based on destination information provided by the control station. The tracking information may include, for example, the position, size, or other characteristics of a target within one or more images.

The tracking information, as by the processing unit 210 determined, via a communication module (not shown), the control station 212 to be provided. In some embodiments, the communication module may be from the image transmission module described above 206 be separate. In other embodiments, the communication module may include the image transmission module 206 include or be included in it. Any communication means may be for the communication module and / or the image transmission module 206 used, such as cable communication or wireless communication. For example, the communication module and / or the image transmission module 206 one or more of local area networks (LANs), wide area networks (WANs), infra-red, wireless, Wi-Fi, point-to-point (P2P) networks, telecommunications networks, cloud communications, and use the same. Wireless communication can be proximity-dependent or proximity-independent. In some embodiments, communications may or may not be required for communications. The communication module may also be configured to transmit and / or receive, in addition to the tracking information, sensor data from other sensors on board the moving object, position and / or motion information determined by processing the sensor data, predetermined control data, user commands from remote control stations, and the like ,

In some embodiments, the image data, such as through the image transfer module 206 provided by the tracking information, such as by the processing unit 210 can be created, augmented, or otherwise combined with them to show a target within a target indicator (such as a circle or box around the target). A user may view the extended image data to see the target as it is being tracked by the imaging device. The user can also interact with the control station based on the enhanced image data. For example, the user may select a different destination to track from the extended image data (eg, by touching a portion of the screen that corresponds to the other destination).

In some embodiments, the tracking information of the control station 212 in response to a request from the control station 212 to be provided. For example, the control station 212 such tracking information only when a user selects a particular viewing mode (eg, a preview mode) with the tracked target highlighted or otherwise displayed. Alternatively or additionally, the tracking information of the control station 212 without any request from the control station 212 to be provided. For example, the tracking information may be periodically (e.g., every 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second or 2 seconds) of the control station 212 be stored.

3 illustrates an exemplary flow 300 control data between components in a tracking system, according to embodiments. As discussed above, control data may include information used by a moving object (eg, a UAV) to track a destination, as well as other data for controlling various aspects of the moving object or a component thereof. The control data may be provided by a user 302 be generated with a control station 304 interacts. The generated control data may contain specific destination information (eg, initial destination information). 306 and destination type information 308 lock in.

The specific destination information 306 may include characteristics about a specific destination, such as coordinates (eg, pixel coordinates), size, and the like. In some embodiments, the specific destination information 306 are generated when a user has a specific target to be tracked over one by the control station 304 provided user interface selects or specifies. For example, the control station may allow a user to choose between a manual tracking mode and an automatic tracking mode. When the manual tracking mode is selected, a user may specify a specific target to be tracked (eg, by selecting a target from one or more displayed images). Based on the user selection, the specific destination information can be generated.

The destination type information 308 may include information that describes a type of target to pursue, rather than information about a specific target. Such target type information may include various target characteristics such as color, texture, pattern, size, shape, dimension, and the like. In some embodiments, the destination information (including specific destination information and destination type information) may be responsive, at least in part, to a user input at the control station 304 be generated. For example, the control station may allow a user to choose between a manual tracking mode and an automatic tracking mode. When the automatic tracking mode is selected, a user can input or select target type information. Additionally or alternatively, the destination information may be generated based on data from other sources. For example, the destination type information may be derived based on previous images and / or derived from local or remote data stores. In some embodiments, predefined target types or patterns may be presented to the user for selection. In some other embodiments, the predefined target types or patterns may be automatically provided by default to the moving object without user intervention.

The destination information may optionally include expected destination information, such as described herein. The expected destination information may or may not overlap with the initial destination information.

The destination information (including the specific destination information 306 and the destination type information 308 ), for example, via a communication module (not shown), a processing system 310 to be provided. The processing system 310 may be on board a moving object (eg a UAV). The processing system 310 can be a data processing unit 312 and a command control unit 314 lock in. The data processing unit 312 may be configured to perform any embodiments of the methods described herein. For example, the data processing unit 312 be configured to specify a destination based on destination information (including the specific destination information 306 and / or the destination type information 308 ), to determine a deviation from the destination information, and the like. The data processing unit 312 For example, a preprocessing unit and / or a processing unit, such as similar to those in FIG 2 described preprocessing unit 208 or processing unit 210 , lock in. For example, the data processing unit 312 include an FPGA and / or one or more ARM processors.

The data processing unit 312 can be effective to a command control module 314 be connected, which is configured to control a state of the movable object. The command control module 314 may be configured to perform any embodiments of the methods described herein. For example, the command control module 314 configured for control commands or signals 316 to generate a component of the mobile object, so as to obtain the destination substantially on the basis of Results from the data processing unit 312 to pursue.

The control commands 316 may also include commands for the moving object driving mechanisms to adjust the spatial arrangement, speed and / or acceleration of the moving object with respect to up to six degrees of freedom, to detect a detected deviation of the target with respect to its position and / or size in one or to correct several of the images. The control commands 316 Also, commands may be included to adjust the state of a wearer so as to adjust the spatial location, velocity, and / or acceleration of a payload (eg, imager) carried by the moving object over the wearer.

The control commands 316 Also, commands may be included to adjust the state of a wearer so as to adjust the spatial location, velocity, and / or acceleration of a payload (eg, imager) carried by the moving object over the wearer.

The control commands 316 may also include commands for setting one or more payload operating parameters, such as taking still or moving pictures, zooming in or out, turning on or off, switching image modes, changing the image resolution, changing the focus, changing the depth of field, changing the Exposure time, changing the light intensity, changing the viewing angle or field of view, or the like.

Alternatively or additionally, any of the above control commands may be provided directly by the control station 304 the processing system 310 to be provided. For example, the user may be the control station 304 to control the zoom in / out of the imager while the processing system 310 generates the control commands for the moving object and / or the carrier. As another example, the user may directly control the moving object while the processing system 310 generates the control commands for the moving object and / or the imaging device. This may allow a user to focus on controlling the moving object without worrying about tracking the target automatically through the processing system 310 is carried out.

In various embodiments, the assignment of tracking control between the user and the automatic tracking system may vary depending on a variety of factors, such as the environment of the moving object, the speed or height of the moving object, user preferences, and available computational resources (e.g. , CPU or memory) on board and / or outside of the moving object, the like. For example, the user may be given relatively more control when navigating the moving object in a relatively complex environment (eg, with numerous buildings or obstacles or inside) than if the moving object were in a relatively simple environment (e.g. open space or outside). As another example, more control may be assigned to the user when the moveable object is at a lower elevation than when the moveable object is at a higher elevation. As yet another example, more control may be assigned to the moving object if the moving object is equipped with a high speed processor configured to perform complex calculations relatively quickly. In some embodiments, the assignment of control over the tracking process between the user and the moving object may be dynamically adjusted based on the factors described herein.

According to one aspect of the present invention, methods and systems for tracking a target are provided by an imaging device connected to a moving object so as to maintain an expected position and / or size of the target within one or more images captured by the imaging device.

4 illustrates an exemplary process 400 to implement goal tracking, according to embodiments. Aspects of the process 400 may be performed by one or more processors on board and / or outside a moving object as described herein, such as a UAV. Some or all aspects of the process 400 (above any other processes or variations and / or combinations thereof described herein) may be performed under the control of one or more computing / control systems that are configured with executable instructions and may be implemented as code (eg, executable instructions) or multiple computer programs or one or more applications) that are executed together on one or more processors, by hardware or combinations thereof. The code may be stored on a computer-readable storage medium, for example in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable storage medium may be non-volatile. It is not intended that the order in which the operations will be described be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the processes.

The process 400 includes capturing 402 of destination information for one or more destinations. The destination information may be received by a control station, such as described herein. Additionally or alternatively, the destination information may be from a component (eg, a memory) aboard a mobile object (eg, a UAV) or a device remote from the mobile object, such as another mobile object (e.g. another UAV), a server or the like. In some cases, the destination information about a destination may be provided by the destination itself.

In various embodiments, the destination information for a particular known destination may include specific initial destination information or destination type information about a possibly unknown destination (s). Specific destination information includes coordinates (eg, pixel coordinates), size, location, and other information about a destination within one or more images. Specific destination information may be generated based on user interaction with existing image data, such as described herein. For example, specific destination information may be generated when a user selects a particular object as a destination from one or more images displayed to the user. The specific destination information may include the initial location and / or size of the destination when the destination is selected by a remote user from within one or more images.

The target information may also include targeting information such as color, texture, dimension, size, position, and / or any other characteristics about a type or group of potentially unknown or unidentified targets. The target type information can be input specifically by a user. Alternatively, the user may select an already existing target pattern or type (eg, a black object or a round object having a radius greater than or less than a certain value). In some embodiments, the user may provide destination type information by selecting one or more destinations from within one or more images. Features or characteristics of the selected targets may then be extracted and / or generalized to generate the target species information that may be used to identify other targets having similar characteristics or characteristics. In various embodiments, such feature extraction may be performed by a control station, a processing unit on the moving object, or a third device.

The destination information (including specific destination information and destination type information) may optionally include expected characteristics where the destination, if identified, should be old when the destination is being tracked. For example, the destination information may include an expected location of the destination expressed by absolute or relative coordinates within an image. The tracking system may be configured to track the target such that within a predefined tolerance level, the target is maintained at substantially the same expected position over time. Alternatively or additionally, the target information may include an expected size of the target (eg, expressed by a number of pixels occupied by the target). The tracking system may be configured to track the target such that the target is maintained within a predetermined degree of tolerance at substantially the same expected magnitude. Such expected destination information may be the same for a specific destination or different from the initial destination information. The initial destination information is typically used to identify a destination. Once the target is identified, the expected target information can be used to detect any deviation from the expected target information so that such a bias can be corrected. In some cases, the destination information may also include other values, such as a time value or an expiration time, indicating a period during which the destination, if identified, should be tracked, a flag indicating whether the destination information includes specific destination information or destination type information, and the like.

In some embodiments, the expected destination information to which the destination is to be maintained may be provided by the control station. For example, the expected. Destination information based on a user input to the control station or on the basis of a configuration file or a data memory, locally or remotely to the control station generated. In some other embodiments, the expected destination information may be provided by the moving object. For example, the movable object may be configured to preset a target substantially at the center of a Image or to hold certain coordinates of the image. As another example, the movable object may be configured to hold by default the target as captured by the imaging device at a particular size. In yet some other embodiments, the expected destination information may be provided by some other objects or a device external to the moving object.

Once the destination information is received, the process closes 400 identifying 404 a destination based on the destination information, e.g. Based on the destination type information. Any suitable image recognition or identification techniques may be used, including approaches based on CAD-like object models (eg, edge detection, primary sketch, Marr, Mohan and Nevatia, Lowe or Olivier Faugeras), appearance-based models (e.g. using edge matching, divide-and-divide search, grayscale matching, gradient matching, histogram of recipe field reactions or large model bases), feature-based models (eg, using evaluation trees, hypothesis generation and verification, site consistency, lag accumulation, invariance geometric hashing, scale-invariant feature transform (SIFT) or accelerated robustness features (SURF)), genetic algorithms, and the like.

After the target has been identified, subsequent images captured by the imaging device may be monitored to detect a deviation of the target from certain expected target information that the target should maintain as the expected position and / or size 406 , As discussed above, the expected destination information may be provided by a control station (eg, based on user input), a configuration file or memory associated with the mobile object, or from some other sources. The expected destination information may be the same as the initial destination information or not. The deviation of the target can be achieved by comparing the respective position, size, and / or any other suitable characteristics in one or more images with expected characteristics. Any suitable image recognition or identification techniques, as discussed herein, may be used

In some embodiments, the expected destination information is considered to be substantially maintained if the detected deviation is within certain predefined tolerance values. In such cases, the deviation may be considered negligible and no adjustment adjustment is required. Only if the deviation exceeds the predefined tolerance, a corrective adjustment is required. For example, if the current position of the target is within a predetermined number of pixels from the expected coordinates, the deviation may be considered negligible and so no corrective adjustment is required. Similarly, if the current size of the target is within a predetermined number of pixels of the expected size, the deviation may be considered negligible. The predefined tolerance level can be defined by system parameters, configured or otherwise defined by users who operate the control station.

To correct the drift and substantially maintain the expected characteristics of the target, control signals or commands may be generated 408 to adjust the moving object, the carrier and / or the imaging device. In some embodiments, deviations in the position of the target may be corrected via an adjustment to the attitude of the moving object and / or the imaging device (via the carrier), as described in more detail in FIG 6 discussed. For example, such an adjustment may include changing the angular velocity of the moveable object and / or the imaging device about one or more axes of rotation. Variations in the size of the target may be corrected via an adjustment to the position of the moving object and / or to the operating parameters of the imaging device, such as in more detail in FIG 7 discussed. For example, such an adjustment may include changing the linear velocity of the moveable object along an axis. Alternatively, or additionally, the adjustment may include changing the zoom, focus, or other characteristics associated with the imager.

In some embodiments, the setting may be limited by constraints imposed by the system configuration, by the user serving a control station, or by other entities. Examples of such constraints may include a maximum and / or minimum limit for rotational angle, angular and / or linear velocity, operating parameters, and the like for the moving object (eg, the drive system thereof), the carrier (eg, an actuator thereof). , the imager or the like. Such thresholds may be used to limit the range of adjustment. For example, a setting concerning the angular velocity of the moving object and / or the imaging device relative to the moving object (via a support) about a particular axis may be capped by a maximum angular velocity indicative of the moving object and / or the object Carrier is allowed. As another example, a setting concerning the linear velocity of the moving object and / or the imaging device relative to the moving object (via a carrier) may be capped by a maximum linear velocity indicative of the moving object and / or the Carrier is allowed. As another example, the focal length setting of the imaging device may be limited by the maximum and / or minimum focal length of the particular imaging device. In some embodiments, such limits may be predetermined and depend on the particular configuration of the moving object, the carrier, and / or the imaging device. In some embodiments, the boundaries (eg, by a manufacturer, administrator, or user) may be configurable.

In some embodiments, warning indicators may be provided if the adjustment is modified according to the constraints described above (eg, if the angular velocity of the moveable object and / or the beam is capped about a particular axis by a maximum angular velocity for the moveable object and / or the wearer is admissible). Examples of such warning indicators may include text, audio (eg, siren or beep), visual (eg, particular light color or flashing light), kinetic (eg, vibration or motion), any other suitable types of signals. Such warning indicators may be provided directly by the moving object or a component thereof. Alternatively or additionally, warning indicators may be provided by the control station (eg, via the display). In the latter case, the control station can provide the warning indicators based on signals from the moving object.

In some embodiments, the adjustment may be performed substantially in real time when the moving object is executing user supplied navigation commands or a predetermined trajectory and / or when the imaging device is capturing one or more images. In some embodiments, the adjustment may be generated based on other information, such as sensor data sensed by one or more sensors aboard the mobile object (eg, proximity sensor or GPS sensor). For example, positional information of the tracked target may be obtained by a proximity sensor and / or GPS sensor provided by the target itself (eg, a GPS position). Such position information, in addition to the detected deviation, may be used to generate the adjustment control signals to track the target, as described herein, for example.

In various embodiments, the recognition 406 the deviation from the expected target information and / or the generation 408 of control commands to correct the detected deviation, be repeated for a predefined or indefinite period. In some embodiments, such deviation detection and / or command generation may be performed at certain intervals (eg, every 0.01 second, 0.1 second, 0.2 second, 0.5 second, or 1 second).

5 illustrates an exemplary configuration 500 a movable object, a carrier and a payload, according to embodiments. The configuration 500 is used to illustrate exemplary types of adjustment to the moving object 502 and / or the payload 506 to illustrate that can be used to track a target. The moving object 502 and the payload 506 may include any embodiments discussed herein. For example, the moving object 502 include a UAV, and the payload 506 may include an imaging device.

The moving object 502 may be able to move up to three vertical axes, such as the X ₁ (pitch) axes 510 , Y ₁ (yaw) 508 and Z ₁ (roles) 512 to turn. The rotations about the three axes can each be called the pitch rotation 522 , Yaw rotation 520 or roll rotation 524 be designated. The angular velocities of the moving object 502 about the axes X _1, Y ₁ and Z ₁ may each be as ω _{X1, Y1} ω or ω are expressed _Z1. The moving object 502 can also cause displacement movements 528 . 526 and 530 along the axes X ₁ , Y ₁ and Z ₁ to be able. The linear velocities of the moving object 502 along the axes X ₁ , Y ₁ and Z ₁ may be expressed as V _X1 , V _Y1 and V _Z1 , respectively.

In the example configuration, the payload is 506 about a carrier 504 to the moving object 502 connected. The carrier 504 may be able to induce the payload 506 in relation to the moving object by up to three vertical axes, X ₂ (pitch) 516 , Y ₂ (yaw) 514 and Z ₂ (roles) 518 turns and / or moves along it. The axes X ₂ , Y ₂ and Z ₁ may each be parallel to the axes X ₁ , Y ₁ and Z ₁ . In some embodiments, the payload is an optical module 507 includes, can the roll axis Z ₂ 518 substantially parallel to a light path or optical axis for the optical module 507 be. The optical module 507 may be optically connected to an image sensor, such as described herein, to capture images. The carrier 504 On the basis of control signals provided to the carrier associated actuators, such as electric motors, cause the payload 506 up to three vertical axes, X ₂ (pitch) 516 , Y ₂ (yaw) 514 and Z ₂ (roles) 518 rotates. The rotations about the three axes can each be called the pitch rotation 534 , Yaw rotation 532 or roll rotation 536 be designated. The angular velocities of the payload 506 about the axes X _2, Y ₂ and Z ₂ may each be as ω _{X2, Y2} ω or ω _Z2 are expressed. The carrier 504 can also cause the payload 506 on displacement movements 540 . 538 and 542 along the axes X ₂ , Y ₂ and Z ₂ in relation to the moving object 502 involved. The linear velocity of the payload 506 along the axes X ₂ , Y ₂ and Z ₂ can be expressed as V _X2 , V _Y2 and V _Z2 , respectively.

In some embodiments, the carrier 504 only allow the payload 506 around a subset of the axes X ₂ , Y ₂ and Z ₂ and / or along the same relative to the moving object 502 emotional. For example, the carrier 504 only allow the payload 506 to X ₂ , Y ₂ , Z ₂ or any combination of the same turns, without allowing the payload 506 moved along one of the axes. For example, the carrier 504 allow for the payload 506 only one of the axes X ₂ , Y ₂ and Z ₂ rotates. The carrier 504 can allow for the payload 506 only about two of the axes X ₂ , Y ₂ and Z ₂ rotates. The carrier 504 can allow for the payload 506 about all three of the axes X ₂ , Y ₂ and Z ₂ rotates.

In some other cases, the wearer may 504 only allow the payload 506 Moved along the axes X ₂ , Y ₂ , Z ₂ or any combination thereof, without allowing the payload 506 rotates about one of the axes. For example, the carrier 504 allow for the payload 506 moved along only one of the axes X ₂ , Y ₂ , Z ₂ . The carrier 504 can allow for the payload 506 moved along only two axes of X _2, Y _2, Z _2nd The carrier 504 can allow for the payload 506 moved along only all three of the axes X ₂ , Y ₂ , Z ₂ .

In still other embodiments, the carrier 504 allow that payload 506 performs both rotational and displacement movements in relation to the moving object. For example, the carrier 504 be configured to allow the payload 506 along one, two or three of the axes X ₂ , Y ₂ , Z ₂ moves and / or rotate about the same.

In some other embodiments, the payload 506 immediately without a carrier 504 to the moving object 502 be connected or the carrier 504 can not allow the payload 506 in relation to the moving object 502 emotional. In such cases, the location, position and / or orientation of the payload 506 in relation to the moving object 502 established.

In various embodiments, adjustment may be to the location, position, and / or orientation of the payload 506 , common or single, can be achieved via a suitable adjustment on the moving object 502 , the carrier 504 and / or the payload 506 , For example, a rotation of 6 degrees about a given axis (e.g., the yaw axis) for the payload can be achieved by a 60 degree rotation through the moving object alone a 60 degree turn by the payload relative to moving object, as effected by the carrier, or a combination of a 40 degree rotation by the moving object and a 20 degree rotation by the payload relative to the moving object.

Similarly, a displacement movement for the payload, commonly or individually, can be achieved via a setting on the moving object 502 and the carrier 504 , The desired setting can be achieved, additionally or alternatively, by adjusting the operating parameters of the payload. Such operating parameters of the payload may include, for example, an in / out zoom level or a focal length of an imager.

6 FIG. 10 illustrates an exemplary tracking method for maintaining an expected position of a target, according to embodiments. FIG. An exemplary picture 600 is shown as being taken, for example, by an imaging device carried by a moving object. Assume that the image has a width of W pixels and a height of H pixels (where W and H are positive integers). A position within the image may be defined by a pair of coordinates along a horizontal axis 601 (along the width of the image) and a vertical axis 603 (along the height of the image), where the upper left corner of the image has the coordinates (0, 0) and the lower right corner of the image has the coordinates (W, H).

Suppose that a goal, as in the picture 600 recorded at a position P (u, v) 602 is arranged and the expected position P ₀ (u ₀ , v ₀ ) 604 of the goal, which is different from P 602 different. In some embodiments, the expected position P ₀ (u ₀ , v ₀ ) of the target may be near the center of the image such that u ₀ = W / 2 and v ₀ = H / 2. In another embodiment, the expected position of the target may be located elsewhere within the image (eg, off-center). In various embodiments, the expected position of the target may be the same as the initial position of the target (eg, as provided by the control station) or not. Assuming that the current position P deviates from the expected position P ₀ such that the deviation exceeds a predetermined threshold (such as expressed by a Δx of u ₀ and a Δy of v ₀ ), then adjustment is required to reach the target position from P close to the expected position P ₀ .

In some embodiments, the deviation from the expected target position may be used to derive one or more angular velocities for rotating the field of view of the imager about one or more axes. For example, a deviation along the horizontal axis 601 of the image (eg between u and u ₀ ) can be used to obtain an angular velocity ω _Y 610 for rotating the field of view of the imaging device about the Y (yaw) axis 606 derive as follows: ω _y = α · (u - u ₀ ), where α ε R (real numbers) (1)

The rotation about the Y axis for the field of view of an imaging device can be achieved by a rotation of the moving object, a rotation of the payload (via a carrier) relative to the moving object, or a combination of both. In some embodiments, adjustment to the payload may be selected when adjustment to the moving object is unfeasible or otherwise undesirable, for example, when the navigation path of the moving object is predetermined. In the equation (1), α is a constant that is predefined and / or based on the configuration of the moving object (for example, when the rotation is achieved by the moving object), the configuration of the carrier (e.g. when the rotation is achieved by the carrier) or both (eg, when the rotation is achieved by a combination of the moving object and the carrier) can be calibrated. In some embodiments, α is greater than zero (α> 0). In other embodiments, α may not be greater than zero (α ≦ 0). In some embodiments, α may be used to map a calculated pixel value to a corresponding control lever amount or sensitivity to control the angular velocity about a particular axis (eg, the yaw axis). In general, the control lever can be used to control the angular or linear velocity of a controllable object (eg, a UAV or carrier). A larger control lever size corresponds to greater sensitivity and greater speed (for angular or linear motion). In some embodiments, the control lever dimension or a portion thereof may be determined by configuration parameters of the flight control system for a UAV or configuration parameters of a control system for a wearer. The upper and lower limits of the range of the control lever measure may include any arbitrary numbers. For example, the range of the control lever amount may be (1000, -1000) for a flight control system and (-1000, 1000) for another flight control system.

For example, assume that the images have a width of W = 1024 pixels and a height of H = 768 pixels. As a result, the size of the images is 1024 × 768. Further assume that the expected position of the target has u ₀ = 512. Consequently, (u-u ₀ ) ε (-512, 512). Assuming that the range of the control lever amount is around the yaw axis (-1000, 1000), then the maximum control lever amount or the maximum sensitivity is 1000 and α = 1000/512. Thus, the value of α may be affected by the image resolution or size provided by the imager, the range of the control lever amount (eg, about a particular axis of rotation), the maximum control lever amount or the maximum sensitivity, and / or other factors.

For example, when the rotation is achieved by a rotation of the moving object, the Y-axis corresponds 606 from 6 the axis Y ₁ 508 for the moving object, as in 5 and the total angular velocity ω _{Y of} the visual field is expressed as the angular velocity ω _Y1 for the moving object: ω _y = ω _y1 = α ₁ · (u-u ₀ ), where α ₁ ε R (2)

In the equation (2), α _{1 is} a constant defined based on the configuration of the moving object. In some embodiments, α _{1 is} greater than zero (α ₁ > 0). The α ₁ can be defined similarly to the α discussed above. For example, a value of α _{1 may be defined} based on the image resolution or size and / or range of the control lever measure for the moving object (eg, about the yaw axis).

Similarly, when the rotation is achieved by a rotation of the payload relative to the moving object (eg, via the carrier) will, the Y-axis 606 from 6 the axis Y ₂ 514 for the payload, as in 5 and the total angular velocity ω _{Y of} the visual field is expressed as the angular velocity ω _Y2 for the payload relative to the mobile object: ω _y = ω _{y 2} = α ₂ · (u - u _0), where α ₂ ε R (3)

In the equation (3), α _{2 is} a constant defined based on the configuration of the carrier and / or the payload. In some embodiments, α _{2 is} greater than zero (α ₂ > 0). The α ₂ can be defined similarly to the α discussed above. For example, the value of α _{2 may be defined} based on the image resolution or size and / or range of the control lever dimension for the wearer (eg, about the yaw axis).

In general, the angular velocity of the visual field can be around the Y (yaw) axis 606 are expressed as a combination of the angular velocity ω _Y1 for the moving object and the angular velocity ω _Y2 for the payload relative to the mobile object, for example as follows: ω _y = ω + ω _{y1 y2} (4)

In the equation (4), either ω _Y1 or ω _{Y2 may be} zero.

As illustrated herein, the direction of rotation about the y (yaw) axis may depend on the sign of u-u ₀ . For example, if the expected position is located to the right of the actual position (as in FIG 6 illustrated), then u - u ₀ <0, and the field of view must be counterclockwise about the yaw axis 606 Turn (eg, turn left) to bring the target to the expected position. On the other hand, if the expected position is located to the left of the actual position, then u - u ₀ > 0, and the field of view must be clockwise about the yaw axis 606 turn (eg, turn right) to bring the target to the expected position.

As illustrated herein, the rotational speed (eg, an absolute value of the angular velocity) about a given axis (eg, the Y (yaw) axis) may be determined from the distance between the expected and actual positions of the target along the Axis (ie, | u - u ₀ |). The farther the distance, the greater the speed of rotation. Likewise, the closer the distance, the slower the rotation speed. If the expected position coincides with the position along the axis (eg, u = u ₀ ), then the rotational speed about the axis is zero and the rotation stops.

The method of adjusting the deviation between the expected target position and the actual target position along the horizontal axis 601 As discussed above, it may be applied in a similar manner to the deviation of the target along another axis 603 to correct. For example, a deviation along the vertical axis 603 of the image (e.g., between v and v ₀ ) may be used to obtain an angular velocity ω _x 612 for the field of view of the imaging device about the X (pitch) axis 608 derive as follows: ω _x = β · (v - v ₀ ), where β ∈ R (real numbers) (5)

The rotation about the X-axis for the field of view of an imaging device can be achieved by a rotation of the moving object, a rotation of the payload (via a carrier) relative to the moving object, or a combination of both. Thus, in Equation (5), β is a constant that is predefined and / or based on the configuration of the moving object (eg, when the rotation is achieved by the moving object), the configuration of the carrier (e.g. B. when the rotation is achieved by the carrier) or both (eg, when the rotation is achieved by a combination of the moving object and the carrier) can be calibrated. In some embodiments, β is greater than zero (β> 0). In other embodiments, β may not be greater than zero (β ≤ 0). In some embodiments, β may be used to map a calculated pixel value to a corresponding control lever amount or sensitivity to control the angular velocity about a particular axis (eg, the pitch axis). In general, the control lever can be used to control the angular or linear velocity of a controllable object (eg, a UAV or carrier). A larger control lever size corresponds to greater sensitivity and greater speed (for angular or linear motion). In some embodiments, the control lever dimension or a portion thereof may be determined by configuration parameters of the flight control system for a UAV or configuration parameters of a control system for a wearer. The upper and lower limits of the range of the control lever measure may include any arbitrary numbers. For example, the range of the control lever amount may be (1000, -1000) for a flight control system and (-1000, 1000) for another flight control system.

For example, assume that the images have a width of W = 1024 pixels and a height of H = 768 pixels. As a result, the size of the images is 1024 × 768. Further assume that the expected position of the target has a v ₀ = 384. Consequently, (v - v ₀ ) ε (-384, 384). Assuming that the range of the control lever amount is around the pitch axis (-1000, 1000), then the maximum control lever amount or the maximum sensitivity is 1000 and β = 1000/512. Thus, the value of α may be affected by the image resolution or size provided by the imager, the range of the control lever amount (eg, about a particular axis of rotation), the maximum control lever amount or the maximum sensitivity, and / or other factors.

For example, when the rotation is achieved by a rotation of the moving object, the X-axis corresponds 608 from 6 the axis X ₁ 510 for the moving object, as in 5 and the total angular velocity ω _{X of} the visual field is expressed as the angular velocity ω _X1 for the moving object: ω _x = ω _x1 = β ₁ · (v - v ₀ ), where β ₁ ε R (6)

In the equation (6), β _{1 is} a constant defined based on the configuration of the moving object. In some embodiments, β _{1 is} greater than zero (β ₁ > 0). The β ₁ can be defined similarly to the β discussed above. For example, a value of β _{1 may be defined} based on image resolution or size and / or range of the control lever measure for the moving object (eg, about the pitch axis).

Similarly, when rotation is achieved by rotation of the payload relative to the moving object (eg, via the carrier), the X-axis corresponds 608 from 6 the axis X ₂ 516 for the payload, as in 5 and the total angular velocity ω _{X of} the field of view is expressed as the angular velocity ω _X2 for the payload relative to the mobile object: ω _x = ω _x2 = β ₂ · (v - v _0), where β ₂ ε R (6)

In the equation (6), β _{2 is} a constant defined based on the configuration of the carrier and / or the payload. In some embodiments, β _{2 is} greater than zero (β ₂ > 0). The β ₂ can be defined similarly to the β discussed above. For example, the value of β _{2 may be defined} based on the image resolution or size and / or range of the control lever measure for the wearer (eg, about the pitch axis).

In general, the angular velocity of the field of view can be around the X (pitch) axis 608 are expressed as a combination of the angular velocity ω _X1 for the moving object and the angular velocity ω _X2 for the payload relative to the mobile object, for example as follows: ω _x = ω _x1 + ω _x2 (7)

In the equation (7), either ω _X1 or ω _{X2 may be} zero.

As illustrated herein, the direction of rotation about the X (pitch) axis may depend on the sign of v-v ₀ . For example, if the expected position is located above the actual position (as in FIG 6 illustrated), then v - v ₀ > 0, and the field of view must be clockwise about the pitch axis 608 rotate (eg, nod down) to bring the target to the expected position. On the other hand, if the expected position is below the actual position, then v - v ₀ <0, and the field of view must be counterclockwise about the pitch axis 608 rotate (eg, nod up) to bring the target to the expected position.

As illustrated herein, the rotational speed (eg, an absolute value of the angular velocity) about a given axis (eg, the X (pitch) axis) depends on the distance between the expected and actual positions of the target along the Axis (ie, | u - u ₀ |). The farther the distance, the greater the speed of rotation. The closer the distance, the slower the rotation speed. If the expected position coincides with the position along the axis (eg, v = v ₀ ), then the rotational speed about the axis is zero and the rotation stops.

In some embodiments, the values of the angular velocities, as calculated above, may be limited or otherwise modified by various limitations of the system. Such limitations may include the maximum and / or minimum speed achievable by the moving object and / or the imaging device, the range of the control lever amount, or the maximum control lever amount or maximum sensitivity of the movable object and / or the support system and control system include such. For example, the rotational speed may be the minimum of the calculated rotational speed and the maximum permitted speed.

In some embodiments, warning indicators may be provided if the calculated angular velocities need to be modified according to the limitations described herein. Examples of such warning indicators may be textual, audio (eg, siren or beep), visual (eg, certain light color or flashing light), mechanical, any other suitable types of Include signals. Such warning indicators may be provided directly by the moving object, the carrier, the payload or a component thereof. Alternatively or additionally, warning indicators may be provided by the control station (eg, via the display). In the latter case, the control station can provide the warning indicators based on signals from the mobile object.

7 FIG. 10 illustrates an exemplary tracking method for maintaining an expected size of a target, according to embodiments. An exemplary picture 700 is shown as being taken, for example, by an imaging device carried by a moving object. Suppose that a goal 702 through the picture 700 is recorded. The actual size of the target within the image may be s pixels (such as calculated as the product of the width of the target and the height of the target). The expected target size S may be smaller (eg, the expected target may be due to 704 can be represented and S = s ₀ ) or greater (eg, the expected target can be represented by 705 and S = s ₁ ) as the actual size s. The expected size of the target may be the same as the initial size of the target (eg as provided by the control station) or not. Assuming that the current size s deviates from the expected size s ₀ or s ₁ such that the deviation exceeds a predetermined threshold (such as predefined Δs pixels), then adjustment is required to bring the target size close to the expected size s ₀ or to bring s ₁ .

Although the display area of the image and the destination are shown as rectangles, this is for illustrative purposes only and not intended to be limiting. Instead, in various embodiments, the display area of the image and / or target may have any suitable shapes, such as circles, ovals, polygons, and the like. Likewise, although the areas discussed herein are expressed in pixels, this is for illustrative purposes only and is not intended to be limiting. In other embodiments, the areas may be expressed in any suitable units, such as mega pixels, mm ² , cm ² , inches ^2, and the like.

In some embodiments, the deviation from the expected target size may be used to derive one or more linear velocities for the moving object and / or the imaging device along one or more axes. For example, a deviation in the target size between an actual target size s and the expected target size S (eg, S = s ₀ or s ₁ ) may be used to set a linear velocity V for moving the movable object along a Z (roll -)Axis 710 to determine as follows: V = δ · (1-s / S), where δ ∈ R (8)

In Equation (8), δ is a constant that may be based on the configuration of the moving object and / or any suitable controllable object (eg, a carrier) that may cause the field of view to approach the target and / or moved from the same, is defined. In some embodiments, δ is greater than zero (δ> 0). In other embodiments, δ can not be greater than zero (β ≤ 0). In some embodiments, δ may be used to map a calculated pixel value to a corresponding control lever amount or sensitivity to control the linear velocity.

In general, V represents the velocity of the moving object toward and / or from the target. The velocity vector points from the UAV to the target. If the actual size s of the target is smaller than the expected size S, then V> 0, and the moving object moves toward the target so as to increase the size of the target as taken in the images. On the other hand, if the actual size s of the target is larger than the expected size S, then V <0, and the moving object moves away from the target so as to reduce the size of the target as taken in the images.

For example, assume that the images have a width of W = 1024 pixels and a height of H = 768 pixels. As a result, the size of the images is 1024 × 768. Assume that the range of the control lever amount for controlling the linear velocity is (-1000, 1000). In one embodiment, δ = -1000 when s / S = 3 and δ = 1000 when s / S = 1/3.

In some embodiments, the values of the speeds as calculated above may be limited or otherwise modified by various limitations of the system. Such limitations may include the maximum and / or minimum speed that may be achieved by the moving object and / or the imaging device, the maximum sensitivity of the moving object and / or the carrier control system, and the like. For example, the speed for the moving object may be the minimum of the calculated speed and the maximum allowed speed.

Alternatively or additionally, the deviation between the actual target size and the expected target size may be used to derive an adjustment to the operating parameters of the imaging device, such as zoom level or focal length, to correct for the deviation. Such adjustment to the imaging device may be necessary if adjustment to the moving object is unworkable or otherwise undesirable, for example, if the navigation path of the moving object is predetermined. An exemplary focal length setting F can be expressed as: F = γ · (1-s / S), where γ ∈ R (9)

Where γ is a constant that is defined based on the configuration of the imaging device. In some embodiments, γ is greater than zero (γ> 0). In other embodiments, γ is not greater than zero (γ ≤ 0). The value of γ may be defined based on the type of lenses and / or imaging devices.

If the actual size s of the target is less than the expected size S, then F> 0 and the focal length decreases by | F | to increase the size of the target as recorded in the images. On the other hand, if the actual size s of the target is greater than the expected size S, then F <0, and the focal length decreases by | F | to decrease the size of the target as shown in the pictures. For example, in one embodiment, γ = 10. This means that, for example, if the actual size s of the target is twice the expected size S, the focal length will be 10 mm (ie, F = 10 * (1 - 2 /). 1) = -10) and vice versa.

In some embodiments, the adjustment to the operating parameters of the imager, such as the focal length, may be limited or otherwise modified by various limitations of the system. Such limitations may include the maximum and / or minimum focal length that can be achieved by the imaging device. As an example, assume the focal length range is (20mm, 58mm). Further, assume that the initial focal length is 40 mm. Then, if s> S, the focal length should be decreased according to the equation (9), and if s <S, the focal length should be increased according to the equation (9). However, such a setting is limited by the lower and upper limits of the focal length range (eg, 20 mm to 58 mm). In other words, the focal length after adjustment should be not less than the minimum focal length (eg, 20 mm) and not more than the maximum focal length (eg, 58 mm).

As in above 6 Warning indicators may be provided if the calculated setting (eg, the linear velocity of the moving object or the focal length) needs to be modified according to the constraints described herein. Examples of such warning indicators may include text, audio (eg, siren or beep), visual (eg, certain light color or flashing light), mechanical, any other suitable types of signals. Such warning indicators may be provided directly by the moving object, the carrier, the payload or a component thereof. Alternatively or additionally, warning indicators may be provided by the control station (eg, via the display). In the latter case, the control station can provide the warning indicators based on signals from the moving object.

According to another aspect of the present invention, methods and systems for tracking a target by an imaging device connected to a moving object while controlling the moving object to navigate by navigation commands are provided.

8th illustrates another example process 800 to implement goal tracking, according to embodiments. Aspects of the process 800 may be performed by one or more processors on board and / or outside a moving object as described herein, such as a UAV.

The process 800 includes receiving 802 user navigation commands and destination information, for example, from a remote control station as described herein. The navigation commands may be used to control navigation parameters of the mobile object, such as the position, velocity, orientation or location of the mobile object. In some cases, the navigation commands may include instructions for the moving object to execute some pre-stored navigation commands, such as those associated with a predetermined navigation path. The navigation commands can be used to control the moving object 804 to navigate according to a user-specified or pre-stored navigation path.

The destination information may include initial destination information (eg, position and / or size) of a particular destination, such as a user operating a guidance station, and / or destination type information, such as color, texture, pattern, size, shape, dimension, and the like. lock in.

The destination information may also include expected destination information (eg, position and / or size). The expected target information specifies the characteristics that are expected to satisfy the tracked target in the images captured by the imager. The expected target information may be used to set the moving object, the carrier, and / or the imaging device so that the tracked target in one or more images will maintain parameters specified by the expected target information. For example, the target may be tracked to maintain an expected position and / or size within one or more images captured by the imaging device. For example, the expected position of the tracked target may be near the center of the image or off-center. The expected size of the tracked target may be a certain number of pixels (eg, the initial target size). The expected destination information may be the same as the initial destination information or not. In various embodiments, the expected destination information may or may not be provided by the control station. For example, the expected destination information in the control logic executed by a processing unit aboard the mobile object may be hard-coded, stored in a data memory at and / or remote from the location of the mobile object, or obtained from other suitable sources.

In some embodiments, the destination information (including specific destination information and destination type information) may be at least in part at the control station 112 be generated. Additionally or alternatively, the destination information may be generated based on data from other sources. For example, the destination type information may be derived based on previous images and / or data extracted from local or remote data stores. Such target type information may be selected by the user and / or automatically provided by presetting the moving object.

The target information may be used to adjust the moving object, the carrier and / or the imaging device 806 to track the destination according to the destination information. The tracking and any other associated data processing may be automatically performed by one or more processors on board the mobile object, one or more processors remote from the mobile object, or any suitable combination thereof. In some embodiments, the target information may be used to identify the target to be followed by the moving object. Such identification of the destination may be performed based on the initial destination information including the specific characteristics of a particular destination (eg, based on provided coordinates of the destination within an image captured by the mobile object) or general destination type information included in the destination information.

Once a destination is identified, the expected destination information (as part of the destination information) may be used to detect a departure from expected characteristics of the destination, such as expected location and / or size. The deviation can be detected by comparing the characteristics of the target (e.g., positions and / or magnitudes) within a series of images captured by the moving objects to determine if the characteristics are changing through the images. The series of images can be captured by the moving object at different times. A change in the position of the target can be detected by comparing coordinates of the center of the target within an image with the coordinates of the expected target position. A change in the size of the target can be detected by comparing the size of the area covered by the target (eg, in pixels) with the expected target size.

At least in part, on the basis of the detected deviation, control signals may be generated automatically (eg, by one or more processors on board the mobile object) that are applicable to the moving object, the carrier, the imaging device, and / or any suitable components thereof , The control signals may be used to substantially correct the detected deviation. In some embodiments, the adjustment may be used to substantially maintain one or more expected target characteristics (eg, target position and / or size) within the images captured by the moving object as the movable object navigates according to user supplied navigation commands. The adjustment may be performed substantially in real time because the moving object is executing user supplied navigation commands or predetermined navigation lanes and / or because the imaging device is capturing one or more images. In some embodiments, the adjustment may be generated based on other information, such as sensor data sensed by one or more sensors aboard the mobile object (eg, proximity sensor or GPS sensor). For example, position information of the tracked Target detected by a proximity sensor and / or provided by the target itself (eg GPS position). Such position information, in addition to the detected deviation, may be used to generate the adjustment.

In some embodiments, when the moving object navigates according to user-supplied navigation commands or predetermined navigation lanes, the target tracking setting is limited to the carrier and / or imager without the state (eg, location, position, orientation or velocity) of the mobile To influence the object itself. This may be because it is unworkable or otherwise undesirable to adjust the moving object while executing user-supplied navigation commands or predetermined navigation lanes. For example, such adjustment may cause the wearer to move the imaging device relative to the moving object, such as one or two or more axes of rotation, to correct for a deviation from the expected position of the target. Additionally or alternatively, the adjustment may cause adjustment to the focal length or zoom, focus, or other operating parameters of the imager to correct for a deviation from the expected size of the target. A discussion of exemplary methods for such adjustments will be made in connection with 6 to 7 provided. In other embodiments, the adjustment may also include adjustment to the moving object.

According to another aspect of the present invention, methods and systems for controlling a moving object to navigate are provided in accordance with one or more navigation commands during the tracking of a destination. 9 illustrates an exemplary process 900 for controlling a moving object to navigate and track, according to embodiments. Aspects of the process 900 may be performed by a control station, such as described herein. For example, the control station may be a hand-held or portable device. For example, the control station may include a smartphone, a tablet, a laptop, a computer, glasses, gloves, a helmet, a microphone, or suitable combinations thereof.

The process 900 includes receiving 902 a user input regarding the navigation or a navigation input. Such navigation input may be used to generate navigational commands for controlling the mobile object or a component thereof, such as in connection with 8th described. The navigation input may be provided via a user interface provided by the control station, such as a keyboard, a mouse, a joystick, a touch screen, buttons, a microphone, a webcam, or the like. Any suitable user input may be used to interact with the controller, such as manually entered commands, voice control, gesture control, or position control (eg, via movement, orientation, and / or location of the device).

The process 900 includes receiving 904 a user input regarding the prosecution or a tracking input. Such tracking input may be used to generate destination information regarding a destination or type of destinations to be tracked by the mobile object, such as described herein. The tracking input may be provided using the same or a different interface as that for providing the navigation input. For example, the tracking input may be provided via a touch screen while the navigation input is provided via a joystick or inertial sensor configured to detect a change in the location of the control station (eg, rotating the control station about a rotation axis) can be. As another example, both navigation input and tracking input may be provided via the same user interface, such as via a touch screen, keyboard, mouse, or joystick.

In some embodiments, the navigation input or the tracking input may each be provided via two or more user interfaces. For example, the navigation input may be provided via both joystick and touch screen. As another example, tracking input may be provided via touch screen, joystick, and keyboard.

The navigation input can be used to generate navigational commands for the moving object 906 , For example, mechanical signals resulting from or about the movement of a joystick by a user along one or more predefined axes may be translated into navigation commands for moving the moveable object along the respective axes (eg, pitch, yaw or roll) become. As another example, the interaction of the user with a graphical user interface (e.g., a movement of a symbol representing an orientation or position of the moving object) may be referred to as corresponding navigation commands are converted. As still another example, movement of the control station by the user along one or more predefined axes, or the same through one or more inertial sensors, may be detected and used to provide navigation commands for moving the mobile object along the respective axes (eg, nodding, Yawing or rolling). In addition to the navigation input, the navigation commands may be generated based on other data, such as sensor data provided by the moving object.

The tracking input can be used to generate destination information 908 that can be used by the moving object to track one or more targets. The destination information may include initial destination information and / or expected destination information, such as described herein. The initial destination information may be specific characteristics of a particular destination (eg, based on provided coordinates of the destination within an image captured by the mobile object) or general characteristics of a type of destination (eg, based on color provided and / or Texture of the target (s) to be tracked). The initial target information may be used to identify the target to be tracked, for example, from one or more images. The expected information may be used to detect a deviation from expected characteristics of the target, such as an expected position and / or size. In some embodiments, the expected information may be optional, and some or all of the initial information may be used as the expected information.

The process 900 also includes receiving user input and generation commands related to the control of the wearer and / or imaging devices. Examples of such control include changing a position and / or orientation of the imager over the carrier, adjusting one or more operating parameters (eg, zoom) of the imager, and the like.

In some embodiments, as described in more detail in, for example, FIG 10 The coordinates, size and other characteristics of a target are selected by the user in a graphical user interface and used to generate initial and / or expected target information.

In some embodiments, the generated navigation commands and / or destination information may be provided to the mobile object 910 , such as via a communication system of a control station. Such commands and information may be provided while the moving object is in use (ie, moving or otherwise operating in the intended environment). In some embodiments, the navigation commands and destination information may be provided to the mobile object directly from a control station. In some other embodiments, the navigation commands and destination information of one or more intermediary instances (eg, relay stations or other moving objects) may be provided before they reach the moving object.

In some embodiments, all steps of the process 900 implemented by the same entity, such as a single control station. In some other cases, the steps of the process 900 implemented by two or more separate instances. Such separate instances can communicate with each other, for example via cable or wireless connections. For example, the receiving 902 a navigation input and generating 906 of navigation commands are implemented by a first device while receiving 904 a tracking input and generating 908 of destination information can be implemented by a second facility. The second device may transmit the destination information to the first device, which may be configured to transmit both the navigation commands and the destination information to the mobile object. As another example, the receiving steps 902 and 904 be performed by a device during the generating steps 906 and 908 can be performed by another device.

In accordance with some other aspects of the present invention, an interactive user interface for selecting and viewing, in real time, targets targeted by a moving object as described herein, such as a UAV, is provided. Such a user interface may be provided by a control station, such as described herein. 10 illustrates an exemplary process 1000 for selecting a destination, according to embodiments. Aspects of the process 1000 may be performed by a user interface such as described herein.

The process 1000 closes the ads 1002 one or more images taken by the moving object. The images may be captured by an imaging device connected to the moving object via a carrier as described herein is. The carrier may allow the imaging device to move relative to the moving object or not. The imaging device may alternatively be connected directly to the movable object without a carrier. The images can be transmitted from the moving object or directly from the imaging device to a control station. The images thus received by the control station may be decompressed or otherwise processed before being displayed on a user interface associated with the control station. The user interface may be provided by a display that is integral with the control station. Alternatively, the display may be separate from, but operatively connected to, the control station. The images displayed in this way can be static (eg still images) or dynamic (eg videos).

In some embodiments, the images may be displayed or streamed in substantially real time as the images are received by the moving object and / or the imaging device. In some embodiments, the images are transmitted in substantially real time from the moving object and / or the imaging device when the images are captured by the imaging device. In other cases, there may be some delay between taking pictures and / or transmitting the pictures and displaying the pictures.

In some embodiments, the process concludes 1000 the receiving 1004 user selection of a destination from within at least one of the displayed images. The user can select the destination via the same user interface that displays the images. For example, in some embodiments, user selection is achieved by the user selecting an area of the at least one of the one or more images displayed on the display, the selected area corresponding to the destination. For example, the user may select the destination through the immediate touch screen using a finger or stylus. As another example, the user may select the destination using a mouse, joystick, gesture, or voice command. In yet another embodiment, the user may select the destination via a wearable device such as a helmet, virtual reality glasses, and the like. The selection of the target may include any suitable movement, including touch or tap, swipe, circle, click, or the like, associated with any suitable input method, such as described herein.

In some embodiments, the selected target is displayed with a selection indicator in the images to indicate that it is selected by the user. The selection indicator may be represented by a circle, a hook, an arrow, a polygon, or any other graphic object of suitable shape having any suitable color or pattern.

The user-selected destination can be used to generate destination information 1006 , For example, the coordinates and pixel size of the selected target within an image may be used as initial information for the target or used to generate the same. Alternatively or additionally, characteristics associated with the selected target may be used to generate at least some of the target species information. In some embodiments, the remaining destination type information may be provided based on another user input. For example, the destination type information may include the color and texture of the target to be tracked. The color of the destination type information may be extracted from a particular user-selected destination, whereas the texture of the destination may be entered manually by the user or selected from a plurality of predetermined patterns.

In some embodiments, a user is allowed to choose between a manual tracking mode and an automatic tracking mode. When the manual tracking mode is selected, a user can specify a specific target to track, such as an image displayed by the control station. On the other hand, when the automatic tracking mode is selected, the user can specify the target type information instead of a specific target to be tracked. The moving object can then use the target type information to automatically identify the target to be tracked and then track the identified target.

Optionally, information about the selected destination can be used to generate expected destination information. For example, the initial destination information may be the same as the expected destination information. As another example, the expected destination information may be expressed with reference to the initial destination information. For example, the expected target position may be expressed using an offset from the initial target position. Similarly, the expected target size may be expressed as a ratio between the expected target size and the initial target size (eg, the expected target size may be double the target size) Size of the initial target size). In various embodiments, the destination information may be based on a user-selected destination, data obtained from other user input methods (e.g., data input from a keyboard), and / or data from non-user sources (e.g., local or remote) remote data store or other suitable data processing system).

The destination information may be provided to the mobile object for tracking purposes 1008 become. For example, the initial destination information and / or the destination type information may be used by one or more processors on the mobile object to identify the destination to be tracked. The expected target information may be used by the moving object to substantially maintain a predetermined position, size, and other characteristics of the target during the tracking operation.

In various embodiments, any portion of the destination information used to control the tracking process may be provided by users and the rest by the tracking system. For example, in one embodiment, initial destination information about a user selection of a specific destination is provided by a user, whereas the expected destination information is provided by the system by default. In another embodiment, a user may provide the destination type information, whereas the system automatically determines the destination based on the destination type information and tracks the destination using user-specified or system-provided expected destination information. In yet another embodiment, a user may provide all destination information, including the initial and expected destination information, and the system will simply track the specified destination accordingly.

The specific assignment of tracking control between the users and the tracking system may vary depending on a variety of factors, such as the environment of the moving object, the speed or height of the moving object, user preferences, and available computational resources (eg, CPU or Memory) on board and / or outside the mobile object, the like. For example, the user may be given relatively more control when navigating the moving object in a relatively complex environment (eg, with numerous buildings or obstacles or inside) than if the moving object were in a relatively simple environment (e.g. open space or outside). As another example, more control may be assigned to the user when the moveable object is at a lower elevation than when the moveable object is at a higher elevation. As yet another example, more control may be assigned to the moving object if the moving object is equipped with a high speed processor configured to perform complex calculations relatively quickly.

11 illustrates an exemplary process 1100 for considering a pursued goal, according to embodiments. Aspects of the process 1100 can through the same control station to implement the process 100 or another.

The process 1100 includes receiving 1102 of images captured by a moving object such as described herein (eg, a UAV). The images may be captured by an imaging device (eg, via a carrier) connected to the moving object. In some cases, the images may be received directly by the imaging device that captured the images. The images may be received by a control station, such as described herein. In some embodiments, the images may be received after destination information, such as described herein, has been provided to the moving object. The destination information may be according to a process similar to the process 900 or 1000 , such as in 9 or 10 have been discussed. The moving object may be configured to specify the destination as identified and / or specified by the destination information according to a process similar to that in FIG 4 discussed process 400 to pursue. Therefore, at least some of the received images may include the tracked target.

The process 1100 also includes receiving 1104 of tracking data or information used to indicate characteristics of one or more targets tracked by the moving object. Examples of tracking information may include relative or absolute coordinates and / or size of the target within one or more images received from the mobile object, changes or offsets of the destination between successive frames, GPS coordinates or other location information of the destination, and the like.

The received images can be displayed with tracking data 1106 be to show that the target is tracked by the moving object. For example, the tracking data may be used to determine the position and / or size of the tracked target in some or all of the images. Based on this information, a tracking indicator, such as similar to the one shown in FIG 10 discussed selection indicators, generated and displayed. For example, given the target position within a given image, the image may be displayed with a tracking indicator approximately at the target position and with a size sufficiently large to cover the target or a portion thereof. Examples of a tracking indicator may include a box, a circle, an oval or a graphic object of any other suitable shape, color or size.

Advantageously, displaying a tracked target with such a tracking indicator helps the users to visually identify the tracked target as they view the images captured by the moving object and / or communicated therefrom in substantially real time. In other words, the users can view the targets substantially simultaneously as they are tracked by the moving object. In other embodiments, the display may be provided after a certain delay.

In some embodiments, while viewing the destination tracked by the moving object, the user may select the tracked destination (eg, as indicated by the tracking indicator) to see more detailed information regarding the destination. The user can also perform other operations on the target or any other displayed objects by interacting with the user interface where the images are displayed. For example, the user may zoom in or out by moving fingers over a touch screen.

Some such user operations can be converted to track commands and sent to the moving object. For example, as described above, user zoom-in / zoom-out actions may be translated to appropriate zoom-in / zoom-out commands for the imaging device and / or control commands to move the mobile object closer to or further away from the target. Alternatively or additionally, the user may decide to switch to another destination for tracking. Such a change can be performed dynamically when the user selects the other destination. The destination information associated with the other destination may be dynamically provided to the mobile object, for example, in a manner similar to that in US Pat 10 described. In response, the moving object can then dynamically switch to tracking the other target using the methods described herein, for example, substantially in real time.

12 illustrates an exemplary control station 1200 for controlling a moving object, according to embodiments. The control station 1200 closes a display area 1202 for displaying sensor data, such as image data captured by a moving object, such as a UAV, coupled imager or image sensor. The image data may be images of one or more objects 1210 and 1206 as captured by an imaging device or image sensor aboard the mobile object. One or more of the objects can be a target 1206 which is tracked by the moving object. The goal may be with an indicator 1208 are displayed. The scoreboard 1208 can display a selection indicator, such as in 10 discussed to indicate that the destination is selected by the user to be tracked by the moving object, and / or a tracking indicator, such as in FIG 11 discussed to indicate that the target is being tracked by the moving object. The scoreboard 1208 may be configured to move with the target as its positions change across different images.

The control station 1200 can also have one or more input devices 1204 and 1205 to receive a user input. Such an input device can be joysticks 1204 , Buttons or switches 1205 , Keyboard, mouse, touch screen, or any suitable mechanism. For example, one or more buttons or switches 1205 be used to switch between different input modes, such as navigation input mode, destination input mode and the like. The switches 1205 may also be used to select between a manual tracking mode and an automatic tracking mode as described herein. The with some of the input devices, such as the joysticks 1204 Associated functionalities may change depending on the selected input mode. For example, such input devices may be used to provide navigation input when the navigation input mode is selected and destination information when the destination input mode is selected. As another example, the input devices may be used to select a specific destination when the manual tracking mode is selected, and destination type information when the automatic tracking mode is selected.

13A 1-5 illustrate exemplary methods for selecting a destination using a user interface, according to embodiments. In such an embodiment, such as in 13A As illustrated, the user interface may allow a user to target 1304 by touching, using a hand 1302 , an area of a touch screen corresponding to the destination. The selected destination 1304 can by a selection indicator 1306 such as a circle containing the selected target 1304 surrounds, to be displayed. In another embodiment, such as in 13B As illustrated, the user interface may allow a user to target 1310 using a mouse 1308 for example, by clicking on an area of a screen corresponding to the destination. The selected destination 1310 can with a selection indicator 1312 are displayed. In yet another embodiment, such as in 13C As illustrated, the user interface may allow a user to target 1316 using a pen 1314 , for example, by tapping on an area of a screen corresponding to the destination. The selected destination 1316 can with a selection indicator 1318 are displayed. In various embodiments, the selection indicator may be any arbitrary shape, such as a circle 1306 , a rectangle 1312 or a rhombus 1318 , to have.

The systems and methods described herein may be implemented or applied to a wide variety of moving objects. The systems, devices, and methods described herein may be applied to a wide variety of moving objects. As previously mentioned, any description of an aircraft herein may apply and be used for any moveable object. A moving object of the present invention may be configured to travel in any suitable environment, such as in the air (e.g., a fixed wing aircraft, a rotary wing aircraft or an aircraft having neither fixed wing nor rotary wing) in the water (e.g. A ship or a submarine) on the ground (eg, a motor vehicle such as a car, a truck, a bus, a motorcycle, a movable structure or a scaffold such as a pole, a fishing rod; Train), in-ground (eg, a subway), in space (eg, a spacecraft, a satellite, or a probe), or any combination thereof. The moveable object may be a vehicle, such as a vehicle described elsewhere herein. In some embodiments, the moving object may be attached to a living subject, such as a human or an animal. Suitable animals may include primates, birds, dogs, cats, horses, cattle, sheep, pigs, dolphins, rodents or insects.

The moving object may be able to move freely within the environment with respect to six degrees of freedom (eg, three degrees of freedom in displacement and three degrees of freedom in rotation). Alternatively, the movement of the moving object may be restricted in one or more degrees of freedom, such as by a predetermined path, path or orientation. The movement may be driven by any suitable drive mechanism, such as an engine or engine. The drive mechanism of the moving object may be powered by any suitable source of energy, such as electric power, magnetic energy, solar energy, wind energy, gravitational energy, chemical energy, nuclear energy, or any suitable combination thereof. The moveable object may be self-propelled via a drive system as described elsewhere herein. The drive system may optionally be run with a source of energy, such as electric power, magnetic energy, solar energy, wind energy, gravitational energy, chemical energy, nuclear energy, or any suitable combination thereof. Alternatively, the movable object may be carried by a living being.

In some cases, the moving object may be a vehicle. Suitable vehicles may include watercraft, aircraft, spacecraft or ground vehicles. For example, aircraft may include fixed wing aircraft (eg, airplanes, gliders), rotary wing aircraft (eg, helicopters, rotorcraft), airplanes that have both fixed wings and rotary wings, or aircraft that have neither (eg, small airships , Hot air balloons). A vehicle may be self-propelled, such as by air, on or in the water, in space, or on or under the ground itself. A self-propelled vehicle may use a propulsion system, such as a propulsion system that includes one or more prime movers, motors, wheels, axles, magnets, rotors, propellers, blades, nozzles, or any suitable combination thereof. In some cases, the drive system may be used to allow the moveable object to lift off a surface, land on a surface, maintain its current position and / or orientation, change orientation, and / or change position.

The moving object may be controlled remotely or locally by an occupant within or on the moving object. In some embodiments, the moving object is an unmanned moving object, such as a UAV. An unmanned moving object, such as a UAV, may not have an occupant aboard the moving object. The moving object may be controlled by a human or an autonomous control system (eg, a computer control system) or any suitable combination thereof. The moving object may be an autonomous or semi-autonomous robot, such as a robot configured with artificial intelligence.

The movable object may be of any suitable size and / or dimensions. In some embodiments, the moveable object may have a size and / or dimensions to have a human occupant within or on the vehicle. Alternatively, the movable object may have a size and / or dimensions less than to be able to have a human occupant within or on the vehicle. The movable object may have a size and / or dimensions suitable for being lifted or carried by a human. Alternatively, the movable object may be larger than a size and / or dimensions to be suitable for being lifted or carried by a human. In some cases, the moving object may have a maximum dimension (eg, length, width, height, diameter, diagonal) less than or equal to about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m or 10 m. The maximum dimension may be greater than or equal to about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m or 10 m. For example, the distance between waves of opposing rotors of the moving object may be less than or equal to about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m or 10 m. Conversely, the distance between waves of opposing rotors of the moving object may be greater than or equal to about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m or 10 m.

In some embodiments, the moving object may have a volume of less than 100 cm x 100 cm x 100 cm, less than 50 cm x 50 cm x 30 cm, or less than 5 cm x 5 cm x 3 cm. The total volume of the moving object may be less than or equal to about 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ³ , 200 cm ³ , 300 cm ³ , 500 cm ³ , 750 cm ³ , 1000 cm ³ , 5000 cm ³ , 10000 cm ³ , 100000 cm ³ , 1 m be ³ or 10 m ³ . Conversely, the total volume of the moving object may be greater than or equal to about 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ³ , 200 cm ³ , 300 cm ³ , 500 cm ³ , 750 cm ³ , 1000 cm ³ , 5000 cm ³ , 10000 cm ³ , 100000 cm ³ , 1 m ³ or 10 m ³ .

In some embodiments, the moveable object may have a footprint (which may refer to the lateral cross-sectional area enclosed by the moveable object) less than or equal to about 32000 cm ² , 20000 cm ² , 10000 cm ² , 1000 cm ² , 500 cm ² , 100 cm ² , 50 cm ² , 10 cm ² or 5 cm ² . Conversely, the footprint may be greater than or equal to about 32000 cm ² , 20000 cm ² , 10000 cm ² , 1000 cm ² , 500 cm ² , 100 cm ² , 50 cm ² , 10 cm ² or 5 cm ² .

In some cases, the moving object can not weigh more than 1000 kg. The weight of the moving object may be less than or equal to approximately 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 v12 kg, 10 kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.50 kg, 0.1 kg, 0.05 kg or 0.01 kg. Conversely, the weight of the moving object may be greater than or equal to about 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg , 30 kg, 25 kg, 20 kg, 15v12 kg, 10 kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.50 kg, 0, 1 kg, 0.05 kg or 0.01 kg.

In some embodiments, a moving object may be small in relation to a load carried by the moving object. The load may include a payload and / or a carrier, as described in more detail below. In some examples, a ratio of a weight of the moving object to a load weight may be greater than, less than, or equal to about 1: 1. In some cases, a ratio of a weight of the moving object to a load weight may be greater than, less than, or equal to about 1: 1. Optionally, a ratio of a carrier weight object to a load weight may be greater than, less than, or equal to about 1: 1. If desired, the ratio of a weight of the moving object to a load weight may be less than or equal to about 1: 2, 1: 3, 1: 4, 1: 5, 1:10, or even less. Conversely, the ratio of a weight of the moving object to a load weight may be greater than or equal to about 1: 2, 1: 3, 1: 4, 1: 5, 1:10, or even greater.

In some embodiments, the moving object may have low power consumption. For example, the moving object may consume less than about 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h or less. In some cases, a wearer of the moving object may have a low power consumption. For example, the carrier object may be less than about 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 Consume W / h or less. Optionally, a payload of the moving object may have a low power consumption, such as less than about 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h or less.

14 illustrates a UAV 1400 , according to embodiments. The UAV may be an example of a moving object as described herein. The UAV 1400 can include a drive system that has four rotors 1402 . 1404 . 1406 and 1408 Has. Any number of rotors (eg, one, two, four, five, six, or more) may be provided. The rotors may be embodiments of the self-locking rotors described elsewhere herein. The unmanned aircraft rotors, rotor assemblies, or other propulsion systems may enable the unmanned aerial vehicle to hover / hold the position, change the orientation, and / or change position. The distance between shafts of opposing rotors may be any suitable length 1410 be. For example, the length 1410 less than or equal to 2 m or less than or equal to 5 m. In some embodiments, the length 1410 within a range of 40 cm to 1 m, from 10 cm to 2 m or from 5 cm to 5 m. Any description of a UAV herein may apply to a moving object, such as a moving object of another type, and vice versa.

In some embodiments, the moving object may be configured to carry a load. The load may include one or more of passengers, cargo, equipment, instruments, and the like. The load can be provided within a housing. The housing may be separate from a housing of the movable object or may be part of a housing for a movable object. Alternatively, the load may be provided with a housing while the movable object has no housing. Alternatively, portions of the load or the entire load may be provided without a housing. The load may be rigidly mounted relative to the moving object. Optionally, the load may be movable (eg, slidable or rotatable relative to the moving object) relative to the moving object.

In some embodiments, the load includes a payload 1412 one. The payload may be configured to perform no operation or function. Alternatively, the payload may be a payload configured to perform an operation or function, also known as a functional payload. For example, the payload may include one or more sensors for monitoring one or more targets. Any suitable sensor may be included in the payload, such as an image capture sensor (eg, a camera), an audio capture device (eg, a parabolic mirror microphone), an infrared imaging device, or an ultraviolet imaging device. The sensor may provide static sensor data (eg, a photograph) or dynamic sensor data (eg, a video). In some embodiments, the sensor provides sensor data for the payload's destination. Alternatively, the payload may include one or more transmitters for providing signals for one or more destinations. Any suitable transmitter may be used, such as a lighting source or a sound source. In some embodiments, the payload includes one or more transceivers, such as for communication with a module remote from the moving object. Optionally, the payload may be configured to interact with the environment or a target. For example, the payload may include a tool, instrument, or mechanism capable of handling objects, such as a robotic arm.

Optionally, the load can be a carrier 1414 lock in. The carrier may be provided for the payload, and the payload may be connected to the movable object via the carrier, either directly (eg, by directly touching the moving object) or indirectly (eg, by moving the movable object not touched). Conversely, the payload may be attached to the moving object without requiring a carrier. The payload may be integrally molded with the carrier. Alternatively, the payload may be detachably connected to the carrier. In some embodiments, the payload may include one or more payload elements, and one or more of the payload elements may be movable relative to the moveable object and / or the carrier, as described above.

The carrier may be integrally molded with the movable object. Alternatively, the carrier may be detachably connected to the movable object. The carrier can be connected directly to the moving object. The carrier may provide support for the payload (eg, carry at least a portion of the payload). The carrier may include a suitable attachment structure (eg, a gimbal platform) that is capable of stabilizing and / or directing the movement of the payload. In some embodiments, the carrier may be configured to control the state of the payload (eg, position and / or orientation) relative to the moving object. For example, the carrier may be configured to move relative to the movable object (eg, with respect to one, two or three degrees of displacement and / or one, two or three degrees of rotation) such that the payload maintains its position and / or orientation relative to a suitable reference frame, regardless of the movement of the mobile object. The frame of reference can be a fixed frame of reference (for example, the surrounding environment). Alternatively, the frame of reference may be a moving frame of reference (eg, the moving object, a payload target).

In some embodiments, the carrier may be configured to facilitate movement of the payload relative to the moveable object and / or the carrier. The movement may be a displacement in relation to up to three degrees of freedom (eg along one, two or three axes) or a rotation in relation to up to three degrees of freedom (eg around one, two or three axes) or one any suitable combination thereof.

In some cases, the carrier may include a carrier frame assembly and a carrier actuating assembly. The carrier frame assembly may provide structural support for the payload. The support frame assembly may include individual support frame components, some of which may be movable relative to each other. The carrier actuating assembly may include one or more actuators (eg, motors) that cause movement of the individual carrier frame components. The actuators may allow the movement of multiple support frame components simultaneously, or may be configured to facilitate movement of a single support frame component, respectively. The movement of the support frame components can create a corresponding movement of the payload. For example, the carrier actuating assembly may cause rotation of one or more carrier frame components about one or more axes of rotation (eg, roll axis, pitch axis, yaw axis). The rotation of the one or more support frame components may cause a payload to rotate about one or more axes of rotation relative to the movable object. Alternatively or in combination, the carrier actuating assembly may cause a displacement of one or more carrier frame components along one or more displacement axes and thereby generate a displacement of the payload along one or more corresponding axes relative to the movable object.

In some embodiments, the movement of the moving object, the carrier, and the payload may be controlled by a terminal relative to a fixed frame of reference (e.g., the surrounding environment) and / or one another. The terminal may be a remote control device at a location remote from the moving object, the carrier and / or the payload. The terminal may be mounted on or secured to a support platform. Alternatively, the terminal may be a hand-held or portable device. For example, the terminal may include a smartphone, a tablet, a laptop, a calculator, glasses, gloves, a helmet, a microphone, or any suitable combination thereof. The terminal may include a user interface such as a keyboard, a mouse, a joystick, a touch screen, or a display. Any suitable user input may be used to interact with the terminal, such as manually entered commands, voice control, gesture control, or position control (eg, about movement, position, or tilt of the terminal).

The terminal may be used to control any state of the moving object, the carrier and / or the payload. For example, the terminal may be used to control the position and / or orientation of the moving object, the carrier, and / or the payload in relation to a fixed frame of reference and / or to each other. In some embodiments, the terminal may be used to control individual elements of the moving object, the carrier, and / or the payload, such as the carrier's actuator assembly, a payload sensor, or a payload transmitter. The terminal may include a wireless communication device configured to communicate with one or more of the mobile object, the carrier, or the payload.

The terminal may include an appropriate display for viewing information of the moving object, the carrier and / or the payload. For example, the terminal may be configured to display information of the moving object, the carrier, and / or the payload with respect to position, displacement speed, displacement acceleration, orientation, angular velocity, angular acceleration, or any suitable combination thereof. In some embodiments, the terminal may display information provided by the payload, such as data provided by a functional payload (eg, images recorded by a camera or other image capture device).

Optionally, the same terminal may control both the moving object, the carrier and / or the payload or a state of the moving object, the carrier and / or the payload as well as receiving and / or displaying information from the moving object, the carrier, and / or the payload. For example, a terminal may control the position of the payload relative to an environment while displaying payload-picked images or payload position information. Alternatively, different terminals can be used for different functions. For example, a first terminal may control a movement or a state of the moving object, the carrier, and / or the payload, while a second terminal may receive and / or display information from the moving object, the carrier, and / or the payload. For example, a first terminal may control the positioning of the payload relative to an environment, while a second terminal may display image data picked up by the payload.

Various communication modes may be used between a moving object and an integrated terminal that both controls the moving object and receives data, or between the moving object and a plurality of terminals that both control the moving object and receive data. For example, at least two different communication modes may be formed between the mobile object and the terminal that controls both the mobile object and receives data.

15 illustrates a moving object 1500 that a carrier 1502 and a payload 1504 includes, according to embodiments. Although the moving object 1500 as depicted as an airplane, this figure is not intended to be limiting, and any suitable type of moving object may be used as previously described herein. One skilled in the art would recognize that any of the embodiments described herein in the context of aircraft systems may be applied to any suitable moving object (eg, a UAV). In some cases, the payload 1504 on the moving object 1500 be provided without the carrier 1502 to require. The moving object 1500 can drive mechanisms 1506 , a sensor system 1508 and a communication system 1510 lock in.

The drive mechanisms 1506 may include one or more of rotors, propellers, blades, engines, motors, wheels, axles, magnets, or nozzles as previously described. For example, the drive mechanisms 1506 self-locking rotors, rotor assemblies or other rotary drive units as disclosed elsewhere herein. The moving object may have one or more, two of the more, three or more, or four or more drive mechanisms. The drive mechanisms can all be of the same type. Alternatively, one or more drive mechanisms may be different types of drive mechanisms. The drive mechanisms 1506 may be attached to the movable object using any suitable means, such as a support member (eg, a drive shaft) as described elsewhere herein 1500 to be appropriate. The drive mechanisms 1506 can be attached to any suitable section of the moving object 1500 , such as attached to the top, bottom, front, back, sides, or any suitable combination thereof.

In some embodiments, the drive mechanisms 1506 allow the moving object 1500 lifts vertically from a surface or lands vertically on a surface without any horizontal movement of the moving object 1500 to require (for example, without going down a runway). Optionally, the drive mechanisms 1506 be functional in order to allow the moving object 1500 hovering in a specified position and / or orientation in the air. One or more of the drive mechanisms 1506 can be controlled independently of the other drive mechanisms. Alternatively, the drive mechanisms 1506 be configured to be controlled simultaneously. For example, the moving object 1500 have several horizontally oriented rotors that the moving object 1500 Can provide lift and / or thrust. The plurality of horizontally oriented rotors may be actuated to provide vertical lift, vertical landing, and hovering capabilities to the moving object. In some embodiments, one or more of the horizontally oriented rotors may rotate clockwise while one or more of the horizontally oriented rotors may rotate counterclockwise. For example, the number of rotors in the clockwise direction may be equal to the number of rotors in the counterclockwise direction. The rotational speed of each of the horizontally oriented rotors may be independently varied to control the lift and / or thrust generated by each rotor, and thereby the spatial arrangement, speed and / or acceleration of the moving object 1500 (eg with respect to up to three degrees of displacement and up to three degrees of rotation).

The sensor system 1508 may include one or more sensors that determine the spatial arrangement, speed, and / or acceleration of the moving object 1500 (eg, with respect to up to three degrees of displacement and up to three degrees of rotation). The one or the plurality of sensors may include any of the sensors previously described herein, including GPS sensors, motion sensors, inertial sensors, proximity sensors, or image sensors. The through the sensor system 1508 provided sensor data can be used to the spatial arrangement, speed and / or acceleration of the moving object 1500 (eg, using a suitable processing unit and / or a control module as described below). Alternatively, the sensor system 1508 be used to provide data relating to the environment surrounding the moving object, such as weather conditions, proximity to possible obstacles, position of geographical features, position of artificial structures, and the like.

The communication system 1510 allows communication with a terminal 1512 that is a communication system 1514 has, over wireless signals. The communication systems 1510 . 1514 include any number of transmitters, receivers and / or transceivers suitable for wireless communication. The communication can be one-way communication so that data can be transmitted in one direction only. For example, the one-way communication may only include that the moving object 1500 Data to the terminal 1512 or vice versa. The data may be from one or more transmitters of the communication system 1510 to one or more recipients of the communication system 1514 or vice versa. Alternatively, the communication may be two-way communication, allowing data in both directions between the moving object 1500 and the terminal 1512 can be transmitted. Two-way communication may involve transmitting data from one or more transmitters of the communication system 1510 to one or more recipients of the communication system 1514 and vice versa.

In some embodiments, the terminal may 1512 Control data one or more of the moving object 1500 , the carrier 1502 and the payload 1504 provide and information from one or more of the moving object 1500 , the carrier 1502 and the payload 1504 (eg, positional and / or moving information of the moving object, the carrier or the payload, data sensed by the payload, such as image data taken by a payload camera). In some cases, control data from the terminal may include instructions for relative positions, movements, or controls of the moving object, the carrier, and / or the payload. For example, the control data may be for modifying the position and / or orientation of the moving object (eg, via control of the drive mechanisms 1506 ) or a movement of the payload with respect to the movable element (eg via a control of the carrier 1502 ) to lead. The control data from the terminal may be used to control the payload, such as controlling the operation of a camera or other image capture device (eg, taking still or moving images, zooming in or out, turning on or off, switching image modes, etc.) image resolution, changing the focus, changing the depth of field, changing the exposure time, changing the aperture, changing the angle of view or field of view). In some cases, the communications from the moving object, the carrier, and / or the payload may include information from one or more sensors (eg, the sensor system 1508 the payload 1504 ) lock in. The communications may include sensed information from one or more different types of sensors (eg, GPS sensors, motion sensors, inertial sensors, proximity sensors, or image sensors). Such information may include the location (eg, location, orientation, movement, or acceleration of the moving object, the carrier, and / or the payload.) Such information from a payload may include data acquired by the payload or a sensed state of the payload through the terminal 1512 transmitted control data may be configured to control the state of one or more of the moving object 1500 , the carrier 1502 and the payload 1504 to control. Alternatively or in combination, the carrier may 1502 and the payload 1504 also include a communication module configured to communicate with the terminal 1512 to communicate so that the terminal is connected to each of the moving object 1500 , the carrier 1502 and the payload 1504 communicate and control them.

In some embodiments, the moving object 1500 be configured with another remote device in addition to the terminal 1512 or instead of the terminal 1512 to communicate. The terminal 1512 may also be configured with another remote device as well as the moving object 1500 to communicate. For example, the moving object 1500 and / or the terminal 1512 communicate with another moving object or a carrier or payload of a moving object. If desired, the other remote device may be a second terminal or other computing device (eg, a computer, laptop, tablet, smartphone, or other mobile device). The remote device may be configured to send data to the moving object 1500 to convey data from the moving object 1500 to receive data to the terminal 1512 to transmit and / or data from the terminal 1512 to recieve. Optionally, the remote device may be connected to the Internet or other telecommunications network, such as the mobile object 1500 and / or the terminal 1512 received data can be uploaded to a website or a server.

16 illustrates an exemplary system 1600 for tracking a moving object, according to embodiments. The system 1600 may be used in conjunction with any suitable embodiment of the systems, devices and methods disclosed herein. For example, the system 1600 be implemented or worn by a moving object. The system 1600 can be a sensor module 1602 , a processing unit 1604 , a non-volatile computer-readable medium 1606 , a control module 1608 and a communication module 1610 lock in.

The sensor module 1602 can use different types of sensors that detect information regarding the moving objects in different ways. Different types of sensors can send different types of signals or signals from different sources. For example, the sensors may include inertial sensors, PGS sensors, proximity sensors (eg, lidar), or vision / image sensors (eg, a camera). The sensor module 1602 can be effective to a processing unit 1604 be connected, which has multiple processors. In some embodiments, the sensor module may be operatively connected to a transmission module 1612 (eg, a WiFi image transmission module) configured to transmit sensor data directly to a suitable external device or system. For example, the transmission module 1612 to be used by a camera of the sensor module 1602 to transmit captured images to a remote terminal.

The processing unit 1604 may include one or more processors, such as a programmable processor (eg, a central processing unit (CPU)). For example, the processing unit 1604 a field programmable gate array (FPGA) and / or one or more ARM processors. The processing unit 1604 can effectively connect to a non-volatile computer-readable medium 1606 The non-volatile computer-readable medium 1606 can store logic, code, and / or program statements issued by the processing unit 1604 to perform one or more steps are executable. The non-transitory computer-readable medium may include one or more storage devices (eg, removable media or external storage such as an SD card or random access memory (RAM)). In some embodiments, data may be from the sensor module 1602 directly to the storage units of the non-volatile computer-readable medium 1606 be forwarded and stored within the same. The storage units of the non-volatile computer-readable medium 1606 can store logic, code and / or program statements by the processing unit 1604 are executable to carry out any suitable embodiment of the methods described herein. For example, the processing unit 1604 be configured to execute instructions that include one or more processors of the processing unit 1604 to perform the tracking functions discussed herein. The storage units can be processed by the processing unit 1604 store sensor data to be processed by the sensor module. In some embodiments, the storage units of the non-volatile computer-readable medium 1606 used by the processing unit 1604 store generated processing results.

In some embodiments, the processing unit 1604 effective to a control module 1608 be connected, which is configured to control a state of the movable object. For example, the control module 1608 be configured to control the drive mechanisms of the moving object to adjust the spatial arrangement, speed and / or acceleration of the moving object with respect to six degrees of freedom. Alternatively or in combination, the control module 1608 control one or more of a state of a carrier, a payload or a sensor module.

The processing unit 1604 can be effective to a communication module 1610 which is configured to send and / or receive data to one or more external devices (eg, a terminal, a display, or other remote control). Any means of communication may be used, such as cable communication or wireless communication. For example, the communication module 1610 use one or more of local area networks (LANs), nationwide networks (WANs), infra-red, wireless, Wi-Fi, point-to-point (P2P) networks, telecommunications networks, cloud communications, and the like. Optionally, relay stations such as masts, satellites or mobile stations may be used. Wireless communication can be proximity-dependent or proximity-independent. In some embodiments, for communications Visual connection may or may not be necessary. The communication module 1610 may include one or more of sensor data from the sensor module and / or by the processing unit 1604 send and / or receive generated processing results, predetermined control data or user commands from a remote terminal and the like.

The components of the system 1600 may be arranged in any suitable configuration. For example, one or more of the components of the system 1600 be arranged on the mobile object, the carrier, the payload, the terminal, the sensor system or an additional external device in communication with one or more of the above. Besides, though 16 a single processing unit 1604 and a single non-transitory computer-readable medium 1606 A person from the subject realizes that this is not intended to be limiting and that the system 1600 multiple processing units and / or non-volatile computer-readable media. In some embodiments, one or more of the plurality of processing units and / or non-transitory computer readable media may be at different locations, such as the moving object, the carrier, the payload, the terminal, the sensor module, an additional external device in FIG Communication with one or more of the above or suitable combinations thereof, such that any suitable aspect of the system 1600 executed processing and / or storage functions can take place at one or more of the aforementioned locations.

17 illustrates an exemplary system 1700 for controlling a moving object, according to embodiments. The system 1700 can be used to control a moving object, such as a UAV. The system 1700 may be used in combination with any suitable embodiment of the systems, devices and methods disclosed herein. For example, the system 1700 by a control station, such as described herein. The system 1700 can be an input module 1702 , a processing unit 1704 , a non-volatile computer-readable medium 1706 , a display module 1708 and a communication module 1710 all interconnected via a bus or similar network.

The input module 1702 may include one or more input mechanisms for receiving input from a user operating the input module. Such input mechanisms may include one or more joysticks, switches, knobs, slide switches, buttons, dials, touch screens, keypads, keyboard, mouse, voice controls, gesture controls, inertial sensors, and the like. Such an input module 1702 can be used to receive a user input that is used to control aspects of the moving object, the carrier, the payload, or a component thereof. Such aspects may include location, location, orientation, navigation, tracking, and the like. For example, the input mechanisms may be set to one or more positions by a user, each of the positions corresponding to a predetermined input for controlling the UAV.

In some embodiments, the input mechanisms may be manipulated by a user to input control commands for controlling navigation of the moving object. For example, the user may use a knob, switch, or similar input mechanism to enter a flying mode for the moving object, such as autopilot or navigation following a predetermined navigation path. As another example, the user may select the position, attitude, orientation, or controlling other aspect of the moving object by tilting the control station in certain ways, which can be detected and used by one or more inertial sensors to generate the corresponding navigation commands. As yet another example, the user may use the input mechanism to provide an operating parameter (eg, zoom) of the payload, a location and / or position of the payload (eg, about the carrier), or other aspects of any object on board of the moving object.

In some embodiments, the input mechanisms may be manipulated by a user to enter. For example, the user may use a knob, switch, or similar input mechanism to select the appropriate tracking mode, such as the manual tracking mode or the automatic tracking mode. The user may also use the input mechanisms to select a particular target to track, specify target type information, or other similar information. In various embodiments, the input module may be implemented by more than one device. For example, the input module may be implemented by a standard remote control with joystick (s) operatively coupled to a mobile device (eg, a smartphone) that runs suitable mobile application software (an "app"), the control instructions for the can generate a moving object. The app may be configured to receive input from a user.

The processing unit 1704 may include one or more processors, such as a programmable processor (eg, a central processing unit (CPU) or a microcontroller). The processing unit 1704 can be effective to a memory 1706 be connected. The memory 1706 may include volatile and / or non-volatile storage media configured to store data and / or logic, code, and / or program instructions issued by the processing unit 1704 to perform one or more routines or functions are executable. The memory may include one or more storage devices (eg, removable media or external storage such as an SD card or Random Access Memory (RAM)). In some embodiments, data may be from the input module 1702 directly to the storage units of the memory 1706 be forwarded and stored within the same. The storage units of the storage 1706 can store logic, code and / or program statements by the processing unit 1704 are executable to carry out any suitable embodiment of the methods described herein. For example, the processing unit 1704 be configured to execute instructions that include one or more processors of the processing unit 1704 cause to process and display sensor data (eg, images) received from the mobile object, to generate control commands including navigation commands and destination information based on user input, to cause the communication module 1710 Sending and / or receiving data, and the like. In some embodiments, the memory units of the memory 1706 used by the processing unit 1704 store generated processing results. Even though 17 a single processing unit 1704 and a single store 1706 A person would discern from the subject that this is not intended to be limiting and that the system 1700 may include multiple processing units and / or memory units of the memory.

In some embodiments, the display module 1708 be configured to display information of the moving object, the carrier and / or the payload with respect to position, displacement speed, displacement acceleration, orientation, angular velocity, angular acceleration or any suitable combination thereof. The display module 1708 may be configured to display information received from the mobile object and / or the payload, such as sensor data (eg, images captured by a camera or other image capture device), tracking data, such as described herein, control feedback data, and the like. In some embodiments, the display module 1708 be implemented by the same device, which is the input module 1702 implements. In other embodiments, the display module 1708 be implemented by a device that by the device, which is the input module 1702 implements, is separate (but which can be connected to the same effective).

The communication module 1710 may be configured to send and / or receive data from one or more remote devices (eg, the moving object, the payload, the base station, and the like). For example, the communication module 1710 be configured to communicate control data (eg, navigation commands, destination information, and tracking control data) to external systems or devices, such as moving objects, carriers, and / or payloads. The communication module 1710 may also be configured to receive data (eg, sensor data and tracking information) from such external systems or devices. In some embodiments, the communication module 1710 a transmitter 1712 and a receiver 1714 each configured to send and receive data to and from remote devices. In some embodiments, the communication module may include a transceiver that combines the functionalities of the transmitter and the receiver. In some embodiments, the transmitter and the receiver may communicate with each other as well as with the processing unit 1704 communicate. Any suitable communication means may be used, such as cable communication or wireless communication described herein.

The tracking methods and systems as described herein may have a variety of applications. Illustrated, for example 18 an exemplary application for the present invention, wherein a user 1802 a UAV 1804 to automatically track them while doing a variety of activities. As in 18 Illustrated, the user can use the UAV 1804 using a control station 1806 Provide destination information. The control station 1806 may provide a user interface for receiving user input to generate input information, such as described herein. For example, the user interface may be configured to have one or more images (eg, images or videos), such as by an imaging device 1805 recorded on board the UAV or elsewhere. The imaging device 1805 can have a carrier 1803 as described herein, may be connected to the UAV. The images may have been transmitted in real time or substantially in real time from the UAV to the control station. Alternatively, the images may be previously stored in the control station and / or provided or generated by other sources. Some of the pictures can show the user. The user may select herself from one of these images, for example, by touching or otherwise selecting an area of the display that corresponds to the user. In some cases, the user may specify destination type information rather than selecting specific destination information. Such target type information can be used to describe the type of goals that should be tracked, such as "a person with a red hat". The destination information, as specified by the user, may be provided to the UAV. In various embodiments, the destination information may be provided prior to or during the operation of the UAV. For example, the destination information may be provided when the UAV is in flight or landed. In some embodiments, the user may also specify the start time, end time, and / or duration of the tracking. Such timing information may be used to control the timing of the tracking operation.

The destination information may be provided by (a) processor (s) on board the UAV 1804 to be used by the user 1802 to track automatically, for example, when the user is doing sports (eg cycling or running). The user can move to remain essentially in place while being tracked while the imaging device 1805 aboard the UAV captures images of the user so that the resulting images maintain a predetermined size or position of the user. For example, the position, location, orientation, speed, zoom, and / or other characteristics of the UAV 1804 , the vehicle 1803 and / or the imaging device 1805 be set in real time to track the user. Such a setting may be automatically generated by the processor (s) aboard the UAV and without user intervention. Alternatively or additionally, the tracking may be performed with some user intervention (eg, adjusting the zoom of the imager). The predetermined size or position of the user may be specified by the user or according to a system preset. In some embodiments, the user may carry or otherwise carry an object to facilitate tracking of the user. Such an object may include, for example, a cap or clothing in light colors.

In some embodiments, images captured during tracking may be transmitted back to the control station or other suitable device by the imaging device or UAV for display, playback, storage, processing, or other purposes. Such transmission may occur in real time or substantially in real time as the images are captured by the imaging device. Alternatively, there may be a significant delay between taking and transmitting the images. In some embodiments, the images are stored in memory onboard the UAV without being communicated anywhere. The user can view such images in real time and adjust the target information or other aspects of the UAV or a component thereof, if necessary. The set destination information may be provided to the UAV, as discussed above, and the repeating process may continue until the desired images are obtained. In some embodiments, the images may be stored for later viewing or playback. In some embodiments, the images may be communicated from the UAV, the imager, and / or the control station to a remote server. For example, the images may be shared on a social network platform, such as Facebook or Twitter.

While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Individuals of the art will now come to numerous variations, changes and substitutions without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims (21)

System for controlling an unmanned aerial vehicle ( 101 ), UAV, the system comprising: one or more receivers configured to move one or more navigation commands from a remote controller (1) to move the UAV along a trajectory, and (2) destination information ( 116 ) by an imaging device ( 104 ), which have a carrier ( 102 ) to the UAV ( 101 ) is to receive to be tracked destination, wherein the carrier is configured to allow the imaging device (10) to 104 ) in relation to the UAV ( 101 ), and one or more processors configured to select between (1) an automatic tracking mode and (2) a manual tracking mode, wherein in the automatic tracking mode the one or more processors are configured for at least one of the UAV ( 101 ) or the carrier ( 102 ) as the UAV moves along the trajectory corresponding to the one or more navigation commands from the remote controller to track the destination in accordance with the destination information to maintain the destination remaining within an image captured by the imager. The system of claim 1, wherein the destination information comprises a GPS location of the destination. The system of claim 1, wherein the destination information comprises initial destination information. The system of claim 3, wherein the destination information is an initial location or an initial size of the destination ( 116 ) within one by the imaging device ( 104 ) included image. The system of claim 1, wherein the destination information comprises destination type information, wherein the destination type information preferably comprises color, texture or pattern information. The system of claim 5, wherein tracking the destination in accordance with the destination information further comprises identifying, based on the destination type information, the destination to be followed ( 116 ) from within the one or more imaging devices ( 104 ) captured images using an image recognition algorithm. The system of claim 1, wherein the target information includes expected target information, wherein the expected target information is an expected position or expected size of the target ( 116 ) within one by the imaging device ( 104 ) recorded image. The system of claim 7, wherein the expected size of the target ( 116 ) is the same as an initial size of the target or where the expected position of the target is the same as an initial position of the target ( 116 ). The system of claim 7, wherein tracking the target in accordance with the target information includes maintaining, within a predetermined tolerance level, the expected position, or the expected size of the target. 116 ) within one or more of the imaging devices ( 104 ) included images. The system of claim 1, wherein the one or more receivers are configured to receive the wireless signals from the target comprising position information about the target substantially in real time. The system of claim 1, wherein the destination information includes expected destination information and wherein tracking the destination ( 116 ) according to the destination information comprises: determining current destination information of the destination ( 116 ) based on one or more by the imaging device ( 104 ), detecting a deviation of the current destination information from the expected destination information and calculating a setting on the UAV ( 101 ), the carrier ( 102 ) or the imaging device ( 104 ) so as to substantially correct the deviation. The system of claim 11, wherein the deviation is with a change in the position of the target ( 116 ) and the calculated setting is at an angular velocity for the imaging device ( 104 ) relative to a yaw axis or pitch axis of the imaging device ( 104 ) is linked. The system of claim 12, wherein the calculated setting is configured to provide control signals to the wearer ( 102 ), so as to cause the imaging device ( 104 ) in relation to the UAV ( 101 ) emotional. A system according to claim 11 or 13, wherein a warning signal is provided if the calculated setting falls outside the predetermined range, the predetermined range corresponding to a predetermined range of the control lever amount of a control system having a navigation control system for the UAV ( 101 ) or a control system for the carrier ( 102 ). The system of claim 11, wherein tracking the target ( 116 ) comprises comparing the calculated setting with a predetermined maximum threshold and providing the predetermined maximum threshold if the calculated setting exceeds the predetermined maximum threshold, or Wherein the tracking of the target ( 116 ) comparing the calculated setting with a predetermined minimum threshold and providing the predetermined minimum threshold, if the calculated setting is less than the predetermined minimum threshold. The system of claim 1, wherein the destination information is received from a remote control device accessible to the remote user. The system of claim 16, wherein the remote control device is configured to: one or more by the imaging device (10); 104 ) captured images from the UAV ( 101 ) to receive the one or more images, to receive a user selection of a destination from within a displayed image, the destination information of the destination based on the user selection of the destination ( 116 ) and send the target information to the UAV ( 101 ). Device for controlling an unmanned aerial vehicle ( 101 ), UAV, which includes an ad ( 1202 ) configured to carry one or more by one via a carrier ( 102 ) to the UAV ( 101 ) connected imaging device ( 104 ), an input device ( 1204 . 1205 ) configured to select a user selection of a destination ( 1304 . 1310 . 1316 ) from within at least one of the one or more on the display ( 1202 receive one or more processors, configured individually or collectively, for target information of the destination based, at least in part, on the user selection of the destination (FIG. 1304 . 1310 . 1316 ) and a sender configured to provide the target information for the UAV ( 101 ) so as to enable the UAV ( 101 ) autonomously the target ( 1304 . 1310 . 1316 ) in accordance with the target information to keep the target substantially within the field of view of the imager. The apparatus of claim 18, wherein the destination information comprises initial destination information, destination type information, and / or expected destination information. Apparatus according to claim 18, wherein the user selection of the target ( 1304 . 1310 . 1316 ) is achieved by a user viewing a region of the at least one of the one or more on the display ( 1202 ), with the selected area ( 1304 . 1310 . 1316 ) corresponds. The apparatus of claim 18, wherein the one or more processors, individually or in common, are configured to display on the display ( 1202 ) the selected destination ( 1304 . 1310 . 1316 ) in response to the user selection of the target ( 1304 . 1310 . 1316 ) with a selection indicator ( 1306 . 1310 . 1318 ), the selection indicator ( 1306 . 1310 . 1318 ) indicates that the destination ( 1304 . 1310 . 1316 ) has been selected by the user.

Images