KR20240031628A - Virtual reality device, server and method for controlling drone swarm using immersive virtual reality - Google Patents

Abstract

A method for controlling a drone cluster based on virtual reality is disclosed. The method for controlling a drone cluster comprises the steps of: displaying, by a virtual reality headset of a virtual reality device, a three-dimensional terrain and a drone cluster projected on a virtual reality space according to a user operation; selecting, by a virtual reality controller of the virtual reality device, a small drone cluster inserted into a region of interest set on the three-dimensional terrain in the drone cluster according to the user operation; and dividing, by a server, the region of interest into a plurality of cells by using the region of interest and the small drone cluster as user input information, allocating the divided cells to drones included in the small drone cluster, respectively, and automatically generating a three-dimensional flight path of the drones allocated to each cell.

Description

Virtual reality device, server, and immersive virtual reality-based drone swarm control method {VIRTUAL REALITY DEVICE, SERVER AND METHOD FOR CONTROLLING DRONE SWARM USING IMMERSIVE VIRTUAL REALITY}

The present invention relates to drone swarm control technology, and more specifically, to technology for controlling a drone swarm using immersive virtual reality.

Recently, high-performance observation and communication equipment has been installed on drones. Accordingly, drones are widely used as a platform for collecting observation data for various purposes at low cost and high efficiency.

Drones can observe a wide area, so when using multiple drones, a wider area can be observed. In this case, it is necessary to set up a mission plan that divides a large area into a plurality of areas, assigns multiple drones to each of the divided areas, and sets the flight path of the drones assigned to each area.

The task of setting up such a mission plan is performed by control personnel through traditional input interfaces such as keyboard, mouse, and screen touch. As the number of drones increases, the time for setting up the mission plan through the traditional input interface increases. Additionally, setting up a mission plan through a traditional input interface is a difficult task even for experienced users.

The purpose of the present invention to solve the above-described problems is to dramatically shorten the time required to set a mission plan for a drone swarm by combining immersive virtual reality technology and automatic mission plan setting technology The goal is to provide virtual reality devices, servers, and drone swarm control methods that can control swarms.

The drone swarm control method according to one aspect of the present invention for achieving the above-described object includes the steps of a virtual reality headset of a virtual reality device displaying a three-dimensional terrain and drone swarm projected in a virtual reality space according to user manipulation; A virtual reality controller of the virtual reality device selects, from within the drone swarm, a small-scale drone swarm to be inserted into a region of interest set on the three-dimensional terrain according to the user's operation, and the server selects the region of interest and the small-scale drone swarm. Using the drone swarm as user input information, the area of interest is divided into a plurality of cells and the divided cells are respectively assigned to the drones included in the small drone swarm, and 3 of the drones assigned to each cell The method may include automatically generating a dimensional flight path.

A virtual reality device according to another aspect of the present invention includes a virtual reality headset that displays a 3D terrain and a drone cluster projected in a virtual reality space according to a user's manipulation, and a region of interest on the 3D terrain according to the user's manipulation. It includes a virtual reality controller that sets and selects a small drone swarm to be deployed in the set area of interest from within the drone swarm. Here, the virtual reality controller may configure the set area of interest and the selected small drone swarm with user input information and transmit it to a server that automatically sets a mission plan for the small drone swarm.

In an embodiment, the virtual reality headset includes a communication interface for receiving virtual reality information from a server, a processor for processing the virtual reality information, and, according to control of the processor, a virtual reality space included in the virtual reality information, and the virtual reality. It may include a display unit that displays the 3D terrain and drone cluster projected in real space.

In an embodiment, the display unit may display a line surrounding the area of interest set by the virtual reality controller and drones included in the small drone group deployed into the area of interest as outlines of different specific fluorescent colors. .

In an embodiment, the virtual reality controller uses a sensor that detects movement of the virtual reality controller, a virtual guide line and a virtual pointer that moves in the virtual reality space according to the movement of the virtual reality controller, to create the three-dimensional terrain. A processor for setting an area of interest on the image, selecting a small drone swarm to be introduced into the set area of interest, and generating user input information including the set area of interest and the selected small drone swarm, and the user input information It may include a communication interface for transmitting to the server.

A server according to another aspect of the present invention includes an area of interest set on a three-dimensional terrain projected from a virtual reality device into a virtual reality space, a small drone group deployed in the area of interest, and each of the drones included in the small drone group. A first communication interface (e.g., 212 in FIG. 4) that receives user input information including the mission performance altitude, dividing the area of interest into a plurality of cells and including the divided cells in the small drone swarm. A processor (222 in FIG. 4: 222A, 222B and 222C) that allocates to each of the drones and automatically calculates a 3D flight path including the coverage path of the drone assigned to each cell and the mission performance altitude (222A, 222B and 222C in FIG. 4) and 3 It may include a second communication interface (eg, 238 in FIG. 4) that transmits mission planning information including a dimensional flight path to drones included in the small drone swarm.

In an embodiment, the processor (222 in FIG. 4) may perform an operation to divide the region of interest into a number of cells equal to the number of drones included in the small drone swarm.

In an embodiment, the processor (222 in FIG. 4) may perform an operation to allocate the divided cells to drones included in the small drone swarm using a task allocation algorithm.

In an embodiment, the processor (222 in FIG. 4) may automatically calculate the 3D flight path of the drone assigned to each cell using a path planning algorithm.

According to the present invention, by generating user input information for setting a swarm mission plan using a VR device, the time required for a user to set up a swarm mission plan can be dramatically reduced.

In existing swarm control, the user grasps the current status and mission situation of the drone swarm through a limited two-dimensional monitor/display, whereas in the present invention, the current status and mission situation of the drone swarm is identified using a VR device, so the user The three-dimensional distribution and flight path of a drone colony can be recognized quickly and easily by considering spatial perspective. Additionally, complex terrain, such as mountainous terrain and urban environments, can be quickly recognized through the virtual 3D terrain displayed on the VR device.

In addition, in existing swarm control, the user sets the mission plan for each drone using traditional input interfaces such as a keyboard, mouse, and touch screen, but in the present invention, input can be intuitively used using a VR device, so one person Users can set up mission plans for drone swarms.

Additionally, in the present invention, the serial process of creating a mission plan for each drone and uploading it to each drone can be automatically processed by the server.

Figure 1 is a configuration diagram of an immersive VR-based drone swarm control system according to an embodiment of the present invention.
Figure 2 is a diagram for explaining virtual reality information displayed through a display device of a VR headset according to an embodiment of the present invention.
Figure 3 is a diagram illustrating the process of setting an area of interest using a VR controller according to an embodiment of the present invention and the process of selecting a small drone swarm to be assigned a specific task for the set area of interest within the drone swarm. am.
Figure 4 is a configuration diagram of a server according to an embodiment of the present invention.
Figure 5 is a diagram for explaining a region division process performed by a region division module according to an embodiment of the present invention.
Figure 6 is a diagram for explaining the coverage path creation process performed by the path creation module according to an embodiment of the present invention.
Figure 7 is a diagram for explaining a 3D flight path generated by the path creation module 222C according to an embodiment of the present invention.
Figure 8 is a block diagram of a VR headset and VR controller according to an embodiment of the present invention.
9A and 9B are diagrams to explain the results of simulating a swarm mission plan according to an embodiment of the present invention.
Figure 10 is a flowchart illustrating a method for controlling a drone swarm based on realistic virtual reality according to an embodiment of the present invention.

Drones can observe a wide area, and when multiple drones are used, a wider area can be observed. Hereinafter, a large number of drones used to observe a wider area are referred to as a drone swarm.

Quickly and effectively controlling a drone swarm consisting of multiple drones (e.g., 10 or more drones) and setting appropriate flight paths and mission plans for each drone is a difficult task even for experienced control personnel.

In particular, when using drone swarms in disaster situations or battlefield situations that require rapid information collection, changes in the mission environment in which the drone swarm will be mapped (e.g., facility deformation, forest fire spread, green/red tide spread in rivers, etc.) are frequent, and drone operation Objectives (e.g., reconnaissance areas of interest to air traffic control personnel at the current time) may change at any time.

In this dynamic mission environment, technology is needed to quickly change the pre-mission plan uploaded to each drone before takeoff (mission replanning) during mission performance. However, in a scenario using a drone swarm, the control burden on control personnel (operators) for mission re-planning increases rapidly. This is because most drone control software (e.g. QGroundControl, UGCS, etc.) currently used for drone swarm control runs on PCs, mobile devices, tablets, etc. that provide traditional input interfaces such as keyboard, mouse, screen touch, etc., so the user's real-time responsiveness is limited. Because it falls.

The user recognizes the drone information displayed by the drone control software through a two-dimensional monitor or display screen, and enters the drone mission command intended by the user into the drone control software through a traditional input interface. During this input process, the following three types of time delays may occur.

(A) Swarm situation awareness: The time it takes for the user to accurately recognize the current status and mission situation of the drone swarm through a two-dimensional monitor or display screen.

(B) User decision-making: The time required for users to judge and design a mission plan (mission plan for a drone swarm) suitable for changing situations.

(C) Giving swarm command: Time for the user to input detailed mission plan for each drone into the control software using keyboard, mouse, screen touch, etc.

Traditional drone control software has fundamental limitations in shortening the time required according to (A), (B), and (C) above. Moreover, when using a swarm of 10 or more drones, the time required for each step increases exponentially.

Accordingly, the present invention aims to dramatically shorten the time required for each step (A), (B), and (C) in a scenario utilizing a drone swarm, that is, the time required to set a mission plan for the drone swarm. Provides a system and method for controlling drone swarms by combining immersive virtual reality technology and automatic mission planning technology.

More specifically, the present invention provides an interface technology for controlling a drone swarm through a realistic VR (Virtual Reality) device, a technology for monitoring a drone swarm using a realistic VR headset, and a drone swarm using a VR controller. We propose technology that intuitively defines mapping mission information for swarms and swarm mission planning technology that automates mapping missions over a wide area using drone swarms.

Through this, the present invention can dramatically shorten the time required for 'swarm situation recognition', 'user decision-making', and 'swarm command giving', and can provide the effect of reducing the operational burden on control personnel.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the attached drawings. In order to facilitate overall understanding when describing the present invention, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

The terms used herein are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an idealized or excessively formal sense. No.

Figure 1 is a configuration diagram of an immersive VR-based drone swarm control system according to an embodiment of the present invention.

Referring to FIG. 1, the realistic VR-based drone swarm control system 500 according to an embodiment of the present invention includes a VR (Virtual Reality) device 100, a server 200, and a drone swarm 300, It may further include first and second networks 150 and 250 that support communication connections between these 100, 200, and 300.

The VR device 100 and the server 200 may exchange commands, data, or information through a wired or wireless communication method through the first network 150. Here, the first network 150 may include, for example, a computer network such as Wi-Fi Direct, Bluetooth, a local area network (LAN), or a wide area network (WAN), or the Internet.

The server 200 and the drone swarm 300 may exchange commands, data, or information through wireless communication through the second network 250. Wireless communications include, for example, LTE, LTE Advance (LTE-A), 5G, code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), Wireless Broadband (WiBro), or GSM. It may include cellular communication using at least one of (Global System for Mobile Communications), etc. Accordingly, the second network 250 may be a network supporting such wireless communication or an independent network including ad-hoc and mobile base stations.

In this document, the first network 150 and the second network 250 are distinguished, but there is no need to distinguish them, and they can be described as one network. In this case, wireless communication is, for example, LTE, LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), Wireless Broadband (WiBro), or It may include cellular communication using at least one of GSM (Global System for Mobile Communications), etc.

According to one embodiment, wireless communication is, for example, WiFi (wireless fidelity), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, near field communication (NFC), Magnetic Secure Transmission, radio. It may include at least one of frequency (RF) or body area network (BAN).

According to one embodiment, the wireless communication may be GNSS. GNSS may be, for example, Global Positioning System (GPS), Global Navigation Satellite System (Glonass), Beidou Navigation Satellite System (hereinafter “Beidou”), or Galileo, the European global satellite-based navigation system. In this document, “GPS” can be used interchangeably with “GNSS.”

Wired communications referred to in this document include, for example, at least one of USB (universal serial bus), HDMI (high definition multimedia interface), RS-232 (recommended standard232), power line communications, or POTS (plain old telephone service). It can contain one.

The VR device 100 is a device to support a realistic user experience (UX), and may basically include a VR headset 110 and a VR controller (or VR motion controller) 130.

The VR headset 110 is a type of wearable device mounted on the user's head and may be a Head Mounted Display (HMD) device.

The VR headset 110 executes or plays VR content received from the server 200 through the first network 150 and outputs a two-dimensional or three-dimensional virtual reality space. In this document, the VR headset 110 is described as running or playing VR content to output a three-dimensional virtual reality space, but is not limited to this, and includes AR (Augmented Reality) content, MR (Mixed Reality) ) content, ER (Extended Reality) content, or SR (Substitutional Reality) content can be run or played to output AR space, MR space, ER space, or SR space.

The virtual reality space is displayed by a display device included in the VR headset 110, and the user recognizes the virtual reality space displayed by the display device while wearing the VR headset 110 on the head.

Additionally, a user may input various commands (e.g., drone swarm command) to a virtual reality space and/or a virtual object (e.g., drone swarm) projected in the virtual reality space, and the means for enabling such input is a VR controller. It is (130).

The VR controller 130 is a device used to input various commands into a virtual reality space. The user holds the VR controller 130 with both hands and uses arm movements and/or hand movements such as motions, gestures, grabs, etc. to create virtual reality. You can enter various commands in real space.

In order to recognize the user's arm motion and/or hand motion, the VR controller 130 may be configured to include a plurality of motion recognition sensors. In addition, the VR controller 130 may further include analog input means such as a joystick, joystick thumbstick, trackpad, button, etc., and can provide various input methods by combining the sensing results of the analog input means and the motion recognition sensor. This can be implemented.

The VR controller 130 may be paired with the VR headset 110 according to a wireless communication (short-range wireless communication) method. When the VR controller 130 is paired with the VR headset 110, the VR headset 110 can receive movement information of the VR controller 130 according to the user's arm movements and/or hand movements from the VR controller 130. there is.

The VR headset 110 can display the pointer pointed by the VR controller 130 according to the movement information of the VR controller 130 in the virtual reality space displayed by the display device built into the VR headset 110, and the pointer can move freely in the virtual reality space according to the movement information of the VR controller 130.

The user can set a mission plan for the drone swarm 300 projected in the virtual reality space by moving the pointer. The mission plan set by the user using the VR headset 110 and/or VR controller 130 is transmitted to the server 200 through the first network 150. Hereinafter, the mission plan set by the user using the VR headset 110 and/or VR controller 130 is referred to as “user input information.”

This user input information may include various information for setting a mission plan for the drone swarm 300. For example, user input information may include “drone information” and “mission information.”

Drone information is provided to drones (small-scale drones swarm selected from the drone swarm) that are deployed in the area of interest (or mapping mission area of interest) set in the virtual reality space by the user using the VR controller 130. It includes information on the number of drones included, takeoff/landing altitude of the drones, horizontal movement speed of the drones, ascent and descent speed of the drones, and takeoff/landing location information of the drones.

Mission information may include location information indicating the area of interest, mission altitude of drones deployed to the area of interest, horizontal movement speed, photo taking interval, etc.

Among these drone information and mission information, information related to speed, time and altitude is

Settings can be made through setting items in a virtual menu window projected in the virtual reality space by the user using the VR controller 130.

The server 200 receives the user input information including the mission information and the drone information from the VR device 100 (VR headset 110 or VR controller 130) through the first network 150. And the server 200 automatically sets a mission plan for the drone swarm 300 or a small drone swarm selected within the drone swarm 300 based on the received user input information. The automatic mission plan setting process performed by the server 200 will be described in detail below.

The drone swarm 300 includes multiple drones (drone entities). The number of drones forming the drone swarm 300 may be 10 to 100 or more.

In the drone operation area, an available communication network can be established in the form of a commercial network such as an LTE network or 5G network, or an independent network such as a mobile base station or an ad-hoc network.

Each drone is equipped with a communication module such as an LTE modem, 5G modem, etc., and can communicate with the server 200 through the established available communication network. Additionally, each drone is equipped with sensing mission equipment (cameras, etc.), a mission computer (MC), and a flight controller computer (FC).

Each drone uses the communication module to transmit its flight status information as a Telemetry (TM) stream to the server 200 based on a commercial protocol for drone control (e.g., Mavlink protocol), and A TeleCommand (TC) defining flight control and mission planning may be received from the server 200.

Figure 2 is a diagram for explaining virtual reality information displayed through a display device of a VR headset according to an embodiment of the present invention.

Referring to FIG. 2, the user 10 sees realistic 3D virtual reality information (or 3D virtual reality space) through the display screen of the VR headset 110 while wearing the VR headset 110 on the head. can do.

Virtual reality information is information projected on the virtual plane 20 and includes a virtual 3D terrain object 21, a virtual 3D facility object, a virtual drone swarm object 22, and drone status information 23. .

Figure 2 shows a 3D terrain object 21 representing mountainous terrain as an example.

The virtual drone swarm object 22 includes a plurality of virtual drone objects, and FIG. 2 shows the virtual drone swarm object 22 consisting of 13 virtual drone objects as an example.

Drone status information 23 may be projected on the virtual plane 20 in the form of text or numbers. Drone status information may include, for example, location information indicating the location, longitude, and altitude of each drone, battery status information of each drone, and communication connection status of each drone. Battery status information may be information indicating the battery charging status. Drone status information may be projected around the virtual drone object.

Photographic images or videos obtained through actual shooting by each drone may be further projected on an arbitrary area of the virtual plane 20.

The user can view 3D virtual reality information from various observer viewpoints through the display device of the VR headset 110. The observer viewpoint is, for example, a viewpoint looking at the virtual drone swarm object 22 from above, a viewpoint looking at the virtual drone swarm object 22 from below, a viewpoint focusing on the main mission point/main facility, and an individual virtual drone. This includes perspectives from a nearby location.

Switching the observer's viewpoint may be performed, for example, through movement of the VR headset 110 according to the user's head movement and manipulation of physical input means (e.g., buttons, joystick, trackpad, etc.) of the VR controller 130. there is.

Figure 3 is a diagram illustrating the process of setting an area of interest using a VR controller according to an embodiment of the present invention and the process of selecting a small drone swarm to be assigned a specific task for the set area of interest within the drone swarm. am.

Referring to Figure 3, first, the area of interest 35 for performing a specific mission such as reconnaissance, exploration, and observation is a virtual guide line 30 generated by the VR controller 130 and an end of the virtual guide line. It can be set by the pointer 32.

The virtual guide line 30 is displayed in the virtual reality space in the direction in which the VR controller 130 is facing. The virtual pointer 32 is displayed on a virtual plane where the end of the virtual guide line 30 is located.

The user draws a line 34 surrounding the area of interest 35 on the virtual plane 20 using the virtual guide line 30 and the virtual pointer 32 generated in the virtual reality space by the VR controller 130. A polygonal area of interest 35 can be set on the virtual plane 20 through a drawing gesture or dragging operation. The line 34 surrounding the set region of interest 35 may be displayed as an outline of a specific fluorescent color.

Additionally, the user may use the virtual guide line 30 and the virtual pointer 32 to set specific mission points on the virtual plane 20 rather than the area. In this case as well, specific set mission points may be displayed in a specific fluorescent color.

Once the polygonal shape area of interest (35) is set, a process of selecting a small-scale drones swarm that will be assigned a specific mission for the set area of interest is performed. The small-scale drone swarm is also operated by a VR controller ( It can be selected using the virtual guide line 30 and the virtual pointer 32 created by 130).

The user uses the virtual guide line 30 and the virtual pointer 32 to draw a line 36 on the virtual plane 20 surrounding the drones to be assigned a mission among all the drones included in the drone swarm. Small drone swarms can be selected through dragging movements, etc. Figure 3 shows an example of selecting a small drone cluster consisting of 10 drones out of 13 drones.

The selected small drone group may be displayed as an outline of a specific fluorescent color different from the specific fluorescent color of the line 34 surrounding the set region of interest 35. This document uses the example of selecting a small drone swarm, but a single drone can also be selected. Selected small drone swarms and unselected small drone swarms can be distinguished and displayed with different fluorescent colors.

Figure 4 is a configuration diagram of a server according to an embodiment of the present invention.

Referring to FIG. 4, the server 200 may be an intermediary device that connects the VR device 100 and the drone swarm 300. The server 200 implements virtual reality expressed by the VR device 100, and creates a drone swarm and individual members included in the drone swarm based on user input information (drone information and mission information) received from the VR device 100. The mission plan of a drone or a small drone group selected within the drone group can be automatically set.

The server 200 must be capable of real-time graphic calculation and information processing to implement virtual reality (virtual reality space), so it may be composed of server devices equipped with high-end processors and memory.

For this purpose, the server 200 includes a VR server 210, a mission plan automatic setting server 220, and a drone control server 230. This document uses three servers as an example, but is not limited to this. For example, one of the three servers (210, 220, and 230) is integrated with one of the other two servers, or the three servers (210, 220, and 230) are integrated into one server. can be integrated.

The VR server 210 may basically include a communication interface 212, a processor 214, a storage medium 216, and a memory 218.

The communication interface 212 may communicate with the VR device 100 wired or wirelessly through the first network (150 in FIG. 1). The communication interface 212 may be composed of known hardware components to support wired or wireless communication.

The processor 214 is a hardware component that controls and manages the operations of the peripheral components 212, 126, and 218, and performs real-time graphics calculations and information processing to implement a virtual reality space. To this end, the processor 214 may be comprised of at least one Central Processing Unit (CPU), at least one Graphics Processing Unit (GPU), at least one Micro-Controller Unit (MCU), or a combination thereof. there is.

Storage medium 216 is a non-volatile storage medium. A VR-specific program built through a VR (AR, MR, ER, or SR) development engine such as Unreal Engine, Unity, etc. may be stored in the storage medium 216. Additionally, the storage medium 216 may further store drone data, such as drone flight logs, videos and images captured and acquired by the drone, etc.

The memory 218 is a hardware component including volatile and/or non-volatile memory, and temporarily stores instructions and data related to a VR-specific program. The VR-specific program stored in the storage medium 216 can be executed by loading related instructions and data into the memory 218 under the control of the processor 214.

The VR dedicated program renders 3D virtual reality information (or 3D virtual reality space) under the control of the processor 214, and the rendered 3D virtual reality information is connected to the communication interface 212 and the first network 150. It is transmitted in real time to the VR headset 110 of the VR device 100.

The VR headset 110 displays the rendered 3D virtual reality information received from the VR server 210 through a display screen, so that a user wearing the VR headset 110 can respond to the rendered 3D virtual reality information. It becomes possible to perceive a 3D virtual reality space.

The processor 214 stores the user's realistic interaction data (headset angle, controller position, angle, button input, etc.) received from the VR device 100 through the first network 150 and the communication interface 212 and the storage medium. Based on the drone data such as flight logs, images, videos, etc. input from (216), the data required for rendering (drone current location, etc.) is projected into the 3D virtual reality space.

In addition, the processor 214 transmits user input information (drone information and mission information) received from the VR device 100 through the first network 150 and the communication interface 212 to the mission plan automatic setting server 220. do. At this time, the user input information may be directly transmitted to the mission plan automatic setting server 220 without going through the VR server 210. In this case, the mission plan automatic setting server 220 is directly connected to the VR device 100. It may include a communication interface for communication.

The mission plan automatic setting server 220 automatically sets the mission plan for the drone swarm or a small drone swarm selected within the drone swarm based on user input information (drone information and mission information) transmitted through the VR server 210. . To this end, the mission plan automatic setting server 220 includes a processor 222, a storage medium 224, and a memory 226, and directly receives user input information (drone information and mission information) from the VR device 100. In this case, although not shown in FIG. 4, a communication interface may be further included.

The processor 222 is a hardware component that controls and manages the operation of the surrounding components 224 and 226, and provides real-time graphic calculations and information to automatically set a mission plan for a drone swarm or a small drone swarm selected within a drone swarm. Perform processing. To this end, the processor 222 may be comprised of at least one Central Processing Unit (CPU), at least one Graphics Processing Unit (GPU), at least one Micro-Controller Unit (MCU), or a combination thereof. there is.

The storage medium 224 is a non-volatile storage medium, and stores a software module that is executed and controlled by the processor 222 to automatically set a mission plan for a drone swarm or a small drone swarm. Here, the software module includes an area division module 222A, a mission assignment module 222B, and a path creation module 222C. Additionally, the storage medium 224 may further store current location information, mission performance speed, and altitude information for each drone.

Memory 226 is a hardware component that includes volatile and/or non-volatile memory and provides execution space for software modules 222A, 222B, and 222C stored in storage medium 224. The software modules 222A, 222B, and 222C stored in the storage medium 216 are loaded into the memory 218 and executed under the control of the processor 214.

The region division module 222A executed by the processor 214 divides the region of interest (35 in FIG. 3) set by the user based on the user input information received from the VR device 100 (110 and 130) into a plurality of cells ( cells). At this time, the user input information includes information indicating a polygonal area of interest (35 in FIG. 3) (hereinafter referred to as area of interest information) and information on the number of drones included in the small drone group selected by the user. The region of interest information may be coordinate information of a virtual line (34 in FIG. 3) surrounding the region of interest (35 in FIG. 3) or coordinate information (virtual pixel coordinates) of points constituting the virtual line (34 in FIG. 3). ) can be.

The area division module 222A divides the polygonal area of interest (35 in FIG. 3) into cells equal to the number of drones included in the small drone group. To segment the polygon's region of interest (35 in FIG. 3) in real time, a region division algorithm such as centroidal Voronoi tessellation may be used. Hereinafter, the region division process according to the above technique will be described in detail with reference to FIG. 5.

Figure 5 is a diagram for explaining a region division process performed by a region division module according to an embodiment of the present invention.

Referring to FIG. 5, in S52, a sufficiently large number of random sampling points (e.g., 10,000 or more) are generated inside the polygonal region of interest 50, and then these points are selected through the k-means algorithm. A process of clustering into clusters equal to the number of drones included in the drone swarm is performed.

Next, in S54, the polygonal area of interest 50 is equal to the number of drones included in the selected small drone cluster using Voronoi Partitions based on the centroid 53 for each cluster. Divide into number of cells.

Figure 5 shows an example of dividing the polygonal area of interest 50 into five cells when the number of drones included in the selected small drone cluster is five.

Referring again to FIG. 4, the mission allocation module 222B performs an operation to allocate each cell divided by the area division module 222A to each drone included in the selected small drone swarm. A known task assignment algorithm can be used to solve the mathematical problem of assigning M tasks (M partitioned cells) to N agents (N drones). Examples of task allocation algorithms may include linear sum assignment algorithm, sequential greedy assignment algorithm, Hungarian algorithm, etc.

The path creation module 222C performs an operation to create (plan) a coverage path that can sufficiently cover the entire area of the cell to which each drone is assigned. The coverage path may be, for example, a zigzag path inside the cell.

Figure 6 is a diagram for explaining the coverage path creation process performed by the path creation module according to an embodiment of the present invention.

Referring to FIG. 6, assuming that the user has set a pentagonal region of interest 60 using the VR device 100, the path creation module 222C first creates line segments constituting the pentagon. ), a random line segment 61 is selected, and an operation is performed to find the starting point 63 closest to the vertex 62 of the selected random line segment 61. Here, the arbitrary line segment may be the longest line segment or the shortest line segment, and in this document, the longest line segment is assumed to be an arbitrary line segment.

Next, the path creation module 222C performs an operation to find a scan direction parallel to the longest line segment 61.

Next, the path creation module 222C determines the number of stripes (S1 to S6) formed along the scan direction, waypoints, and the distance (d) between adjacent stripes (S1 to S6). ) and calculates consecutive waypoints. Finally, the path creation module 222C creates a coverage path by designating the start point 63 and the end point 65 of the last stripe S6.

The coverage path generation method may use a variety of coverage path planning algorithms depending on the geometric complexity of the cell, for example, a back-and-forth coverage path planning algorithm, or a coverage path planning algorithm used in vehicles or robots. Autonomous driving algorithms, etc. may be used.

In addition, in order to prevent collisions between drones moving to each cell, the path creation module 222C reflects the mission performance altitude for each drone and/or the takeoff/landing altitude for each drone included in the user input information in the coverage path to create a 3D data for each drone. The flight path can be finally confirmed.

Figure 7 is a diagram for explaining a 3D flight path generated by the path creation module 222C according to an embodiment of the present invention.

In Figure 7, the three-dimensional flight path for two drones (UAV1 and VAV2) is shown. The three-dimensional flight path of the first drone (UAV1) generated by the path creation module 222C includes a first takeoff/landing altitude 71 and a first input path 72 according to the first takeoff/landing altitude 71. , a mission altitude (73), a first coverage path (74) according to the mission altitude (73), and a first return path (75) according to the first takeoff/landing altitude (71).

The three-dimensional flight path of the second drone (UAV2) includes a second takeoff/landing altitude (76) higher than the first takeoff/landing altitude (71), and a second input path according to the second takeoff/landing altitude (76) ( 77), a mission altitude 73, a second coverage path 78 according to the mission altitude 73, and a second return path 79 according to the second takeoff/landing altitude 76.

Figure 7 shows an example where the mission altitude of two drones (UAV1 and VAV2) is the same, but is not limited to this, and the mission altitudes of the two drones (UAV1 and VAV2) may be different. there is.

Referring again to FIG. 4, the drone control server 230 performs control of all drones in the drone swarm through the second network (250 in FIG. 1), and all data packets (telemetry, Save captured images, captured videos, etc.)

The drone control server 230 receives drone status information in the form of telemetry (TM) through a drone control protocol (e.g., Mavlink protocol). The drone control server 230 uses a telecommand (TC) defined in the drone control protocol (Mavlink, etc.) to provide flight control information according to the mission plan (e.g., 3D flight path) transmitted from the mission plan automatic setting server 220. ) and transmit it to each drone.

This drone control server 230 includes a processor 232, a storage medium 234, a memory 236, and a communication interface 238.

The processor 232 is a hardware component that controls and manages the operations of the peripheral components 234, 236, and 226. The processor 232 loads and executes the drone control program stored in the storage medium 234 into the memory 236, and executes the executed program. Process and process the intermediate data and result data generated by.

In addition, the processor 232 processes and stores all data packets (telemetry, flight log (position, angle, battery status, etc.), captured images, captured videos, etc.) received from the drone in the storage medium 234. Alternatively, it can be transmitted to the VR server 210. The VR server 210 configures the information processed and processed by the processor 232 into virtual reality information so that the user can recognize it in the virtual reality space and transmits it to the VR device 100.

The processor 232 may include at least one Central Processing Unit (CPU), at least one Graphics Processing Unit (GPU), at least one Micro-Controller Unit (MCU), or a combination thereof. Storage medium 234 is a non-volatile storage medium, and memory 236 includes volatile and non-volatile memory. The communication interface 238 performs wireless communication with the drone swarm based on a drone control protocol (Mavlink, etc.).

Figure 8 is a block diagram of a VR headset and VR controller according to an embodiment of the present invention.

Referring to FIG. 8, the VR headset 110 includes a processor 111, a memory 112, a first communication interface 113, a display unit 114, and at least one device for detecting movement of the VR headset 110. It may include a sensor 115 and a second communication interface 116.

The processor 111 can control and manage the operations of the memory 112, the first communication interface 113, the display unit 114, at least one sensor 115, and the second communication interface 116, and various Data processing and calculations can be performed.

The processor 111 may be implemented, for example, as a system on chip (SoC). The processor 111 may further include at least one CPU and/or at least one GPU. The processor 111 may load commands or data received from at least one of the other components (eg, non-volatile memory) into the volatile memory, process the commands, and store the resulting data in the non-volatile memory.

The memory 112 is a hardware component that provides an execution space for storing and executing instructions and programs for processing virtual reality information received by the processor 111 from the server 200 through the second communication interface 116. , may include internal memory or external memory. Built-in memory includes, for example, volatile memory (such as DRAM, SRAM, or SDRAM), non-volatile memory (such as one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, flash memory, etc. , a hard drive, or a solid state drive (SSD). The external memory may include a flash drive, for example, compact flash (CF), secure digital (SD), Micro-SD, It may include Mini-SD, xD (extreme digital), MMC (multi-media card), memory stick, etc. The external memory may be physically connected to the VR headset 110 through various interfaces.

The first communication interface 113 can communicate with the VR controller 130 to exchange various information through short-distance wireless communication such as Bluetooth or Wi-Fi. The first communication interface 113 may include a WiFi module, a Bluetooth module, etc.

The display unit 114 displays three-dimensional terrain information projected into a virtual reality space according to virtual reality information received from the server 200 through the second communication interface 116 under the control of the processor 111 and the three-dimensional terrain. Drone clusters (e.g., virtual drone clusters) overlapping with information can be displayed. Additionally, the display unit 114 can display 3D terrain information and drone clusters converted to various viewpoints by the processor 111.

The sensor 115 can detect the movement of the VR headset 110 and the viewpoint from which the VR headset 110 is viewed, and convert the measured or sensed information into an electrical signal. The sensor 115 may include, for example, a gesture sensor, a gyro sensor, an acceleration sensor, an eye tracking sensor, etc.

The second communication interface 116 may communicate with the server 200 through the first network 150 to exchange various information. The second communication interface 116 may include, for example, a cellular module, a WiFi module, a Bluetooth module, a GNSS module, an NFC module, and an RF module. At least some (e.g., two or more) of the cellular module, WiFi module, Bluetooth module, GNSS module, or NFC module may be included in one integrated chip (IC) or IC package. For example, the RF module may transmit and receive communication signals (e.g., RF signals). The RF module 229 may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna.

The VR control device 130 includes a processor 131, a memory 132, a first communication interface 133, a sensor 134 for detecting the movement of the control device 130, and a second communication interface 135. can do.

The processor 131 controls and manages the operation of the memory 132, the first communication interface 133, the sensor 134 for detecting the movement of the control device 130, and the second communication interface 135, and data Processing and calculations can be performed. The processor 131 may be implemented, for example, as a system on chip (SoC). The processor 131 may further include at least one CPU and/or at least one GPU.

The processor 131 provides information on the area of interest set within the three-dimensional terrain projected on the virtual reality space according to the movement of the VR controller 130, information on the small drone swarm to be input into the area of interest (e.g., information on the small drone swarm) Generates user input information configured to include information on the number of drones included) and other information (mission altitude of drones included in a small drone swarm), and sends it to the server 200 through the second communication interface 135. Can be sent.

The memory 132 is a hardware component that provides an execution space for storing and executing instructions and programs for processing the virtual reality space displayed by the VR headset 110, and may include internal memory or external memory.

The first communication interface 133 may communicate with the first communication interface 113 of the VR headset 113 by wire or wirelessly. The first communication interface 133 and the first communication interface 113 may be paired through wired or wireless communication.

The sensor 134 can detect the movement of the VR controller 130 and the direction the VR controller 130 is pointing, and convert the measured or sensed information into an electrical signal. The sensor 134 may include, for example, a gesture sensor, a gyro sensor, an acceleration sensor, etc.

The second communication interface 135 may communicate with the server 200 through the first network 150 to exchange various information. The second communication interface 135 may include, for example, a cellular module, a WiFi module, a Bluetooth module, a GNSS module, an NFC module, and an RF module. At least some (e.g., two or more) of the cellular module, WiFi module, Bluetooth module, GNSS module, or NFC module may be included in one integrated chip (IC) or IC package. For example, the RF module may transmit and receive communication signals (e.g., RF signals). The RF module 229 may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), or an antenna.

The VR headset 110 can execute or reproduce virtual reality information from the server 200 and output 3D terrain and drone clusters projected on the virtual reality space through the display unit 114.

Additionally, the VR headset 110 may display a real terrain image or a real terrain video that was actually captured. The VR headset 110 can display a virtual drone swarm rendered on a real terrain image or a real terrain video. The VR headset 110 can simulate user actions in a virtual reality space.

The VR headset 110 can display a virtual guide line indicating the direction pointed by the VR controller 130 and a virtual pointer that is the end point of the virtual guide line in the virtual reality space. The pointer may move on the three-dimensional terrain according to the movement of the VR controller 130.

9A and 9B are diagrams to explain the results of simulating a swarm mission plan according to an embodiment of the present invention.

Referring to FIG. 9A, when 5 drones are assigned to each of the 5 cells in an area of interest divided into 5 cells, the user can check the simulation results of the swarm mission plan as shown in FIG. 9A.

The simulation result 90 shows the coverage path set in the five cells as viewed from above, and can be transmitted from the server 200 to the VR headset 110, and the VR headset 110 displays the simulation result 90 as a virtual By projecting it into real space and providing it to the user, the user can visually check whether his/her intended swarm mission plan is appropriate through the VR headset 110.

Additionally, the user can check the simulation result 91 expressed in a three-dimensional coordinate system as shown in FIG. 9B. The simulation result (91) shows the coverage path and mission performance altitude set for the five cells in a three-dimensional coordinate system, and the user can check whether the swarm mission plan reflecting the user's intended mission performance altitude is appropriate through the VR headset (110). You can check it visually.

Figure 10 is a flowchart illustrating a method for controlling a drone swarm based on realistic virtual reality according to an embodiment of the present invention.

Referring to FIG. 10, first, in S1010, a process of displaying the three-dimensional terrain and drone cluster projected in the virtual reality space is performed in the virtual reality headset 110 of the virtual reality device 100 according to user manipulation. .

Next, in S1020, the virtual reality controller 130 performs a process of selecting a small group of drones to be inserted into the area of interest set on the three-dimensional terrain within the drone group according to the user's operation.

Next, the server 200 uses the area of interest and the small drone cluster as user input information, divides the area of interest into a plurality of cells, and divides the divided cells into drones included in the small drone cluster. The process of assigning each cell to each cell and automatically generating the 3D flight path of the drone assigned to each cell is performed.

In an embodiment, S1010 is the user's manipulation of drawing a line surrounding the area of interest on the three-dimensional terrain using a virtual guide line indicating a direction pointed by the virtual reality controller and a virtual pointer indicating an end point of the virtual guide line. It may include the process of setting the area of interest of the polygon. Here, the line may be displayed as an outline of a specific fluorescent color.

In an embodiment, S1010 uses a virtual guide line indicating the direction pointed by the virtual reality controller and a virtual pointer indicating the end point of the virtual guide line to determine which drone to be deployed to the area of interest among all drones included in the drone swarm. It may include a process of selecting the small drone group including drones to be deployed in the area of interest according to the user's manipulation of drawing a line surrounding the fields. Here, the drones included in the small drone swarm may be displayed as outlines in a specific fluorescent color.

In an embodiment, S1010 is a process of dividing the region of interest into cells with the same number of cells as the number of drones included in the small drone swarm, and dividing the divided cells into drones included in the small drone swarm using a task allocation algorithm. It may include a process of assigning each cell to each cell and automatically generating a 3D flight path of the drone assigned to each cell using a path planning algorithm.

In an embodiment, the process of dividing the region of interest into a number of cells equal to the number of drones included in the small drone swarm includes generating a plurality of random sampling points inside the region of interest, using a clustering algorithm. A process of clustering the random sampling points into clusters equal to the number of drones included in the small drone swarm and dividing the area of interest based on the centroid of each cluster using the Voronoi Partitions method. Accordingly, a process of dividing into cells equal to the number of drones included in the small drone swarm may be included.

In an embodiment, the task allocation algorithm may be any one of a linear sum assignment algorithm, a sequential greedy assignment algorithm, a Hungarian algorithm, and a combination thereof.

In an embodiment, the process of automatically generating the 3D flight path includes, when the area of interest is set as a polygon, setting a starting point closest to the vertex of a random line segment selected from among the line segments constituting the polygon. A process of setting a scan direction parallel to the arbitrary line segment, a process of setting the number of stripes formed according to the scan direction and the distance between adjacent stripes, and the set number of stripes and adjacent stripes. It may include a process of generating the three-dimensional flight path including a coverage path generated in a zigzag shape based on the distance between them.

In an embodiment, the process of automatically generating the 3D flight path may include automatically generating the 3D flight path including the mission performance altitude for each drone included in the user input information. Here, the mission performance altitude for each drone may be set by the virtual reality controller through setting items in a virtual menu window projected in the virtual reality space according to the user's manipulation.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements can be made by those skilled in the art using the basic concept of the present invention defined in the following claims. It falls within the scope of rights.

Claims (19)

the virtual reality headset displaying a projected three-dimensional terrain and drone swarm in the virtual reality space according to user operations;
selecting, by a virtual reality controller, a small drone swarm within the drone swarm to be inserted into a region of interest set on the three-dimensional terrain according to the user's manipulation; and
The server divides the area of interest into a plurality of cells based on user input information indicating the area of interest and the small drone swarm, and allocates the divided cells to drones included in the small drone swarm. After that, the step of automatically generating the 3D flight path of the drone assigned to each cell.
Drone swarm control method including. In paragraph 1:
The selection step is,
A polygonal area of interest is created according to the user's manipulation of drawing a line surrounding the area of interest on the three-dimensional terrain using a virtual guide line indicating the direction pointed by the virtual reality controller and a virtual pointer indicating the end point of the virtual guide line. Drone swarm control method including setting steps. In paragraph 2,
A drone swarm control method in which the line is displayed as an outline in a specific fluorescent color. In paragraph 1:
The selection step is,
Drawing a line surrounding drones to be deployed in the area of interest among all drones included in the drone swarm using a virtual guide line indicating the direction pointed by the virtual reality controller and a virtual pointer indicating the end point of the virtual guide line. Selecting the small drone group including drones to be deployed to the area of interest according to user operation
Drone swarm control method including. In paragraph 4,
A drone swarm control method wherein the drones included in the small drone swarm are displayed as outlines in a specific fluorescent color. In paragraph 1:
The generating step is,
Dividing the area of interest into a number of cells equal to the number of drones included in the small drone swarm;
Allocating the divided cells to drones included in the small drone swarm using a task allocation algorithm; and
Step of automatically generating the 3D flight path of the drone assigned to each cell using a path planning algorithm
Drone swarm control method including. In paragraph 6:
The step of dividing the area of interest into a number of cells equal to the number of drones included in the small drone swarm,
generating a plurality of random sampling points inside the region of interest;
Clustering the random sampling points into clusters with a number equal to the number of drones included in the small drone swarm using a clustering algorithm; and
Dividing the region of interest into cells equal to the number of drones included in the small drone cluster according to the Voronoi Partitions method based on the centroid of each cluster.
Drone swarm control method including. In paragraph 6:
The task allocation algorithm is,
A drone swarm control method that is any one of a linear sum assignment algorithm, a sequential greedy assignment algorithm, a Hungarian algorithm, and a combination of these algorithms. In paragraph 6:
The step of automatically generating the 3D flight path is,
When the region of interest is set as a polygon, setting a starting point closest to a vertex of an arbitrary line segment selected from among the line segments constituting the polygon;
setting a scan direction parallel to the arbitrary line segment;
Setting the number of stripes formed according to the scanning direction and the distance between adjacent stripes, and
Generating the three-dimensional flight path including a coverage path generated in a zigzag shape based on the set number of stripes and the distance between adjacent stripes.
Drone swarm control method including. In paragraph 6:
The step of automatically generating the 3D flight path is,
A drone swarm control method comprising automatically generating the 3D flight path including the mission performance altitude for each drone included in the user input information. In paragraph 10:
The mission performance altitude for each drone is,
A drone swarm control method in which the virtual reality controller is set through setting items in a virtual menu window projected in a virtual reality space according to the user's manipulation. A virtual reality headset that displays three-dimensional terrain and drone swarms projected in virtual reality space, depending on user operations; and
A virtual reality controller that sets an area of interest on the three-dimensional terrain according to the user's operation and selects a small drone swarm to be inserted into the set area of interest from within the drone swarm,
The virtual reality controller is,
A virtual reality device that configures the set area of interest and the selected small drone swarm with user input information and transmits it to a server that automatically sets a mission plan for the small drone swarm. In paragraph 12
The virtual reality headset is,
a communication interface for receiving virtual reality information from a server;
a processor that processes the virtual reality information; and
A display unit that displays the virtual reality space included in the virtual reality information, the three-dimensional terrain and the drone cluster projected into the virtual reality space, under the control of the processor.
A virtual reality device comprising: In paragraph 13:
The display unit is,
A virtual reality device that displays, under control of the processor, a line surrounding the area of interest set by the virtual reality controller and an outline of drones included in the small group of drones introduced into the area of interest as outlines of different specific fluorescent colors. In paragraph 13:
The virtual reality controller is,
A sensor that detects movement of the virtual reality controller;
Using virtual guidelines and virtual pointers that move in the virtual reality space according to the movement of the virtual reality controller, an area of interest is set on the three-dimensional terrain, and a small group of drones to be introduced into the set area of interest are selected, , a processor that generates user input information including the set area of interest and the selected small drone group; and
Communication interface for transmitting the user input information to the server
A virtual reality device including a. User input information including an area of interest set on a three-dimensional terrain projected from a virtual reality device into a virtual reality space, a small drone swarm to be deployed in the area of interest, and the mission performance altitude of each drone included in the small drone swarm. a first receiving communication interface;
Dividing the area of interest into a plurality of cells and assigning the divided cells to drones included in the small drone swarm, respectively, 3 including the coverage path of the drone assigned to each cell and the mission performance altitude A processor that automatically calculates dimensional flight paths; and
A second communication interface for transmitting mission planning information including the 3D flight path to drones included in the small drone swarm.
Server containing . In paragraph 16:
The processor,
A server that performs an operation to divide the area of interest into a number of cells equal to the number of drones included in the small drone swarm. In paragraph 16:
The processor,
A server that performs an operation to allocate the divided cells to drones included in the small drone swarm using a task allocation algorithm. In paragraph 16:
The processor,
A server that automatically calculates the 3D flight path of the drone assigned to each cell using a path planning algorithm.