Unmanned aerial vehicle autonomous target searching system and method based on large language model
Technical Field
The invention belongs to the technical field of unmanned aerial vehicle autonomous navigation and target searching, and particularly relates to an unmanned aerial vehicle autonomous target searching system and method based on a large language model.
Background
Along with the rapid development of unmanned aerial vehicle technology, unmanned aerial vehicle is more and more widely applied in fields such as military reconnaissance, search rescue, environmental monitoring, etc. Traditional unmanned aerial vehicle search systems mainly rely on preset flight paths and target positions, lack flexibility and autonomy, and are difficult to cope with complex and changeable environments and task demands. In recent years, with the development of artificial intelligence and deep learning technology, unmanned aerial vehicle autonomous navigation and target recognition capability are remarkably improved, but some technical bottlenecks still exist.
Conventional unmanned aerial vehicle systems generally rely on data of a single sensor for environment sensing and target recognition, and are easily affected by environmental noise and sensor errors, so that sensing accuracy and robustness are insufficient. Meanwhile, when the existing path planning algorithm is used for processing dynamic obstacles in a complex environment, the calculation efficiency and the instantaneity still need to be improved. In addition, the conventional unmanned aerial vehicle task planning system generally adopts traditional methods such as rules, finite state machines and the like, is difficult to process complex and unstructured task descriptions, and limits the autonomous decision making capability and task execution efficiency of the unmanned aerial vehicle.
Disclosure of Invention
The invention aims to provide an unmanned aerial vehicle autonomous target searching system and method based on a large language model, so as to solve the technical problems.
In order to solve the technical problems, the unmanned aerial vehicle autonomous target searching system and method based on the large language model have the following specific technical scheme:
The unmanned aerial vehicle autonomous target searching system based on the large language model comprises a natural language processing module, a multi-sensor fusion module, a control and flight module, a target detection module, a path planning module and a task planning and control module, wherein the system processes task description and state information in a natural language form by utilizing the general large language model to realize unmanned aerial vehicle autonomous planning and execute complex searching tasks;
The natural language processing module is composed of a main body by a general large language model, and outputs a task planning result according to a current task target and state information, wherein the task target is a text in a natural language form, the state information is usually a text in a natural language form for describing a current scene and other information related to the task, and the task planning result is a task frame generated by the general large language model on the basis of input information and comprises task steps and execution sequences;
The multi-sensor fusion module is used for receiving environmental data from a plurality of sensors and preprocessing the acquired sensor data;
The control and flight module is used for receiving flight instructions and controlling the movement of the unmanned aerial vehicle, adjusting the flight track of the unmanned aerial vehicle according to a preset path and a real-time position, adjusting the gesture of the unmanned aerial vehicle in real time to keep flight stability, and managing and adjusting the power system of the unmanned aerial vehicle to realize high-efficiency flight;
The target detection module acquires environment data from the multi-sensor fusion module, synchronizes and fuses the multi-sensor data, processes the fused data by using a deep learning algorithm to detect and identify a target, and determines the accurate coordinates of the target by calculation based on the position of the detected target in the environment;
The task planning and control module calls a corresponding path planning module and a corresponding control and flight module to execute corresponding task steps according to a task framework, and ensures the task to be carried out according to the conversion of the control state of the target detection result given by the target detection module.
The invention also discloses an unmanned aerial vehicle autonomous target searching method of the unmanned aerial vehicle autonomous target searching system based on the large language model, which comprises the following steps:
S1, controlling an unmanned aerial vehicle to perform environment map modeling by using a laser radar in a multi-sensor fusion module;
S2, the task planning and control module enters an initial waiting state and controls the unmanned aerial vehicle to fly to an initial appointed place;
S3, after judging that the unmanned aerial vehicle reaches an initial designated place, the task planning and control module enters a waiting task frame state, and receives task targets and state information in a natural language form by calling a natural language processing module to generate a task frame comprising task steps and execution sequences;
S4, after the task planning and control module receives the execution plan, entering an action execution state, and calling a path planning module according to steps in a task frame to enable the unmanned aerial vehicle to fly to a designated search place;
s5, after the task planning and control module judges that the unmanned aerial vehicle reaches a target place, the unmanned aerial vehicle enters a searching target state, invokes a path planning module and performs round-robin searching at the target place;
S6, the task planning and control module continuously calls the result of the target detection module, if the target is found, the step 7 is skipped, and if the target is not found after the target place searching is finished, the step 9 is skipped;
s7, after the unmanned aerial vehicle is judged to find a target, the task planning and control module enters a target positioning state, and a multi-sensor fusion module is called to determine target coordinates and report the target coordinates;
S8, after judging that the unmanned aerial vehicle reaches the upper air of the target place, the task planning and control module enters a reset state, invokes the path planning module, flies to the initial designated place, returns to the step 3, and waits for the input of the next target searching task;
s9, at the moment, after searching the target place, the designated target is not found, and the task planning and control module enters a waiting task framework state again.
Further, the state equation corresponding to the environment map modeling in the step1 is as follows:
xk=f(xk-1,uk)+wk
Where x k is the state vector at time k, f (·) is a nonlinear state transfer function, u k is the control input, w k is the process noise, and the observation equation is as follows:
zk=h(xk)+vk
where z k is the observation data at time k, h (·) is a nonlinear observation function, representing the relationship from the state vector to the observation data, and v k is the observation noise.
Further, in S2, the unmanned aerial vehicle flight control uses PID control to ensure the flight from the current position to the target site, and the PID control formula is as follows:
Wherein e (t) is the error between the current position and the target position, and K p,Ki,Kd is the proportional, integral and differential gains of the PID controller.
Further, the S3 receives the task object g t in natural language form and the status information S t including the history search information h t and the scene map information o t by calling the natural language processing module M n to generate an execution plan including n task steps and execution sequencesThe natural language processing module formula is as follows:
pt=Mn(gt,st)。
Further, after the S4 task planning and control module receives the execution plan p t, it enters an action execution state, and according to the steps in the task framework, invokes the path planning module to fly the unmanned aerial vehicle to the designated search location, where the formula of the path planning algorithm a is as follows:
f(n)=g(n)+h(n)
Where f (n) is the total estimated cost of the current node, g (n) is the actual cost from the starting node to the current node, and h (n) is the estimated cost from the current node to the target node.
Further, the path generation formula in S5 is as follows:
x(t)=r·cos(ωt)
y(t)=r·sin(ωt)
where r is the radius, ω is the angular velocity, and t is the time.
Further, the target detection module M d in S6 gives the target detection probability and the position of the target in the camera coordinate system, and the formula is as follows:
ximage,yimage,Pobj=Md(I)
Where (x image,yimage) represents the coordinates of the object in the image, P obj represents the probability that the object is present, and I is the input image.
Further, the camera internal reference matrix in S7 converts the pixel coordinates into normalized coordinates in the camera coordinate system, and the formula is as follows:
Here, K -1 is an inverse of the camera's reference matrix, and the camera's reference matrix converts three-dimensional coordinates in the camera's coordinate system into three-dimensional coordinates in the world coordinate system, and the formula is as follows:
Where R is the rotation matrix and t is the translation vector.
Further, in S9, the current task object and the execution action are added into the history search information h t+1, and then the task object g t+1 in the form of natural language and the state information S t+1 including the history search information h t+1 and the scene map information o t+1 are received again by calling the natural language processing module to generate an execution plan including n task steps and execution sequences for the next roundJump to step 4.
The unmanned aerial vehicle autonomous target searching system and method based on the large language model have the advantages that the universal large language model is adopted to process natural language input, so that the system can quickly and accurately understand task targets. By automatically generating the task frame and the steps, human intervention is reduced, continuous target searching tasks are automatically completed, and the efficiency and accuracy of target searching are improved. According to the invention, a plurality of sensor data are integrated through the multi-sensor fusion module, so that denoising, calibration and synchronization of the data are realized, and the accuracy and the robustness of environmental awareness are improved. The complementarity of the different sensors enhances the adaptability of the system to complex environments. The invention processes the multi-sensor fusion data by using a deep learning algorithm through the target detection module, and has high-precision and high-robustness target detection and recognition capability. Through accurate calculation of the target position, the capability of the unmanned aerial vehicle for autonomously searching the target in a complex environment is improved. According to the invention, the optimal flight path is generated by combining the real-time environment data with the preset target through the path planning module, and the dynamic obstacle avoidance is performed in the flight process, so that the unmanned aerial vehicle can safely and quickly reach the target position. The real-time adjustment capability of the module adapts to the dynamically-changing environment, and the autonomy and flexibility of the unmanned aerial vehicle are enhanced. According to the invention, the stability and the high efficiency of the unmanned aerial vehicle in the flight process are ensured by adjusting the flight track and the gesture in real time through the control and flight module. And the flight performance is optimized by managing the power system, so that the endurance time and the task execution capacity of the unmanned aerial vehicle are prolonged. The system can operate in various complex environments, is suitable for various application scenes such as disaster relief, environment monitoring, patrol security and logistics transportation and the like, and has wide application prospect and market value. By combining advanced technologies such as natural language processing, multi-sensor fusion, deep learning and the like and through the organic combination of the modules, the intelligent and high-efficiency of the unmanned aerial vehicle autonomous target search system is realized.
Drawings
FIG. 1 is a schematic diagram of a policy network based on a pre-trained language model in the present invention.
FIG. 2 is a schematic diagram of the process of the present invention.
FIG. 3 is a schematic diagram of a memory adapter modification action in accordance with the present invention.
Detailed Description
In order to better understand the purpose, structure and function of the present invention, the following describes in further detail an unmanned aerial vehicle autonomous target searching system and method based on a large language model with reference to the accompanying drawings.
As shown in FIG. 1, the unmanned aerial vehicle autonomous target searching system based on the large language model comprises a natural language processing module, a multi-sensor fusion module, a control and flight module, a target detection module, a path planning module and a task planning and control module.
The natural language processing module is composed of a main body by a general large language model, and outputs a task planning result according to a current task target and state information, wherein the task target is a text in a natural language form, the state information is usually a text in a natural language form for describing a current scene and other information related to the task, and the task planning result is a task frame generated by the general large language model on the basis of input information and comprises task steps and execution sequences;
The multi-sensor fusion module can receive environmental data from a plurality of sensors, including but not limited to cameras, lidars, infrared sensors and inertial measurement units, and can pre-process acquired sensor data, including denoising, calibration and synchronization;
the control and flight module can receive flight instructions and control the movement of the unmanned aerial vehicle, can adjust the flight track of the unmanned aerial vehicle according to a preset path and a real-time position, can adjust the attitude of the unmanned aerial vehicle in real time to maintain flight stability, and can manage and adjust the power system of the unmanned aerial vehicle to realize high-efficiency flight;
The target detection module acquires environment data from the multi-sensor fusion module, can synchronize and fuse the multi-sensor data to improve the accuracy and the robustness of environment sensing, can process the fused data by using a deep learning algorithm to detect and identify a target, and can determine the accurate coordinates of the target by calculation based on the position of the detected target in the environment;
the path planning module can generate an optimal path of the unmanned aerial vehicle based on the environmental data and the preset target generated by the multi-sensor fusion module, and detect and avoid obstacles in real time in the flight process of the unmanned aerial vehicle;
The task planning and control module can call the corresponding path planning module and control and flight module to execute corresponding task steps according to the task framework, and can control the state conversion according to the target detection result given by the target detection module, so that the task is ensured to be carried out according to the steps.
According to the system, a natural language processing module is used as a decision core, tasks of the unmanned aerial vehicle can be planned in real time through task targets and state information in a natural language form, and environment perception and target detection are realized by utilizing multi-sensor data and a target detection module, so that efficient and accurate target searching task execution is realized.
As shown in fig. 2, the unmanned aerial vehicle autonomous target searching method based on the large language model of the invention comprises the following steps:
S1, controlling the unmanned aerial vehicle to perform environment map modeling by using a laser radar in a multi-sensor fusion module. The corresponding state equation is as follows:
xk=f(xk-1,uk)+wk
Where x k is the state vector at time k, f (·) is a nonlinear state transfer function, u k is the control input, and w k is the process noise. The observation equation is as follows:
zk=h(xk)+vk
where z k is the observation data at time k, h (·) is a nonlinear observation function, representing the relationship from the state vector to the observation data, and v k is the observation noise.
S2, the task planning and control module enters an initial waiting state, and the unmanned aerial vehicle is controlled to fly to an initial appointed place. Unmanned aerial vehicle flight control typically uses PID control to ensure flight from a current location to a target site. The PID control formula is as follows:
Wherein e (t) is the error between the current position and the target position, and K p,Ki,Kd is the proportional, integral and differential gains of the PID controller.
S3, after judging that the unmanned aerial vehicle arrives at the initial designated place, the task planning and control module enters a waiting task framework state, and receives a task target g t in a natural language form and state information S t containing history search information h t and scene map information o t through calling a natural language processing module M n to generate an execution plan containing n task steps and execution sequencesThe natural language processing module formula is as follows:
pt=Mn(gt,st)
S4, after the task planning and control module receives the execution plan p t, the task planning and control module enters an action execution state, and calls the path planning module according to the steps in the task framework, so that the unmanned aerial vehicle flies to the appointed searching place. The path planning algorithm formula is as follows:
f(n)=g(n)+h(n)
Where f (n) is the total estimated cost of the current node, g (n) is the actual cost from the starting node to the current node, and h (n) is the estimated cost from the current node to the target node.
S5, after the task planning and control module judges that the unmanned aerial vehicle reaches the target place, the unmanned aerial vehicle enters a searching target state, and the path planning module is called to perform round-robin searching at the target place. The path generation formula is as follows:
x(t)=r·cos(ωt)
y(t)=r·sin(ωt)
where r is the radius, ω is the angular velocity, and t is the time.
S6, the task planning and control module continuously calls the result of the target detection module, if the target is found, the step 7 is skipped, and if the target is not found after the target place searching is finished, the step 9 is skipped. The object detection module M d gives the object detection probability and the position of the object in the camera coordinate system, and the formula is as follows:
ximage,yimage,Pobj=Md(I)
Where (x image,yimage) represents the coordinates of the object in the image, P obj represents the probability that the object is present, and I is the input image.
S7, after the unmanned aerial vehicle is judged to find the target, the task planning and control module enters a target positioning state, a multi-sensor fusion module is called to determine target coordinates and report the target coordinates, and then a path planning module is called to fly to a target place. The camera internal reference matrix converts the pixel coordinates into normalized coordinates in the camera coordinate system, and the formula is as follows:
Here K -1 is the inverse of the camera reference matrix. The camera's extrinsic matrix converts the three-dimensional coordinates in the camera's coordinate system to three-dimensional coordinates in the world's coordinate system, as follows:
Where R is the rotation matrix and t is the translation vector.
S8, after the unmanned aerial vehicle is judged to reach the upper air of the target place, the unmanned aerial vehicle enters a reset state, the path planning module is called, the unmanned aerial vehicle flies to the initial appointed place, the step 3 is returned, and the next input of the target searching task is waited.
S9, when no specified target is found after searching the target place, the task planning and control module enters a waiting task frame state again, adds the current task target and execution action into history searching information h t+1, then receives the task target g t+1 in natural language form and state information S t+1 containing history searching information h t+1 and scene map information o t+1 through calling the natural language processing module again, and generates an execution plan containing n task steps and execution sequence in the next roundJump to step 4.
As shown in fig. 2, the solution overall frame is built up in this form. The task planning and control module transmits instructions such as a construction map or acquiring environment data to the multi-sensor fusion module, then the multi-sensor fusion module returns an environment map or an environment photo, the channel 2 transmits current task targets and state information to the natural language processing module, then the natural language processing module returns a task frame containing task steps and execution sequences, the channel 3 transmits destination coordinates to the path planning module, then the path planning module returns a planning result, the channel 4 transmits the path planning to the control and flight module, then the control and flight module returns a flight state to the path planning module, and the channel 5 transmits search target information and environment picture data to the target detection module, and then the target detection module transmits the detection result to the task planning and control module.
As shown in fig. 3, one application scenario of the present invention is for a cargo-carrying unmanned aerial vehicle to search for a designated recipient at a designated receiving point. After the system receives the customer order, the logistics system generates a task instruction comprising goods information, a receiving address, an appointed addressee and the like. The natural language processing module analyzes the instruction and generates a task frame, wherein the unmanned aerial vehicle carries goods to take off from the warehouse A and fly to the receiving point B. And searching the receiver with the specified characteristics at the receiving point B, after the searching of the target is successful, flying to the target receiver, and then unloading the goods from the unmanned aerial vehicle by the receiver, and completing signing. Finally, the unmanned aerial vehicle completes the task and returns to warehouse a.
It will be understood that the application has been described in terms of several embodiments, and that various changes and equivalents may be made to these features and embodiments by those skilled in the art without departing from the spirit and scope of the application. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the application without departing from the essential scope thereof. Therefore, it is intended that the application not be limited to the particular embodiment disclosed, but that the application will include all embodiments falling within the scope of the appended claims.