RU2846175C1 - Method and apparatus for emulating a signal of a satellite navigation receiver with neural network regularization of data fusion - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to systems for autonomous emulation of the output signal of a global navigation satellite system (GNSS) receiver in a positioning system for an unmanned aerial vehicle (UAV). Essence of invention consists in the following. Calculation of the UAV motion parameters occurs after the video camera images stream processing, the inertial system sensors readings, altimeter based on a system of linear algebraic equations (SLAE), taking into account the quality of reference points on video stream frames and regularization coefficients determined on each frame by a neural network trained on a set of experimental flight UAV trajectory parameters. Generation of the emulated signal in the GNSS receiver signal format takes place continuously using a microcontroller which is integrated with the single-board computer. GNSS receiver signal generated autonomously by the device is transmitted to the input of the UAV flight controller. Starting coordinates are set upon receiving a valid GNSS signal or selected using a UAV controller.

EFFECT: high accuracy of determining navigation parameters.

2 cl, 8 dwg

Description

Field of technology

The claimed method and device relate to navigation systems, namely to systems for autonomous emulation of the output signal of a global navigation satellite system (GNSS) receiver in an unmanned aerial vehicle (UAV) positioning system under conditions of complete or periodic absence of a GNSS signal, its substitution, suppression or erroneousness.

State of the art

Currently, the main means of determining the location of UAVs in space is the global satellite navigation system (GNSS), however, in cases of complete or periodic absence, substitution, suppression, erroneous GNSS signal, the aircraft becomes disoriented in space and loses control, which leads to the loss of the UAV or its breakdown. The chips of inertial systems used in UAVs are microelectromechanical devices with a low accuracy class, which leads to rapid accumulation of errors and the impossibility of continuing the flight without satellite navigation.

There is a well-known review article [1] that outlines the range of tasks and research directions that arise when using the optical flow of a video camera and various sensors to measure and subsequently evaluate the speed and orientation of the apparatus.

At the same time, the topic of joint use of the obtained estimates and data coming from the video camera and other onboard equipment is not fully disclosed, the topic of using neural networks in the article is touched upon only in matters of comparing pre-prepared images of objects with known coordinates and images coming from the video camera during the flight. The article does not provide a description of a generalized method for integrating data from various navigation devices as a set of operations for developing a device and achieving the result of increasing the accuracy of navigation.

The Intel Tracking Camera T265/T261 [2] is known to have the ability to track an observed object based on the fusion of data from visual and inertial sensors. The video camera and tracking module are a computer vision solution that outputs 6DoF data to the main system to provide full immersion, navigation, and mapping. The T265/T261 uses input data from two fisheye video cameras (OV9282) and an inertial system unit (BMI055), as well as the capabilities of the Movidius MA215x ASIC processor to provide 6DoF positioning for the main system.

The disadvantage of the device is the impossibility of its use on board the UAV due to the lack of binding of the local coordinate system to geodetic coordinates, the lack of an interface for interaction with the UAV flight controller. The small stereoscopic base of the device cannot be used to create an optical flow with the required accuracy characteristics.

A navigation combined optical system is known (RU 2694786 C1, 16.07.2019), consisting of an optical-electronic system, a contour extraction and improvement unit, a combination unit, a complex image generator, a virtual terrain model, an extended-angle generator, a correlation-extreme processor, a navigation correction calculator, and a strapdown inertial navigation system. The technical result is an increase in the accuracy of determining navigation parameters, a reduction in processing time due to the algorithms used, the correction of navigation parameters in real time, the formation of a synthesized image of the terrain, which is combined with images received from the optical-electronic system and displayed on the aircraft indicators. All together, it allows having reliable information about the underlying surface in various weather conditions.

The disadvantages of the method are the use of a computationally expensive method of combining a synthesized image of the terrain with transformed data from an optical-electronic system, the need to isolate and improve the contours of the underlying surface. The system does not have a flight controller and is a device designed to inform the crew of aircraft, and is not suitable for use on UAVs.

A radio navigation device (US 6901331 B1, 31.05.2005) is known that emulates GPS. In one embodiment, the radio navigation device that emulates GPS receives an identifier associated with a conventional radio navigation device. The present radio navigation device that emulates GPS then retrieves latitude and longitude information corresponding to the received conventional radio navigation information from a database. A satellite positioning information system generates position information for an aircraft on which the present radio navigation device that emulates GPS is located. The present radio navigation device that emulates GPS then generates navigation information for the aircraft using the received latitude and longitude information and satellite-based aircraft position information. The present radio navigation device that emulates GPS then presents the navigation information in a manner that simulates the presentation of navigation information generated by a conventional radio navigation device. The entire emulation process is transparent to the user of the real GPS emulating radio navigation device.

The disadvantage of this method is the lack of use of optical information from the video camera, as well as the dependence on the radio transmitter of navigation signals and possible failures of the radio navigation device simulating the GPS signal under conditions of electronic suppression.

A method for determining the actual dimensions of SLAM data and measuring the location is known (JP 2019074532 A, 16.05.2019), in which a camera and an inertial system unit are installed in a structure in which their relative position is fixed, acceleration is measured by video recording using the camera and the inertial system unit in motion, the captured video is exported as a still image for each frame, the captured image is analyzed by SfM (Structure from motion), the relative position of the camera is determined to obtain the camera acceleration by second-order differentiation and to obtain the camera acceleration, and the scaling factor is calculated by synchronizing the camera acceleration and the acceleration information obtained from the inertial system unit and by multiplying the relative position of the camera by the scaling factor.

The disadvantages of the method are the use of the computationally expensive SLAM method in the mapping part, the lack of compensation for errors and inaccuracies of the inertial system sensors, and the lack of reference to geodetic coordinates. In addition, second-order differentiation applied to noisy data, such as data from a video camera and inertial sensors, leads to unreliable results of calculating the motion parameters [9].

An integrated navigation method for post-processing for surveying and mapping an oil and gas pipeline route is known (CN 102636165 A, 15.08.2012), which provides an integrated navigation method for post-processing for surveying and mapping an oil and gas pipeline route. The method of integrated navigation post-processing provided by the invention includes the following steps: performing integrated navigation and combined inertial navigation with/without taking into account the location at a point without changing the GPS by means of inertial navigation with a reference, without taking into account the location and GPS signals at a fixed point; at a GPS modification point, performing inertial navigation with a time reference to the combined GPS; compensating for the error of the inertial navigation system with a reference, the error of the orientation angle and the error of the speedometer scaling factor using the Kalman filter; at the same time, using a non-instantaneous characteristic of the system data processing to repeatedly estimate and compensate for various elements of the system errors until the system error becomes less than a set threshold value. Thus, the positioning accuracy of the system is effectively improved and the actual technical requirements can be met.

The disadvantages of the system are the lack of autonomy, the need for subsequent processing for shooting and mapping. The use of the Kalman filter requires setting up a large number of unknown parameters (elements of the noise covariance matrices) and measurements. The Kalman filter has numerical instability up to a complete loss of functionality when receiving an input signal with emissions, as a result of which there is no possibility of automatically replacing the GNSS receiver signal data with the emulated signal data.

An intelligent technical vision system of an unmanned aerial vehicle is known for solving navigation problems, constructing a three-dimensional map of the surrounding space and obstacles, and autonomous patrolling (RU 195749 Ш, 05.02.2020), designed for automatic control of unmanned aerial vehicles based on the analysis of data obtained from various sources and in conditions of unavailability of signals from global navigation satellite systems. The intelligent system has the functions of a navigation system, a technical vision system, a system for monitoring the surrounding space, a decision-making system, and a semi-automatic and manual control system of the BILA. The intelligent system consists of a software and hardware module comprising an on-board computer based on a single-board computer with an expansion board based on a 32-bit controller, an inertial measuring unit that allows determining the position of the unmanned aerial vehicle in space and includes a gyroscope, accelerometer, barometer, on-board video camera, RGB-D camera, flight controller, radio signal receiver, rotating laser rangefinder, stationary laser rangefinder, interfaces for connecting motion control devices, interfaces for connecting external sensors.

The disadvantage of the system is the use of infrared depth sensors with a range limitation, as a result of which the solution is applicable only in closed spaces. The system relies on three-dimensional maps of the surrounding space, the construction of which requires additional computing resources, the system does not have the ability to automatically replace the GNSS receiver signal data with emulated signal data.

The closest in technical essence to the claimed invention is a method for correcting a strapdown inertial navigation system of a short-range unmanned aerial vehicle using an intelligent geospatial information system (RU 2722599 C1, 02.06.2020), related to a method for positioning an unmanned aerial vehicle (UAV) in autonomous mode. For this purpose, the current coordinates are continuously determined by the information and measuring devices of the low-precision strapdown inertial navigation system of the UAV. The current position is periodically corrected using satellite navigation (SNS) signals. The SNS data are checked for reliability. In automatic mode, areas or objects of observation are recorded based on information from the optical-electronic system. Correction amendments are generated for the current coordinates of the UAV. At the pre-launch preparation stage, the period of formation of correction areas of the strapdown inertial system data is calculated. During the flight, processing, geospatial referencing is performed and images are saved on board the UAV. Increased accuracy of UAV positioning in autonomous mode is ensured.

The disadvantage of this method is its pre-operational preparation of data correction areas or objects with previously known geospatial characteristics and their computationally expensive recognition on board the UAV.

A common additional drawback of the devices and methods cited as analogues is the lack of completeness of the descriptions of the proposed actions and methods leading to the creation of the claimed device and obtaining the result.

The essence of the technical solution

Current navigation parameters of the UAV movement are generated by the GNSS receiver signal emulation device, which is an autonomous navigation device and connected to the UAV flight controller. The movement parameters are calculated based on the integration and neural network regularization of data obtained from the video camera, inertial system sensors, barometer, laser rangefinder or radio altimeter, and the earth's surface altitude database installed in the device. This technical solution is aimed at increasing the autonomy of various types of UAVs in the event of GNSS signal loss and eliminating the shortcomings known from the state of the art.

The technical task is to create a low-cost autonomous device for determining the location of a UAV, ensuring the operability of a typical UAV flight controller in conditions of complete or periodic absence of a GNSS signal, its substitution or suppression with the ability to generate an output signal from a GNSS receiver with coordinates as close as possible to real ones.

The technical result of the claimed invention consists in:

- increasing the accuracy of determining navigation parameters when using an inertial system that has low accuracy and is prone to error accumulation;

- achieving continuous operation of the UAV flight controller navigation system;

- increasing the integrity of the data of the UAV flight controller navigation system;

- reducing the likelihood of emergency situations associated with the absence of a GNSS receiver signal;

- creation of a GNSS signal emulation device that is autonomous from the UAV navigation system;

- achieving universality of the GNSS signal emulation device for a wide range of UAV flight controllers using standard protocols for interaction with GNSS receivers;

- low computational and weight-dimensional load of the device. The technical result obtained by implementing the invention,

achieved by using all the features of the method in which:

- continuously receive a video stream of images from a matrix video camera;

- automatically control the exposure and light sensitivity of the video camera;

- capture frames with a time stamp from a video stream of images and place these frames in a queue;

- extract frames from the queue;

- detect an array of reference points in each frame;

- save the reference and subsequent frames with an array of reference points and a timestamp;

- adaptation of reference points is carried out, which includes setting the optimal length of connections between the corresponding reference points, assessing the quality of reference points, which calculates the error in detecting reference points, and selecting stable connections between the corresponding points;

- form a flow of reference points for calculating motion parameters, including frame time stamps, coordinates of corresponding reference points on all frames of the video stream in the local image coordinate system;

- calculate the video camera velocity vector based on the flow of reference points;

- receive data from the altimeter and from the inertial system (INS) sensors, which include a gyroscope, accelerometer, and compass;

- calculate heights above ground level using a barometer and a database of terrain heights above sea level or a laser rangefinder;

- capture and save all readings from the INS sensors and altimeter to calculate regularization coefficients;

- pre-calculate the basic regularization coefficients for the optimal solution of the system of linear algebraic equations (SLAE) for calculating the speed, angular velocity, acceleration and altitude above ground level, while using the available motion parameters from the database of experimental flight trajectories;

- form a training data set for training the neural network forecast module based on the existing database of flight trajectory parameters, including sets of values of angular velocity, acceleration, altitude, time interval between frames, length of connections between reference points, number of reference points for each trajectory;

- train the Perceptron artificial neural network [7] using a training data set.

- the basic regularization coefficients, the current generated parameters of the video stream, the readings of the inertial system sensors and the altimeter are fed to the input of the neural network;

- at the output of the neural network, the regularization weight coefficients λ _i , λ _a , λ _ω , λ _h are dynamically obtained, predicted by the neural network forecast module for solving the SLAE;

- calculate the velocity vector of the video camera in the local coordinate system using a unified system of linear algebraic equations (SLAE) with neural network regularization coefficients to calculate the speed, angular velocity, acceleration and height above ground level;

- filter and integrate the data of the video camera velocity vector in the local coordinate system and the data of the orientation quaternion received from the inertial system sensors;

- obtain the speed vector of the video camera and the location coordinates in the global coordinate system with the count from the coordinates of the starting point or the last point using reliable GNSS coordinates;

- generate GNSS protocol packets in the GNSS receiver output signal emulation mode in standard specifications, such as uavcan, msp, NMEA;

- generated packets with an emulated GNSS signal are sent to the UAV flight controller;

- in the "GNSS signal present" mode of the method, data is captured from the GNSS signal receiver; the presence and validity of the GNSS receiver signal is determined by the dilution of precision (DOP) value [8]; if the GNSS receiver signal is present and valid, it is sent to the device's computer to set the starting coordinates of the route or the initial coordinates during the route and, using a multiplexer and a generation unit, the GNSS signal is transmitted to the UAV flight controller;

- in the “no GNSS signal” mode, the starting coordinates are obtained from the flight task located in the UAV flight controller and emulate the GNSS signal;

- perform a smooth correction of the current location over time upon receiving a reliable GNSS signal without interrupting the emulation of an autonomous GNSS signal by gradually reducing the difference between the reliable coordinates received from the GNSS receiver and the coordinates generated by the device;

- provide support for widely used interfaces uavcan, msp, NMEA, mavlink for interaction with the UAV flight controller.

The implementation of the invention can be illustrated with the help of drawings:

Fig. 1 - GNSS signal emulation diagram;

Fig. 2 - Shifting the array of reference points in the frame;

Fig. 3 - Matrix structure of the system of linear algebraic equations (SLAE) for emulating the output signals of a GNSS receiver;

Fig. 4 - Functional diagram of the UAV with an installed hardware and software module (HSM);

Fig. 5 - General view of the implemented example of an on-board software and hardware module;

Fig. 6 - Installation of the PAK on the UAV;

Fig. 7 - Maximum absolute error of the emulated GNSS signal, path length 2000 m;

Fig. 8 - Example of a test flight trajectory. Implementation of the invention

The method of emulating a GNSS signal includes pre-installation of software in the device's computer. The software operates on the basis of a multi-threaded model, and the entire processing process is divided into separate interacting functional blocks (Fig. 1).

Description of the GNSS receiver signal emulation circuit (Fig. 1):

1) Auto exposure module

The auto exposure module automatically determines the exposure and light sensitivity for every 10th frame (the frequency value depends on the type of video camera used), while the video camera is pre-calibrated based on the possible pixel brightness values and the upper limit of pixel brightness; calibration is necessary for automatic control of the video camera's light sensitivity.

2) Block for generating a flow of reference points

A video stream frame coming from a video camera is extracted from the queue;

in each frame, anchor points are detected using the Harris corner detector [3], [4] (a corner detection operator that is commonly used in computer vision algorithms to extract corners and detect image features);

the reference frame with reference points and a time stamp is calculated and saved;

The reference points on the current frame corresponding to the reference points of the reference frame are detected using the sparse optical flow calculated by the Lucas-Kanade method [5]. The initial reference points of the array P ₁ , P ₂ , …P _i have coordinates:

where i=1, 2, 3, …N, i is the point number, N is the number of points.

the optimal length of connections between the corresponding reference points L (the value of the video camera offset on the video camera detector matrix) is set; if L is less than the minimum connection length L _min , this means that the video camera is not moving, or the distance traveled by the UAV is insignificant, or the maximum time between connections is exceeded (timeout is the time determined by the speed report frequency in the GNSS emulator - 10 Hz multiplied by 2, equal to 0.05 s, during which time the speed is calculated 2 times); the optimal connection length L in pixels of the video camera detector matrix should not be too short, but not too long in time. L _min depends on the video camera resolution, is selected experimentally and is a constant value, as an example, L _min = 10 pixels is set;

a uniform number of found reference points is established throughout the entire image by dividing the image into equal areas, in each of which an equal number of points is determined, the total number of points in the image is limited by the capabilities of the computer and is 200 points;

stable connections are selected between the corresponding reference points of two adjacent frames, provided that after the selection at least 10 connections remain; a connection is considered stable if its lifetime is greater than or equal to the time of passage of two consecutive frames;

an assessment of the quality of the reference points is carried out, in which the error in detecting the reference points is calculated, taken into account by the regularization coefficient λ _i , which is inversely proportional to the root-mean-square error of the solution of the system of equations for the array of reference points of the optical flow (1);

From the data on the coordinates of the reference points on the frames, an input data stream is compiled, including the time stamps of the frames.

The displacement of the array of reference points in the frame is shown in Fig. 3. The first frame shows the initial points with coordinates x _i , y _i , the second frame shows the position of these same points in space after a time interval of Δt, and frame K shows the change in the coordinates of these same points after a time interval of Δt×(K-1).

3) Block for calculating the optical part of the SLAU matrix

From the coordinates of the corresponding reference points in the local coordinate system (LCS) of the video camera and the time stamps of the frames of the input data set, the velocity vector of the video camera is calculated in the local coordinate system along the axes For this purpose, a system of equations for an array of reference points (1) is used in general form, where the unknown quantities are ν _x , ν _y , ν _z , ω _x , ω _y , ω _z .

Where - the speeds of arrays (projections) of points in the i-th connection of the corresponding reference points (units of measurement on the video camera detector matrix are meters per second),

x _i , y _i - coordinates of the point image relative to the center of the frame,

β=1/f, ƒ - focal length from the video camera detector matrix to the optical center (units of measurement on the video camera detector matrix, meters per second),

X _i , Y _i , Z _i - coordinates of a point in 3D space,

ν _x , ν _y , ν _z - video camera speeds,

ω _x , ω _y , ω _z - angular velocities,

in the expression, the coordinate matrices are indicated in square brackets, and the velocity vectors are indicated in round brackets,

The velocity vector of the reference points on the frame in the coordinate system of the video camera detector matrix in m/s is represented as:

,

the displacement value L of the video camera in the local coordinate system is calculated by multiplying the calculated vector and the time interval Δt obtained from the frame timestamps,

the velocity vector is calculated in the global coordinate system (WCS), based on the generated quaternion of the current orientation of the UAV, and the velocity vector of the calculated local coordinates in this case, reliable compass readings are taken into account, the use of an unknown azimuth is taken into account (in the absence of an azimuth when the UAV is moving, an azimuth with a direction to the north equal to 0° is taken), and corrections for the drift (error) of the gyroscope are taken into account;

the rates of change of the values of the coordinates of points along the X, Y axes are calculated in meters on the matrix of the video camera detector from the ratio of the length of the connections between the corresponding points and the time interval between them ΔT=t _n -t _(n-1) (change in time between the previous and current position of the point);

Z-axis velocities are calculated from barometer or laser rangefinder (radio altimeter) readings per unit of time. The barometer or laser rangefinder readings are selected based on the maximum distance determined by the laser rangefinder.

4) Inertial system unit (INS) with compass and altimeter

The block processes data from the INS sensors (gyroscope, accelerometer, compass) and altimeter; the orientation quaternion is calculated;

all readings from the INS and altimeter sensors are captured and saved:

- from the gyroscope about the current angular velocity;

- from the accelerometer about the current acceleration;

- from the compass about azimuth;

- from the altimeter (barometer and laser rangefinder) data is obtained on the altitude above sea level, the altitude above the ground level, while in this case a database of the altitudes of the Earth's surface above sea level previously installed on the UAV is used, the database of the altitudes of the Earth's surface above sea level is used if the terrain has sharp changes in altitude above sea level, for example, in mountainous areas).

5) Block for calculating basic regularization components

Regularization coefficients are introduced to take into account the information about motion obtained from sensors in the SLAE.

The coefficients of equations (1) (velocities of reference points for the current frame of the video stream) and the basic regularization coefficients for the equations of accelerations, angular velocities, and heights are calculated.

At the first stage, the basic coefficients in the flight trajectory database are sorted and significant boundaries are determined within which the variation of the coefficients makes sense.

At the second stage, the coefficients are refined using the gradient descent method. The initial approximation is set to values that are in the middle between the values of the significant boundaries obtained at the first stage. The optimization criterion is the following errors or their combination: the average absolute error in determining the UAV position when compared with positioning using the GNSS signal, the average error in determining the UAV position relative to the path traveled, the average error in determining the UAV speed taking into account the direction of movement, the average error in determining the absolute speed. Optimization is performed simultaneously for all coefficients. The optimization process ends when a local extremum is reached, at which the set of coefficients gives a minimum error when modeling based on flight trajectory data.

6) Neural network forecast module

Regularization of SLAE equations is carried out using the simplest artificial neural network Perceptron [7].

Based on the basic regularization coefficients, the neural network calculates the predicted regularization coefficients:

λ _i (λ ₁ , λ ₂ , …, λ _n ) are the coefficients of equations (2) (the velocities of reference points for the current frame of the video stream), which are inversely proportional to the mean square residual of the solution of the system of equations for the array of reference points of the optical flow (1),

λ _a is the coefficient of the equation for the data obtained from the accelerometer, which is directly proportional to the base coefficient and inversely proportional to the root-mean-square residual of the solution of equation (3),

λ _ω is the coefficient of the equation for the data obtained from the gyroscope, which is directly proportional to the base coefficient and inversely proportional to the root-mean-square residual of the solution of equation (4),

λ _h is the coefficient of the equation for the data obtained from the altimeter, which is directly proportional to the base coefficient and inversely proportional to the root-mean-square error of the solution of equation (5).

The input data for the trained neural network are:

basic values of regularization coefficients,

values of the number of points in the optical flow,

values of uniformity of distribution of density of reference points on the image,

values of flight altitude above ground level.

The values predicted by the neural network are the values of the regularization weighting coefficients λ _i , λ _a , λ _ω , λ _h , at which the difference between the predicted velocity values and the corresponding velocity values obtained from a reliable GNSS signal is minimal.

The neural network dynamically calculates new regularization coefficients on each frame of the optical flow. Based on the coefficients, the optimal source for obtaining the current UAV coordinates is automatically selected, which minimizes the expected error. One of the following navigation modes is possible:

- coordinates calculated using readings from a video camera, INS sensors and an altimeter in the “no GNSS” mode are used;

- coordinates calculated using the readings of a video camera in the “no GNSS” mode are used;

- coordinates calculated using the readings of the INS sensors and the altimeter are used (the “no GNSS” mode);

- coordinates received from the GNSS receiver are used (GNSS mode).

For example, data extraction from a video camera is only possible under acceptable visibility conditions; in the absence of visibility, the λ _i coefficients will be equal to zero and emulation of the GNSS receiver signal can only occur using INS sensors, an altimeter, or a reliable GNSS receiver signal.

The regularization coefficients take into account the combined systematic error of the accelerometer, gyroscope and altimeter, calculated by comparing the parameters of flight trajectories using a GNSS signal and flight trajectories without a GNSS signal;

The correction of the velocities of the reference points for the current frame of the video stream of the optical part of the SLAE matrix occurs by multiplying the equations (1) by the corresponding regularization coefficient λ _i , as shown in equality (2).

The error in estimating acceleration calculated using data from the accelerometer is taken into account by multiplying equation (3) by the regularization coefficient λ _a .

The left-hand side of equality (3) contains an estimate of the velocity increment vector in local coordinates over time Δt (from the penultimate frame to the current one), obtained on the basis of inertial sensor data. The estimate of the velocity increment in local coordinates, obtained from the solution of the SLAE, is on the right-hand side of the equality.

where ν _x , ν _y , ν _z are the speeds of movement of the video camera in the local coordinate system along the axes X, Y, Z,

ν' _x , ν' _y , ν' _z - the speeds of movement of the video camera in the local coordinate system along the X, Y, Z axes determined on the previous processed frame,

a _x , a _z , a _z - accelerations along the X, Y, Z axes in the local coordinate system,

ω _x , ω _y , ω _z - angular velocities along the X, Y, Z axes in the local coordinate system,

R ^wcs2lcs is a 3×3 matrix of transition from the global coordinate system to the local coordinate system.

The error in estimating the angular velocities obtained from the gyroscope is taken into account by multiplying equation (4) by the regularization coefficient 2. _m .

The values of angular velocities ω _x , ω _y , ω _z in the horizontal plane are corrected by the regularization coefficient λ _ω of the gyroscope. The vector of angular velocities along the axes X, Y, Z obtained from the gyroscope must be equal to the vector of angular velocities in the local coordinate system from the solution of SLAE (4).

where ω' _x , ω' _y , ω' _z are the angular velocities along the X, Y, Z axes in the local coordinate system obtained from the gyroscope.

The error in altitude obtained from the altimeter (barometer or laser rangefinder) is taken into account by multiplying equation (5) by the regularization coefficient.

Where - the rate of change of altitude obtained from a barometer or laser rangefinder,

R ^lcs2wcs is a 3×3 matrix of transition from the local coordinate system to the global coordinate system.

The training set for the neural network is formed on the basis of the existing database of experimental flight trajectory parameters under various flight conditions. The training set includes the values of: angular velocity, acceleration, altitude, time interval between frames, length of connections between reference points, number of reference points. The process of creating a training set is carried out by simulating flights with variation of regularization coefficients in specified ranges (the variation range is selected experimentally), while for each section of the trajectory, the optimal values of regularization coefficients are selected using the least squares method.

7) SLAE solution block

The matrix structure of the SLAE for calculating the UAV motion velocity vector (Fig. 3) is a set of interconnected linear inhomogeneous algebraic equations with respect to the coefficients of the linear and angular velocity vectors, UAV acceleration, and altitude above ground level. The equations of the set are multiplied by the regularization coefficients. The result of the multiplication is an overdetermined SLAE, the approximate solution of which using the iterative least squares method [6] allows us to estimate the UAV velocity vector at the time of receiving the current frame. The multiplication is performed row by row. In Fig. 3, the area designated by footnote 1 is a set of equations for calculating the optical part of the SLAE matrix, which determines the rates of change of the coordinates of reference points on the video stream frame, area 2 is the equations for calculating acceleration, area 3 is the equations for calculating angular velocity, area 4 is the equation for calculating the change in altitude, and empty cells denote zero elements of the submatrices.

8) Global speed filtering and integration block

Smooth out the outliers obtained from the SLAE solution block for the velocity vector values and orientation quaternions obtained from the ANN using the confidence interval method with a reliability of 95%. The output of the block is the velocity vector and the coordinates of the video camera in the global coordinate system.

At the starting point of the flight path O _WCS is selected from the GNSS receiver signal or the known coordinates of the starting point of the flight path are entered from the flight controller. The coordinates of the video camera in the global coordinate system calculated based on the last reliable position of the UAV obtained from the GNSS receiver and the calculated change in coordinates over a period of time in this case, compass readings are taken into account;

They estimate the coordinates of the video camera in the global coordinate system calculated based on video camera data, taking into account altimeter data.

The velocity vector of the video camera in the global coordinate system is calculated as the product of the transposed matrix and velocity vectors

The transformation of data from the global coordinate system to the local coordinate system and the inverse transformation is carried out using 3×3 transition matrices:

R ^wcs2lcs - matrix of transition from the global coordinate system to the local coordinate system,

where R ^wcs2lcs ×S _wcs =S _lcs ,

S _WCS , S _LCS - any 3D vector, velocity or acceleration column in the global and local coordinate systems, respectively.

R ^lcs2wcs - the inverse matrix of the transition from the local coordinate system to the global coordinate system is represented similarly to R ^wcs2lcs .

The rotation matrix is calculated using a well-known formula for a given quaternion.

9) GNSS protocol packet generation block

The block receives data containing the values of the velocity vector and coordinates in the global coordinate system, the data comes from the global velocity filtering and integration block or from the GNSS receiver via the GNSS protocol validation and parsing block, provided that the GNSS signal is reliable. The block ensures a gradual reduction in the difference between the reliable coordinates received from the GNSS receiver and the coordinates generated by the device, while the coordinate correction occurs seamlessly and without interrupting the GNSS signal emulation process on the device. The block ensures structuring and generation of messages in the format of well-known interaction protocols uavcan, msp, NMEA.

10) GNSS protocol validation and parsing block

The unit determines the presence or absence of a GNSS output signal based on the dilution of precision (DOP) [8] value obtained from the GNSS signal.

The block deletes messages with missing coordinates, with exceeding the set thresholds of speed error values, time deviation, altitude range, geographic latitude and longitude, or messages with other errors. The thresholds of error values are set during operation. If the GNSS signal is reliable, the block switches the device to the state of using the GNSS receiver output signal using the multiplexer. If the GNSS signal is absent or unreliable, the emulation mode is in effect.

11) Flight controller interface unit

The block provides support for standard interfaces for interaction with a wide range of UAV flight controllers: uavcan, msp, NMEA, mavlink.

As shown in Fig. 5, the UAV navigation system includes a GNSS receiver 20, a flight controller 18, a receiver of radio signals for controlling the device 19, and a software and hardware module for autonomous emulation of a GNSS signal 9, made with:

- a video camera 17 connected to the calculator of module 10, detecting reference points in a frame of a video image, periodically selected from a formed queue, and determining reference points on the current frame corresponding to the reference points of the frame selected from the formed queue, and the amount of displacement of the reference points in the compared frames;

- sensors: gyroscope 13, barometer 16, compass 15 and accelerometer 14 connected via microcontroller 22 to module calculator 10.

Thus, the computer 10 receives data on the current angular velocity from the gyroscope 13, data on the altitude from the barometer 16 or the laser rangefinder 21, data on the azimuth from the compass 15, and data for correcting the value of the speed and displacement of the unmanned aerial vehicle in the global coordinate system from the accelerometer 14. The sensors and the microcontroller 22 are mounted on the expansion board 11. In addition, the computer 10 uses the terrain map in its memory, containing data on the altitudes above sea level of the surface of the terrain sections over which the UAV is located.

The GNSS receiver 20 is connected to the hardware and software module of the autonomous emulation 9, which selects data for transmission to the flight controller 18 between the data of the GNSS receiver 20 and the data of the autonomous emulation module 9, the flight controller 18 transmits to the hardware and software module 9 the values of the coordinates of the route start point or the data of the UAV movement parameters received by the inertial system of the flight controller.

The relationship of all elements of the claimed device, taking into account the UAV itself and its standard equipment, is shown in Fig. 4, an example of the physical implementation of the GNSS signal emulation device is shown in Fig. 5, where the following are designated:

5 - UAV;

6 - storage battery (AKB);

7 - four cruise electric motors;

8 - four motor drivers (speed controllers for controlling the rotation frequency of each electric motor);

9 - hardware and software module (HSM);

10 - computer (single-board computer Raspberry PI 4);

11 - expansion board;

12 - inertial system (INS);

13 - gyroscope;

14 - accelerometer;

15 - compass;

16 - barometer;

17 - video camera;

18 - UAV flight controller;

19 - receiver of radio signals for controlling the device;

20 - GNSS radio signal receiver;

21 - laser rangefinder;

22 - microcontroller.

As an example of implementation, a working software and hardware module was manufactured, mounted on a UAV (Fig. 6) in the gap of the output signal line of the GNSS receiver with a connection to the UAV flight controller and with a connection to the on-board power supply (Fig. 5).

The solution of the set goals is achieved due to the following design: The hardware and software module is mounted on board the UAV and consists of a calculator based on a single-board computer Raspberry PI 4 Model B, 2 GB RAM and an expansion board with a 32-bit microcontroller STM32f405RG for the implementation of the possibility of processing signals in real time. The module receives signals from sensors and processes them using the calculator, and also receives and transmits these signals to the UAV flight controller. The code of the software installed on the single-board computer is developed in the C++ programming language. The inertial system in the form of a chip of the microelectromechanical system LSM6DSO (manufacturer STMicroelectronics), allows to determine the position of the UAV in space, is hardware located on the expansion board and includes:

- a gyroscope that determines the angular velocity around its own axes X, Y, Z;

- an accelerometer that determines the magnitude of acceleration along the X, Y, Z axes;

- a compass that determines the angles between its own axes X, Y, Z and the lines of force of the Earth's magnetic field;

The following devices are installed in the module:

- matrix video camera with a resolution of 1280×800 pixels - designed to receive a video stream and transmit it to the device’s computer;

- a barometer that determines atmospheric pressure and altitude above sea level.

- a stationary laser rangefinder designed to determine the height above the earth's surface and transmit this information to the computer.

The exchange of signals between the computer, the computer expansion board and the sensors is carried out using a standard serial CAN bus.

The GNSS signal emulation testing was performed on a working sample of the device by comparing the coordinates of the UAV flight paths received from the GNSS receiver and the coordinates generated using the emulation method at altitudes of the specified circular path from 0 m to 200 m, the path length up to 2000 m, while the maximum absolute error in meters is no more than 5% of the flight path length. As an example, a typical test result of measuring the error of the flight path deviation from the GNSS coordinates is given (Fig. 7), where the dotted straight line shows the maximum error, the solid straight line shows the average cumulative error.

An example of a test flight trajectory relative to the X and Y coordinates in meters is shown in Fig. 8, where the solid line denotes the trajectory with real GNSS coordinates and the dotted line denotes the trajectory with emulated coordinates. Using the device's inertial system without a video camera and with no GNSS signal results in an error exceeding the length of the flight trajectory.

The weight of the integrated UAV expansion board with a hardware and software module based on a single-board computer, including a video camera, inertial system sensors, and a body is no more than 170 g and has dimensions of 240×70×40 mm.

In another embodiment of the invention, current movement parameters are continuously determined based on data from an infrared matrix video camera;

In another embodiment of the invention, information about orientation, angular velocities and accelerations is obtained from sensors of the inertial system of the UAV flight controller.

List of references

1. Stepanyan K.V. et al. Optical flow in problems of navigation and control of unmanned autonomous vehicles // Information processes. - 2023. - Vol. 23. - No. 4. - P. 526-544.

2. Tracking Camera T265 / T261 Datasheet URL: https://www.intelrealsense.com/wp-content/uploads/2019/09/Intel_RealSense_Tracking_Camera_Datasheet_Rev004_release.pdf (accessed: 02.02.2025).

3. Chris Harris and Mike Stephens (1988). "A Combined Corner and Edge Detector". Alvey Vision Conference. 15.

4. Harris corner detector URL: https://docs.opencv.org/2.4/doc/tutorials/features2d/trackingmotion/harris_detector/harris_detector.html accessed: 02.02.2025).

5. V.D. Lucas and T. Kanade (1981), An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, pages 121-130.

6. Nguyen H.D., McLachlan G.J. Iteratively-Reweighted Least-Squares Fitting of Support Vector Machines: A Majorization-Minimization Algorithm Approach // arXiv preprint arXiv: 1705.04651. - 2017.

7. Rosenblatt F. The Perceptron: A probabilistic model for information storage and organization in the brain // Psychological Review. 1958. Vol. 65. P. 386-408.

8. Antonovich K.M. 8.2.3. Accuracy loss coefficients // Use of satellite radio navigation systems in geodesy. - Moscow, 2006. - V. 2. - P. 12.

9. Zhmud V.A. et al. Study of the causes of noise during multiple differentiation of a signal by a digital method // Automation and software engineering. 2017. No. 2 (20). - P. 80-89.

Claims (2)

1. A method for emulating the output signal of a GNSS receiver, in which data are received from a matrix video camera; current motion parameters are continuously determined based on data from the matrix video camera in the local coordinate system of the video camera; data are received from inertial system sensors and an altimeter; current motion parameters are continuously determined based on data from the inertial system sensors and an altimeter; the GNSS data are checked for reliability, characterized in that a neural network module for predicting regularization coefficients is pre-trained using a generated training data set based on an existing database of flight trajectory parameters with reliably known GNSS coordinates and UAV velocities, including values of angular velocity, acceleration, altitude, time interval between frames, length of connections between corresponding reference points, and number of reference points; implement the integration of optical information from the video camera, accelerometer, gyroscope and altimeter together in a single SLAE solution block, simultaneously replenishing the deficit of optical information and reducing the influence of systematic errors of the accelerometer and gyroscope; implement adaptive dynamic control of the process of integration of information from the video camera, accelerometer, gyroscope and altimeter by neural network forecasting of optimal regularization coefficients for all SLAE equations; implement the decision to change the reference frame in the video camera matrix to the next frame based on the conditional criterion of sufficient displacement to exclude the accumulation of error due to insufficient observed displacement velocity of reference points arising at a low ratio of the UAV altitude to the speed relative to the ground; generate an autonomous signal similar in structure to the output signal of the GNSS receiver and replacing it in the event of the absence of or substitution of the satellite signal; implement a smooth correction over time of the current location upon receipt of a reliable GNSS signal. 2. A device for emulating the output signals of a GNSS receiver, comprising a device calculator based on a single-board computer with an expansion board based on a microcontroller, an inertial system unit located on the expansion board and including a gyroscope, an accelerometer, a compass, a matrix video camera, a barometer, a stationary laser rangefinder, characterized in that it is mounted in a break in the circuit of the GNSS receiver output signal with a connection to the flight controller of the UAV, and contains an implementation of a neural network in the RAM and processor of the calculator for regularizing the process of data integration.