CN117974809B - Airborne area array camera temporal and spatial calibration method, device, equipment and storage medium - Google Patents

Abstract

The invention discloses a space-time calibration method, a device, equipment and a storage medium for an airborne area array camera, wherein the method comprises the following steps: performing coverage imaging in a control field with preset dimensions and known gravity acceleration direction to obtain SIFT feature control points; acquiring calibration data, wherein the calibration data comprises multi-frame area array images and IMU data; acquiring pose information of a camera when each frame of area array image is imaged by using a SIFT feature control field; performing pre-integration processing on IMU data among the frame area array images to obtain state descriptions of rotation, speed and displacement among the adjacent area array images; and calibrating the angle element, time deviation and line element between the camera and the IMU based on the pose information and the state description of rotation, speed and displacement. The invention has the property of controlling the direction and the scale of the gravity acceleration of the field to be known, reduces the parameters to be estimated in the space-time calibration process, and improves the calibration precision.

Description

Airborne area array camera temporal and spatial calibration method, device, equipment and storage medium

Technical Field

The present application relates to the field of remote sensing technology, and in particular to a method, device, equipment and storage medium for time-space calibration of an airborne array camera.

Background Art

In airborne remote sensing, the fusion of array camera + IMU + GNSS data is an important method to achieve 3D reconstruction. The accuracy of 3D reconstruction depends largely on the relative spatial and temporal parameter accuracy between the camera and IMU. Spatial parameters are the bridge for state conversion between the camera reference system and the IMU reference system, while time offset is used to align different sensor data streams. In order to process sensor measurements, each camera image and IMU measurement result is attached with a timestamp, which is taken from the sensor itself or the operating system of the computer receiving the data. Due to unsynchronized clocks, transmission delays, sensor responses, and operating system overhead, there is always a delay between the actual sampling moment and the attached timestamp. Due to the different delays of each sensor, the measurement streams from the camera and IMU are usually inconsistent. If the spatial and temporal parameters are not considered or the calibration is incorrect, the 3D reconstruction performance will be seriously affected. Therefore, it is necessary to study the spatiotemporal calibration method of cameras and IMUs in airborne remote sensing equipment.

Some commercial software or scholars choose to use the external orientation elements between the camera and IMU as parameters to be estimated, and estimate the parameters in the 3D reconstruction adjustment process, or ignore the external orientation elements and directly perform 3D reconstruction. Obviously, redundant estimated parameters or inaccurate external orientation elements may affect the accuracy of 3D reconstruction. Flight calibration collects flight data from a calibration field with known ground landmarks, and then adjusts the attitude and track data (acquired by tightly coupling GNSS data and IMU data) with the camera data to achieve spatiotemporal calibration. This method not only requires more manpower and material costs, but also the errors in the attitude and track data fusion process may affect the calibration accuracy.

Among the existing ground calibration schemes, most are used to complete the spatiotemporal calibration of the VIO system online. They do not use a control field or use a simple control field (such as the ChArUco calibration plate). The scheme that does not use a control field requires estimation of the scale and direction of gravity acceleration. The clear imaging distance of an aerial camera is generally greater than 30 meters, and a simple calibration plate cannot be used for calibration tasks at this distance. In addition, a simple calibration plate requires the known magnitude of gravity acceleration and an estimation of its direction. These factors greatly limit the accuracy of spatiotemporal calibration, resulting in unsatisfactory spatiotemporal calibration results between the camera and IMU.

Summary of the invention

In view of this, the present application provides a method, device, equipment and storage medium for spatiotemporal calibration of an airborne area array camera to solve the problem of low spatiotemporal calibration accuracy between existing cameras and IMUs.

In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a method for spatiotemporal calibration of an airborne array camera, which includes: performing coverage imaging in a pre-constructed control field, and performing three-dimensional reconstruction based on the first SIFT feature extracted from the image to obtain SIFT feature control points, the scale of the control field is preset and the direction of gravity acceleration is known in advance; based on the control field, motion imaging is performed in combination with a preset method, and calibration data is collected, the calibration data includes multiple frames of array images and IMU data; extracting the second SIFT feature from each frame of the array image and matching it with the SIFT feature control point, and matching the second SIFT features between different frames of array images, and obtaining the camera's posture information when each frame of the array image is imaged according to the matching results; pre-integrating the IMU data between each frame of the array image to obtain a state description of the rotation, velocity and displacement of the array image; calibrating the angular element and time deviation between the camera and the IMU based on the state description of the posture information and the rotation, and calibrating the line element between the camera and the IMU based on the state description of the posture information, velocity and displacement.

In order to solve the above technical problems, another technical solution adopted by the present application is: to provide an airborne area array camera spatiotemporal calibration device, which includes: a three-dimensional reconstruction module, which is used to perform coverage imaging in a pre-constructed control field, and perform three-dimensional reconstruction based on the first SIFT feature extracted from the image to obtain SIFT feature control points, the scale of the control field is preset and the direction of gravity acceleration is known in advance; an acquisition module, which is used to perform motion imaging based on the control field in combination with a preset method, and to acquire calibration data, the calibration data including multiple frames of area array images and IMU data; a posture estimation module, which is used to extract the second SIFT feature from each frame of the area array image and match it with the SIFT feature control point, and match the second SIFT features between different frames of area array images, and obtain the camera's posture information when each frame of the area array image is imaged according to the matching results;

The pre-integration processing module is used to perform pre-integration processing on the IMU data between each frame of the array image to obtain the state description of the rotation, speed and displacement between adjacent array images;

The calibration module is used to calibrate the angular elements and time deviations between the camera and the IMU based on the state description of the posture information and rotation, and to calibrate the line elements between the camera and the IMU based on the state description of the posture information, velocity and displacement.

In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer device, the computer device comprising a processor and a memory coupled to the processor, the memory storing program instructions, and when the program instructions are executed by the processor, the processor executes the steps of the airborne area array camera spatiotemporal calibration method as described in any one of the above items.

In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage medium storing program instructions capable of implementing any of the above-mentioned airborne area array camera spatiotemporal calibration methods.

The beneficial effects of the present application are as follows: the airborne area array camera spatiotemporal calibration method of the present application obtains SIFT feature control points by performing overlay imaging in a control field with a preset scale and a known direction of gravity acceleration, and then collects calibration data in combination with the control field. The calibration data includes multiple frames of area array images and IMU data, and then extracts the camera's posture information when each frame of area array image is imaged from each frame of area array image, and then pre-integrates the IMU data between each frame of area array image to obtain a state description of the rotation, speed and displacement of the IMU. Finally, based on the posture information and the state description of rotation, speed and displacement, the angular elements, time deviations and line elements between the camera and the IMU are calibrated. Based on the pre-constructed large-scale SIFT feature control field, which has the characteristics of a known direction and scale of gravity acceleration, the clear imaging quality of the aerial image can be guaranteed. In the calibration process, it is no longer necessary to estimate the direction of gravity acceleration, thereby reducing the amount of parameters that need to be estimated in the calibration process, thereby improving the accuracy of the calibration.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of a method for temporal and spatial calibration of an airborne area array camera according to an embodiment of the present invention;

2 is a schematic diagram of a sparse point cloud of a three-dimensional reconstruction of a SIFT feature control field in a method for spatiotemporal calibration of an airborne area array camera according to an embodiment of the present invention;

3 is a schematic diagram of the misalignment of camera and IMU timestamps in the airborne area array camera spatiotemporal calibration method according to an embodiment of the present invention;

4 is a schematic diagram of functional modules of an airborne area array camera spatiotemporal calibration device according to an embodiment of the present invention;

5 is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

FIG. 6 is a schematic diagram of the structure of a storage medium according to an embodiment of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of this application are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

FIG1 is a schematic flow chart of a method for time-space calibration of an airborne area array camera according to an embodiment of the present invention. It should be noted that the method of the present invention is not limited to the process sequence shown in FIG1 if substantially the same results are achieved. As shown in FIG1 , the method for time-space calibration of an airborne area array camera includes the following steps:

Step S101: Perform coverage imaging in a pre-constructed control field, and perform three-dimensional reconstruction based on the first SIFT feature extracted from the image to obtain SIFT feature control points. The scale of the control field is preset and the direction of gravity acceleration is known in advance.

It should be noted that the control field is a guarantee system for realizing camera positioning and orientation. To achieve this goal, it should have the following characteristics: to ensure imaging clarity, the imaging distance (i.e., the average distance from the camera imaging position of the control field to the control field) should generally be greater than 30 meters; the ratio of the control field depth to the imaging distance should be greater than 1/30; the ratio of the control field range to the mainstream camera imaging range should be greater than 2; the control field needs to be equipped with control point marks, which can be accurately measured by the total station and accurately obtained on the imaging image; ensure that at least 8 control marks can be imaged when the mainstream camera is imaging, and are relatively evenly distributed on the image. After completing the control field layout, a high-precision total station is set up at an appropriate location, and the coordinates of all control points in the control field are accurately measured after leveling. Obviously, the scale of the obtained coordinates is 1 and the direction of gravity is known.

This embodiment uses a three-dimensional reconstruction technology (Structure from motion, referred to as SFM) to achieve SIFT feature control field construction. Specifically, step S101 specifically includes the following steps:

1. Use a high-resolution fixed-focus camera to cover and image the control field to obtain an image of the control field.

Specifically, when performing coverage imaging, first, three imaging sites are selected with a site spacing greater than 5 meters so that the control field can be clearly imaged; then, at one of the imaging sites, a handheld camera uses the simulated flight path method to image the control field to ensure that the image overlap within and between flight strips is greater than 50%, and all images can cover the entire calibration field. The process is repeated to complete imaging at the three imaging sites.

2. Associate the coordinates of the control points in the control field obtained in advance with the coordinates of the imaging points at the corresponding imaging locations of the control points on the control field image.

3. Based on the relationship between the control point coordinates and the imaging point coordinates, the initial exterior orientation elements of the camera are obtained using the initial internal parameters of the camera.

Specifically, the relationship between the coordinates of the control points and the coordinates of the imaging points is used to perform a direct linear transformation (such as the Kneip-P3P method) based on the initial intrinsic parameters of the camera to obtain the initial exterior orientation elements of the camera (i.e., the position and posture of the camera in the calibration field).

4. Extract the first SIFT feature from each control field image.

Specifically, the first SIFT feature and its corresponding descriptor are extracted from the control field image.

5. Match the first SIFT features between different control field images, and retain the control field image pairs whose number of associated feature points after matching meets the preset number threshold.

Specifically, the descriptor is used to achieve the first SIFT feature matching of different control field images, and the basic matrix constructed using the initial exterior orientation elements is used to filter out incorrect matches, retaining the control field image pairs whose number of associated feature points after matching meets a preset number threshold, and the preset number threshold is preferably set to 8.

6. Based on the initial exterior orientation elements, the initial internal parameters of the camera and the associated feature points successfully matched in the control field image, three-dimensional reconstruction is performed to obtain SIFT feature control points.

Specifically, triangulation is first performed, that is, the initial 3D coordinates are estimated based on the initial exterior orientation elements and the initial camera internal parameters for the successfully matched associated feature points. Then, bundle adjustment without camera internal parameters and bundle adjustment with camera internal parameters are performed respectively; bundle adjustment without camera internal parameters, that is, the initial exterior orientation elements and initial 3D coordinates are optimized by optimization methods, control point constraints and SIFT feature association constraints; bundle adjustment with camera internal parameters, that is, the camera internal parameters, initial exterior orientation elements and initial 3D coordinates are optimized by optimization methods, control point constraints and SIFT feature association constraints. Secondly, the quality assessment of aerial triangulation is performed, that is, whether there are abnormal associated feature points (such as points with too small triangulation angles, points with large projection errors) and points with large punctum errors in the adjustment results; if not, the data is considered qualified, otherwise it needs to be adjusted and bundle adjustment without camera internal parameters and bundle adjustment with camera internal parameters are performed again. Finally, the SIFT feature control points are output. If there are no abnormalities in the aerial triangulation quality assessment, the SIFT 3D reconstructed point coordinates and their descriptors are exported as SIFT feature control points. The typical reconstruction result is shown in Figure 2.

It should be noted that, theoretically, Pixel coordinates of (control points or SIFT feature reconstruction points) on the image plane It can be expressed as:

in, The midpoint of the control field In the image space coordinate system, , The rotation and translation from the camera coordinate system to the calibration field coordinate system; is the camera focal length, is the camera principal point. The residual formula between a point in the calibration field and the imaging pixel point can be constructed by formulas (1) to (2), thereby realizing the bundle adjustment in the above steps and obtaining the precise coordinate value of each SIFT feature control point in the control field coordinate system.

Step S102: Based on the control field, motion imaging is performed in combination with a preset method, and calibration data is collected. The calibration data includes multiple frames of array images and IMU data.

Specifically, after a large-scale control field is pre-built, motion imaging is performed based on the control field in combination with a preset method, and calibration data is collected. The calibration data includes multi-frame array images and IMU data.

It should be noted that the existing ground calibration scheme is mainly used for visual-inertialodometry (VIO) exterior orientation elements and IMU initialization calibration, and is suitable for high-frequency trigger cameras (above 10Hz). Therefore, there is an assumption that the IMU angular velocity deviation and acceleration deviation remain unchanged at different times during the calibration. This is inconsistent with the performance of commonly used aerial cameras (generally with a trigger frequency of less than 1Hz), and the relevant assumptions cannot be used. Therefore, this embodiment completes the acquisition of camera and IMU synchronous calibration data through "static-dynamic-static", which not only ensures that the velocity is zero at the initial and end times, but also allows the IMU angular velocity deviation at the initial and end times to be directly read through the IMU data, thereby using these constraints to increase the reliability of the estimation results. Specifically, step S102 specifically includes the following steps:

1. When the payload device is oriented toward the sky and the imaging direction is toward the control field, the payload device is placed in an initial static state, and at least a first preset number of array images and IMU data are collected simultaneously.

Specifically, this embodiment can collect data by holding the device or placing it on a moving platform according to the weight of the device. When collecting data, the payload heading is toward the zenith direction and the imaging direction is toward the control field. Then, the payload device is placed in an initial static state, and at least a first preset number of area array images are collected and IMU data is collected at the same time. The first preset number is preferably set to at least 2 groups.

2. Control the payload device to rotate clockwise and counterclockwise around its own X, Y, and Z axes respectively, and collect a second preset number of array images and IMU data while rotating around each axis.

Specifically, the payload device is controlled to rotate clockwise and counterclockwise around its own X, Y, and Z axes, with a rotation angle greater than 20 degrees, and each axis collects at least a second preset number of array images and IMU data, and the second preset number is preferably set to 10 groups.

3. Place the payload device in a static state, collect at least a first preset number of array images and simultaneously collect IMU data.

This embodiment completes the space-time calibration by constructing pre-constraint conditions (the IMU angular velocity deviation at the initial and end times is known, and the velocity at the initial and end times is 0) through the "static-dynamic-static" data acquisition mode, and the direction of the gravity acceleration is known but the magnitude is unknown. These more accurate assumptions will improve the calibration accuracy and make the final calibration result more accurate.

Step S103: extracting the second SIFT features from each frame of the array image and matching them with the SIFT feature control points, matching the second SIFT features between different frames of array images, and obtaining the camera posture information when each frame of the array image is imaged according to the matching results.

Specifically, after obtaining the area array images, the second SIFT features are extracted for all the area array images, and then the second SIFT features are matched with the SIFT feature control points, and the second SIFT features of the area array images of different frames are matched. Finally, the matching results are combined to obtain the camera posture information when each frame of the area array image is imaged.

Further, step S103 specifically includes the following steps:

1. Extract the second SIFT feature from each frame of the array image.

2. Match the second SIFT feature with the SIFT feature control point to confirm the external orientation elements of the image.

Specifically, the second SIFT feature is matched with the SIFT feature control point, and the image exterior orientation elements are confirmed through kneip-P3P+RANSAC (Random Sample Consensus).

3. Match the second SIFT features between different frame array images, and use the initial exterior orientation elements to filter the matched feature points to obtain the matched feature points.

4. The stationary area array images in the initial stationary state and the final stationary state are confirmed according to the pixel displacement of the matching feature points between the area array images.

Specifically, the area array images in the initial static state and the final static state are judged according to the pixel displacement of the matching feature points between the area array images (i.e., the parallax of each matching feature point), and some static area array images are eliminated, so that two static area array images are retained in the initial stage and the final stage respectively.

5. Based on the bundle adjustment, the pose of the non-stationary array image is estimated, and finally the pose information of the camera when each frame of the array image is imaged is obtained.

Specifically, the image pose data is used as the input data for spatiotemporal calibration, where The pose of the frame image is: ,in, , For the The rotation parameters and position parameters of the camera in the control field coordinate system when the frame image is formed.

Step S104: pre-integrate the IMU data between each frame array image to obtain a state description of the rotation, speed and displacement of the IMU.

It should be noted that, in principle, given the initial posture and velocity, the IMU posture can be measured by the IMU to obtain the angular velocity And acceleration value Integral acquisition, but its measured value not only contains Gaussian white noise and wandering deviation, but also the acceleration measurement value should be subtracted from the gravity acceleration Therefore, the IMU measurement model can be described as:

in, and is the angular velocity and linear acceleration value of IMU in the global frame, is the rotation matrix of IMU in the world coordinate system, and are Gaussian white noise of angular velocity and wandering deviation, and are Gaussian white noise and wandering deviation of linear acceleration value respectively. In this embodiment, pre-integration is used to pre-process the IMU data between camera frames. Suppose the IMU data corresponding to the adjacent frames of a continuous camera is To At this moment, the rotation, velocity and displacement of the IMU pre-integration can be described as:

in, Indicates To The rotation difference at time, Indicates To The speed difference of time, Indicates To The displacement difference at time, Indicates To The moments between moments, express To The time difference of the moment.

If the angular velocity deviation in the pre-integrated quantity is taken into account and acceleration deviation , then the IMU integral acquisition status is:

in: , For the The rotation matrix and translation vector of the IMU in the world coordinate system at this moment; , For the The rotation matrix and translation vector of the IMU in the world coordinate system at this moment; and For the and The speed of the IMU in the world coordinate system at this moment; for To The time difference between moments; is the rotation pre-integrated component pair The derivative of , The speed pre-integration , The derivative of , is the displacement pre-integrated quantity , The derivative of , the above derivative can be obtained by the pre-integration process.

Furthermore, as shown in Figure 3, due to clock asynchrony, transmission delay, sensor response and operating system overhead, these delays cause misalignment between the sensor sampling time and the measurement time. The camera and IMU data are sampled at the same time, and the actual timestamps of the camera and IMU measurements are:

in, , are the actual timestamps of the camera and IMU respectively, , The time delays of the camera and IMU are defined as follows: :

From this, we can obtain the following relationship between the camera and IMU posture:

in, for The camera pose at the sampling moment, for The position and posture of the IMU at the sampling time, is the position of the camera in the IMU coordinate system. Obviously, the so-called spatiotemporal calibration is the time offset , relative transformation matrix between camera and IMU (including corner elements , Line elements ) is used to estimate the process. Therefore, step S104 specifically includes the following steps:

1. Build time offset to build the spatiotemporal synchronization state between the camera and IMU.

Specifically, the present invention will At this moment, the rotation matrix and translation vector of the IMU in the world coordinate system are linearly interpolated as follows:

in, for The angular velocity of the IMU in the body coordinate system at this moment, for The linear velocity of the IMU in the world coordinate system at this moment. From this, the spatiotemporal synchronization state of the camera and IMU can be described as:

2. Based on the spatiotemporal synchronization state, the IMU data between each frame of the array image is pre-integrated to obtain the state description of the rotation, speed and displacement of the array image.

Specifically, the rotation, speed, and displacement states of the array image are described as follows:

; (13)

; (14)

; (15)

in, Indicates The pre-integrated quantity of the camera's rotation in the control field coordinate system when the frame image is formed. Indicates The pre-integrated quantity of the camera's rotation in the control field coordinate system when the frame image is formed. represents the corner element, represents the exponential function, express The angular velocity of the IMU in the body coordinate system at this moment, express The angular velocity of the IMU in the body coordinate system at this moment, represents the time offset between the camera and the IMU, Indicates To The rotation difference at time, Represents the rotation pre-integrated quantity pair The derivative of represents the angular velocity deviation, Indicates The velocity pre-integrated quantity of IMU in the world coordinate system at the moment, Indicates The velocity pre-integrated quantity of IMU in the world coordinate system at the moment, represents the acceleration due to gravity, Indicates To The time difference between the moments Indicates To The speed difference of time, Indicates the speed pre-integrated quantity The derivative of Indicates the speed pre-integrated quantity The derivative of represents the acceleration deviation, Indicates The pre-integrated displacement of the camera in the control field coordinate system when the frame image is formed. Indicates The pre-integrated displacement of the camera in the control field coordinate system when the frame image is formed. Indicates To The displacement difference at time, Represents the displacement pre-integrated quantity The derivative of Represents the displacement pre-integrated quantity The derivative of Represents a line element.

Step S105: Calibrate the angular elements and time deviations between the camera and the IMU based on the state description of the posture information and rotation, and calibrate the line elements between the camera and the IMU based on the state description of the posture information, velocity and displacement.

Specifically, after obtaining the posture information, the angular elements and time deviations between the camera and IMU can be calibrated using the posture information and the predicted rotation component, and the line elements between the camera and IMU can be calibrated using the posture information, velocity pre-integration, and displacement pre-integration.

Furthermore, the step of calibrating the angular element and time deviation between the camera and the IMU based on the state description of the pose information and the rotation includes:

1. Construct the residual formula based on the rotation state description.

Specifically, since the time difference between the continuous images to be processed in this embodiment is large, it is assumed that the IMU angular velocity deviation is different at different times. In this embodiment, the residual formula is constructed by formula (13) to achieve calibration. The rotation matrix in formula (13) can be expressed based on quaternion as:

in, Quaternion representation of the camera's corner elements in the IMU coordinate system; for The conjugate quaternion of The inverse transform of .

Quaternions It can be defined as:

in, are the real part of the quaternion, Direction imaginary part, Direction imaginary part, Direction imaginary part, .

Then the quaternion left multiplication operation and right multiplication Can be defined as:

in To get the vector Antisymmetric array, Represents a 3×3 unit diagonal matrix.

2. Initialize the angular velocity deviation at different times.

Specifically, the data collected in this embodiment Frame array image, The frame array images are all in a static state, so the true value of the IMU angular velocity during this period is 0. According to the IMU measurement model (see formula (3)), its measured value is the angular velocity wandering deviation With measurement noise The sum of The 5 frames of IMU angular velocity data of the adjacent second frame array image between the frame array images, The adjacent frames Take the average value of 5 frames of IMU angular velocity data of the frame array image , , as the second, The angular velocity deviation of the IMU at the moment is obtained by linear interpolation. The initial value of the angular velocity deviation of the IMU at the moment is used to pre-integrate the IMU data between image frames. It should be understood that in the existing scheme, the camera frequency used is high, so the deviations of the cameras at different moments can be considered the same. However, in the present invention, the cameras used are no longer limited to high-frequency cameras, and low-frequency cameras can also be used. However, low-frequency cameras cannot simply be considered to have the same deviations at different moments. Therefore, the present invention uses static data at the initial and end moments to initialize the gravity acceleration deviations at different moments.

3. Based on the angular velocity deviation at different times, the residual formula is used to perform angular element calibration, time deviation calibration and angular velocity deviation calibration at different times.

Specifically, for corner elements Calibrate and obtain the first The data between array images has an equation:

exist Under these conditions, the least square method is used to Estimates:

in, , The conjugate quaternion representation of the camera's corner elements in the IMU coordinate system is converted into a rotation matrix The calibration is completed.

For time deviation Calibrate and obtain the first The data between array images has an equation:

If you , then rearranging the imaginary part of the quaternion in formula (21) yields:

The time deviation is calculated using the least square method. To perform calibration:

;(twenty three)

Angular velocity deviation at different times To perform calibration:

By shifting the terms in formula (16), we can obtain The data between array images has an equation:

If:

;

but:

Using least squares, we can get The IMU angular velocity deviation correction at this moment is:

This amends the The angular velocity deviation at different moments can be calibrated.

3. Determine whether the time deviation exceeds the preset time deviation threshold.

Specifically, the preset time deviation threshold is preset.

4. If it does not exceed, confirm that the calibration of the angle element, time deviation and angular velocity deviation at different times is completed.

5. If it exceeds, the time deviation obtained by calibration is cyclically used to perform pre-integration processing on the IMU data again and then calibrate, and the time deviation in the new calibration result is compared with the preset time deviation threshold again until the time deviation is lower than the preset time deviation threshold. Further, the step of calibrating the line element between the camera and the IMU based on the state description of the posture information, speed and displacement includes:

1. Based on the calibration data of the angular element, time deviation and angular velocity deviation at different times, the IMU data is re-pre-integrated to obtain the state description of the new velocity and new displacement.

Specifically, before calibrating the line elements, the calibrated angle elements, time deviations, and angular velocity deviations at different times are used to perform pre-integration again. Formulas (14) and (15) can be transformed into:

This embodiment performs line element calibration based on this formula. In addition to this element, there is also an unknown gravitational acceleration in this embodiment. , acceleration deviation , and in this embodiment the direction of the control field gravity is known and can be described as , so only the norm of gravitational acceleration By estimating, we can get The moment formulas (27)~(28) are expressed as:

in:

.

2. Initialize the gravity acceleration deviation at different times.

Specifically, the data collected in this embodiment Frame array image, The frame array images are all in a static state, so the true value of the IMU acceleration during this period is 0. According to the IMU measurement model (see formula (3)), its measured value is the acceleration wandering deviation , measurement noise and the acceleration due to gravity, take the The average value of the 5 frames of IMU acceleration data between the adjacent second frame array images , The adjacent frames Average value of 5 frames of IMU acceleration data of the frame array image After that, we can calculate the second Gravity acceleration deviation of IMU at this moment:

in: and For the 2nd The transpose of the rotation matrix of the camera in the world coordinate system at that moment, The diagonal elements are calibrated. The first The initial value of the gravity acceleration deviation of the IMU at the moment is used to pre-integrate the IMU data between the array image frames. It should be understood that in the existing scheme, the camera frequency used is high, so the deviations of the cameras at different moments can be considered the same. However, in the present invention, the cameras used are no longer limited to high-frequency cameras, and low-frequency cameras can also be used. However, low-frequency cameras cannot simply be considered to have the same deviations at different moments. Therefore, the present invention uses static data at the initial and end moments to initialize the gravity acceleration deviations at different moments.

3. When the direction of gravity acceleration in the control field is known, the line elements, gravity acceleration, velocity and acceleration deviation are calibrated respectively in combination with the state description of gravity acceleration direction, gravity acceleration deviation at different times, new velocity and new displacement.

Specifically, for line elements , Gravitational acceleration and speed To perform calibration:

Will middle After setting the corresponding columns to 0, the least squares method is used to estimate the unknown parameters:

in, Represents the acceleration due to gravity The conjugate quaternion of Indicates speed The conjugate quaternion of , due to the particularity of data acquisition in this embodiment, its initial velocity , End Speed 0, set in the above optimization The corresponding column can be set to 0.

For acceleration deviation To perform calibration:

Will middle , , After setting the corresponding columns to 0, the least squares method is used to estimate the unknown parameters:

in, express.

3. Determine whether the gravity acceleration exceeds the preset acceleration threshold.

Specifically, the preset acceleration threshold is set in advance.

4. If it does not exceed, confirm that the calibration of line elements, gravity acceleration, velocity and acceleration deviation is completed.

5. If it exceeds, the acceleration deviation obtained by calibration is cyclically used to pre-integrate the IMU data again and then calibrate it, and the gravity acceleration in the new calibration result is compared with the preset acceleration threshold again until the gravity acceleration is lower than the preset acceleration threshold.

It is understandable that in the existing scheme, since a high-frequency camera is used for data collection, the angular velocity deviation and acceleration deviation of the IMU at different times can be considered the same. However, the present invention uses a low-frequency camera to collect data, so the angular velocity deviation and acceleration deviation of the IMU at different times cannot be directly considered the same, resulting in an increase in unknowns. In order to reduce the unknowns in the calibration process, the present invention constrains the data collection process during data collection, so that the scale and gravity direction of the SIFT feature control field are known, and then the method of solving different estimated values in steps is used to achieve calibration of all values.

The airborne area array camera spatiotemporal calibration method of the present embodiment performs coverage imaging in a control field with a preset scale and a known gravity acceleration direction to obtain SIFT feature control points, and then collects calibration data in combination with the control field. The calibration data includes multiple frames of area array images and IMU data, and then extracts the camera's posture information when each frame of the area array image is imaged from each frame of the area array image. Then, the IMU data between each frame of the area array image is pre-integrated to obtain a state description of the rotation, speed and displacement of the area array image. Finally, based on the posture information and the state description of the rotation, speed and displacement, the angular element, time deviation and line element between the camera and the IMU are calibrated. Based on the pre-constructed large-scale SIFT feature control field, which has the characteristics of known gravity acceleration direction and known scale, the clear imaging quality of the aerial image can be guaranteed. In the calibration process, it is no longer necessary to estimate the gravity acceleration direction, thereby reducing the amount of parameters that need to be estimated in the calibration process, thereby improving the calibration accuracy.

Fig. 4 is a functional module diagram of an airborne area array camera spatiotemporal calibration device according to an embodiment of the present invention. As shown in Fig. 4, the airborne area array camera spatiotemporal calibration device 20 includes a 3D reconstruction module 21, an acquisition module 22, a pose estimation module 23, a pre-integration processing module 24 and a calibration module 25.

The three-dimensional reconstruction module 21 is used to perform coverage imaging in a pre-constructed control field and perform three-dimensional reconstruction based on the first SIFT feature extracted from the image to obtain SIFT feature control points. The scale of the control field is pre-set and the direction of gravity acceleration is pre-known.

The acquisition module 22 is used to perform motion imaging based on the control field in combination with a preset method, and to acquire calibration data, the calibration data including multi-frame array images and IMU data;

The pose estimation module 23 is used to extract the second SIFT features from each frame of the array image and match them with the SIFT feature control points, and to match the second SIFT features between different frames of the array image, and to obtain the pose information of the camera when each frame of the array image is imaged according to the matching results;

The pre-integration processing module 24 is used to perform pre-integration processing on the IMU data between each frame of the array image to obtain a state description of the rotation, speed and displacement of the array image;

The calibration module 25 is used to calibrate the angular elements and time deviations between the camera and the IMU based on the state description of the posture information and the rotation, and to calibrate the line elements between the camera and the IMU based on the state description of the posture information, the velocity and the displacement.

Optionally, the three-dimensional reconstruction module 21 performs coverage imaging in a pre-constructed control field, and performs three-dimensional reconstruction based on the first SIFT feature extracted from the formed image to obtain an operation of SIFT feature control points, specifically including:

Use a high-resolution fixed-focus camera to cover and image the control field to obtain an image of the control field;

Associating the coordinates of the control points in the control field acquired in advance with the coordinates of the imaging points at the imaging locations corresponding to the control points on the control field image;

Based on the correlation between the coordinates of the control points and the coordinates of the imaging points, the initial exterior orientation elements of the camera are obtained using the initial intrinsic parameters of the camera;

Extract the first SIFT feature from each control field image respectively;

Matching the first SIFT features between different control field images, and retaining the control field image pairs whose number of associated feature points after matching meets a preset number threshold;

Based on the initial exterior orientation elements, the initial internal parameters of the camera and the successfully matched associated feature points in the control field image, three-dimensional reconstruction is performed to obtain the SIFT feature control points.

Optionally, the acquisition module 22 performs motion imaging based on the control field in combination with a preset method and acquires calibration data, specifically including:

When the payload device is oriented toward the sky and the imaging direction is toward the control field, the payload device is placed in an initial static state, and at least a first preset number of array images and IMU data are collected simultaneously;

Control the payload device to rotate clockwise and counterclockwise around its own X, Y, and Z axes respectively, and collect a second preset number of array images and IMU data simultaneously when rotating around each axis;

The payload device is placed in an end-stationary state, and at least a first preset number of array images are collected and IMU data are collected simultaneously.

Optionally, the pose estimation module 23 performs operations of extracting the second SIFT features from each frame of the array image and matching them with the SIFT feature control points, matching the second SIFT features between different frame array images, and obtaining the pose information of the camera when each frame of the array image is imaged according to the matching results, which specifically includes:

Extract the second SIFT feature from each frame of the array image;

Match the second SIFT feature with the SIFT feature control point to confirm the external orientation elements of the image;

Match the second SIFT features between different frame array images, and use the initial exterior orientation elements to filter the matched feature points to obtain the matched feature points;

Confirm the stationary area array images in the initial stationary state and the final stationary state according to the pixel displacement of the matching feature points between the area array images;

The pose of non-stationary array images is estimated based on bundle adjustment, and finally the pose information of the camera at the time of imaging each frame of the array image is obtained.

Optionally, the pre-integration processing module 24 performs pre-integration processing on the IMU data between each frame of the array image to obtain the state description of the rotation, speed and displacement of the array image, specifically including:

Build time offset to build the spatiotemporal synchronization state between the camera and IMU;

Based on the spatiotemporal synchronization state, the IMU data between each frame of the array image is pre-integrated to obtain the state description of the rotation, velocity and displacement of the array image.

Optionally, the calibration module 25 performs an operation of calibrating the angular element and time deviation between the camera and the IMU based on the state description of the posture information and the rotation, specifically including:

Construct the residual formula based on the rotation state description;

Initialize the angular velocity deviation at different times;

Based on the angular velocity deviation at different times, the residual formula is used to calibrate the angle element, time deviation and angular velocity deviation at different times.

Determine whether the time deviation exceeds a preset time deviation threshold;

If it does not exceed, it is confirmed that the calibration of the angle element, time deviation and angular velocity deviation at different times is completed;

If it exceeds, the time deviation obtained by calibration is recycled to perform pre-integration processing on the IMU data again and then calibrate it, and the time deviation in the new calibration result is compared with the preset time deviation threshold again until the time deviation is lower than the preset time deviation threshold.

Optionally, the calibration module 25 performs an operation of calibrating the line elements between the camera and the IMU based on the state description of the posture information, velocity and displacement, specifically including:

Based on the calibrated data of the angle element, time deviation and angular velocity deviation at different times, the IMU data is pre-integrated again to obtain a state description of the new velocity and new displacement;

Initialize gravity acceleration deviation at different times;

When the direction of gravity acceleration in the control field is known, the line element, gravity acceleration, velocity and acceleration deviation are calibrated respectively by combining the state description of gravity acceleration direction, gravity acceleration deviation at different times, new velocity and new displacement;

Determine whether the gravity acceleration exceeds a preset acceleration threshold;

If not, then confirm that the calibration of line elements, gravity acceleration, velocity and acceleration deviation is completed;

If it exceeds, the acceleration deviation obtained by calibration is recycled to perform pre-integration processing on the IMU data again and then calibrate it, and the gravity acceleration in the new calibration result is compared with the preset acceleration threshold again until the gravity acceleration is lower than the preset acceleration threshold.

For other details of the technical solutions for implementing each module in the airborne area array camera spatiotemporal calibration device in the above embodiment, reference may be made to the description of the airborne area array camera spatiotemporal calibration method in the above embodiment, which will not be repeated here.

It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Please refer to Fig. 4, which is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. As shown in Fig. 4, the computer device 30 includes a processor 31 and a memory 32 coupled to the processor 31, wherein the memory 32 stores program instructions, and when the program instructions are executed by the processor 31, the processor 31 executes the steps of the airborne area array camera spatiotemporal calibration method described in any of the above embodiments.

The processor 31 may also be referred to as a resource (Central Processing Unit). The processor 31 may be an integrated circuit chip having the ability to process signals. The processor 31 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

Refer to Figure 5, which is a schematic diagram of the structure of the storage medium of an embodiment of the present invention. The storage medium of the embodiment of the present invention stores program instructions 41 that can implement the above-mentioned airborne area array camera spatiotemporal calibration method, wherein the program instructions 41 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer device such as a computer, a server, a mobile phone, and a tablet.

In the several embodiments provided in the present application, it should be understood that the disclosed computer equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims (7)

1. The space-time calibration method for the airborne area array camera is characterized by comprising the following steps of:

Performing coverage imaging in a pre-constructed control field, and performing three-dimensional reconstruction based on a first SIFT feature extracted from the formed image to obtain SIFT feature control points, wherein the scale of the control field is preset and the direction of gravitational acceleration is known in advance;

Based on the control field, performing motion imaging by combining a preset mode, and acquiring calibration data, wherein the calibration data comprises a multi-frame area array image and IMU data;

Respectively extracting second SIFT features from each frame of area array image, matching the second SIFT features with the SIFT feature control points, matching the second SIFT features among different frames of area array images, and obtaining pose information of a camera when each frame of area array image is imaged according to a matching result;

Performing pre-integration processing on IMU data among the frame area array images to obtain state descriptions of rotation, speed and displacement among the adjacent area array images;

Calibrating an angle element and a time deviation between a camera and an IMU based on the pose information and the rotated state description, and calibrating a line element between the camera and the IMU based on the pose information, the speed and the displaced state description;

Performing coverage imaging in a control field constructed in advance, and performing three-dimensional reconstruction based on a first SIFT feature extracted from the formed image to obtain SIFT feature control points, wherein the method comprises the following steps:

performing coverage imaging on the control field by using a high-resolution fixed-focus camera to obtain a control field image;

Associating the control point coordinates in the control field obtained in advance with imaging point coordinates of imaging positions corresponding to the control points on the control field image;

based on the association relation between the control point coordinates and the imaging point coordinates, obtaining initial external azimuth elements of the camera by utilizing initial internal parameters of the camera;

Extracting the first SIFT features from each control field image respectively;

matching the first SIFT features among different control field images, and reserving control field image pairs with the number of the matched associated feature points meeting a preset number threshold;

Three-dimensional reconstruction is carried out on the basis of the initial external azimuth element, the initial internal reference of the camera and the associated characteristic points successfully matched in the control field image pair, so as to obtain SIFT characteristic control points;

Based on the control field, the motion imaging is performed in combination with a preset mode, and calibration data are collected, including:

under the condition that the heading of the load equipment is controlled to face the sky direction and the imaging direction is controlled to face the control field, the load equipment is placed in an initial standing state, at least a first preset number of area array images are collected, and IMU data are collected at the same time;

The load equipment is controlled to rotate clockwise and anticlockwise around the X, Y, Z shaft respectively, and a second preset number of area array images are collected and IMU data are collected simultaneously when the load equipment rotates around each shaft;

placing the load equipment in an ending and standing state, collecting at least a first preset number of area array images and simultaneously collecting IMU data;

The steps of extracting second SIFT features from each frame of area array image, matching the second SIFT features with the SIFT feature control points, matching the second SIFT features among different frames of area array images, and obtaining pose information of cameras when each frame of area array image is imaged according to a matching result include:

respectively extracting second SIFT features from each frame of area array image;

matching the second SIFT feature with the SIFT feature control point to confirm the external azimuth element of the image;

Matching the second SIFT features among the different frame area array images, and filtering the matched feature points by utilizing the initial external azimuth elements to obtain matched feature points;

Confirming a static area array image in an initial static state and an ending static state according to the pixel displacement of the matching feature points among the area array images;

And carrying out pose estimation on the non-stationary area array image based on the adjustment of the beam method, and finally obtaining pose information of the camera when each frame of area array image is imaged.

2. The space-time calibration method of an airborne area array camera according to claim 1, wherein the pre-integration processing is performed on IMU data between each frame of area array images to obtain a state description of rotation, speed and displacement between adjacent area array images, including;

constructing time offset to construct a space-time synchronization state between the camera and the IMU;

and carrying out pre-integration processing on IMU data among the frame area array images based on the space-time synchronization state to obtain state description of rotation, speed and displacement among the adjacent area array images.

3. The method for space-time calibration of an on-board area array camera according to claim 2, wherein the calibrating the angular element and the time offset between the camera and the IMU based on the pose information and the rotated state description comprises:

Constructing a residual formula based on the rotated state description;

Initializing angular speed deviations at different moments;

Based on the angular velocity deviation at different moments, respectively carrying out angle element calibration, time deviation calibration and angular velocity deviation calibration at different moments by utilizing the residual error;

judging whether the time deviation exceeds a preset time deviation threshold value or not;

If the angular element, the time deviation and the angular speed deviation at different moments are confirmed to be calibrated;

If the time deviation obtained by the calibration is exceeded, the IMU data is calibrated after the pre-integration treatment is carried out again by recycling, and the time deviation in the new calibration result is compared with the preset time deviation threshold again until the time deviation is lower than the preset time deviation threshold.

4. The method for space-time calibration of an on-board area array camera according to claim 3, wherein the calibrating line elements between the camera and the IMU based on the pose information, the velocity, and the state description of the displacement comprises:

Pre-integrating IMU data again based on the angle element, the time deviation and the calibration data of the angle speed deviation at different moments after calibration is completed, so as to obtain a new speed and a new displacement state description;

initializing gravity acceleration deviation at different moments;

Calibrating a linear element, a gravitational acceleration, a velocity and an acceleration deviation respectively in combination with the gravitational acceleration direction, the gravitational acceleration deviation at different moments, the new velocity and the state description of the new displacement under the condition that the gravitational acceleration direction of the control field is known;

Judging whether the gravity acceleration exceeds a preset acceleration threshold value or not;

If the acceleration deviation is not exceeded, confirming that calibration of the line element, the gravity acceleration, the speed and the acceleration deviation is completed;

If the acceleration deviation obtained by the calibration is exceeded, the IMU data is calibrated after the pre-integration treatment is carried out again by the cyclic utilization of the acceleration deviation obtained by the calibration, and the gravity acceleration in the new calibration result is compared with the preset acceleration threshold again until the gravity acceleration is lower than the preset acceleration threshold.

5. An airborne area array camera space-time calibration device, which is characterized in that the airborne area array camera space-time calibration method according to any one of claims 1-4 is adopted; it comprises the following steps:

the three-dimensional reconstruction module is used for performing coverage imaging in a control field which is built in advance, performing three-dimensional reconstruction based on a first SIFT feature extracted from the formed image to obtain SIFT feature control points, wherein the scale of the control field is preset and the direction of gravitational acceleration is known in advance;

The acquisition module is used for carrying out motion imaging by combining a preset mode based on the control field and acquiring calibration data, wherein the calibration data comprises multi-frame area array images and IMU data;

The pose estimation module is used for respectively extracting second SIFT features from each frame of area array image, matching the second SIFT features with the SIFT feature control points, matching the second SIFT features among different frames of area array images, and obtaining pose information of a camera when each frame of area array image is imaged according to a matching result;

The pre-integration processing module is used for carrying out pre-integration processing on IMU data among the frame area array images to obtain state descriptions of rotation, speed and displacement among the adjacent area array images;

And the calibration module is used for calibrating the angle element and the time deviation between the camera and the IMU based on the pose information and the rotating state description, and calibrating the line element between the camera and the IMU based on the pose information, the speed and the displacement state description.

6. A computer device comprising a processor, a memory coupled to the processor, the memory having stored therein program instructions that, when executed by the processor, cause the processor to perform the steps of the space-time calibration method of an onboard area array camera of any of claims 1-4.

7. A storage medium storing program instructions for implementing the space-time calibration method of an onboard area array camera according to any one of claims 1 to 4.