CN112700502B - Binocular camera system and binocular camera space calibration method - Google Patents

Abstract

The invention proposes a binocular camera system and a binocular camera space calibration method, which solve the problems of poor imaging quality and no depth information in the prior art. The binocular camera system is sequentially cascaded with a camera module, a correction module, a calibration module and a storage module; the camera module includes a parallel event camera and a common CMOS camera. The spatial calibration method includes: the camera module obtains the address-event data stream, image and shooting time; the correction module performs distortion and epipolar correction on the data stream and image; the calibration module calibrates the corrected data stream and image spatial position; the storage module corrects The final data streams and images and their corresponding relations are stored in the persistent memory as a set of homography matrices. The present invention does not use a beam splitter, thus avoiding the resulting poor imaging. The invention has parallax, and can restore spatial depth information through the parallax. It is used for high frame rate and high resolution data reconstruction, identification and tracking of high-speed moving targets and other fields.

Description

Binocular camera system and binocular camera space calibration method

technical field

The invention belongs to the technical field of computer vision, and relates to the structure and spatial calibration of a binocular camera, in particular to a binocular camera system and a binocular camera space calibration method, which are used to reconstruct image data with high temporal resolution and high spatial resolution. Improve the accuracy of high-speed moving target recognition and tracking, and can also be used for spatial calibration of binocular camera systems including event cameras and ordinary CMOS cameras.

Background technique

The binocular camera system can obtain the distance between the object and the camera through the parallax of the two cameras, and is widely used in the fields of robotics and SLAM. However, there are many defects in the binocular camera system using a common CMOS camera. When the relative motion between the binocular camera system and the target is too fast, serious image blurring will occur, making the binocular camera unable to work well.

The event camera is a new type of camera, each pixel of which is individually sensitive to light, and the event camera will only output data when the light intensity felt by a certain pixel changes, otherwise the event camera will not output data. There is also no blurring when the relative motion between the event camera and the target is fast. Moreover, the event camera has the advantages of large dynamic range and small data volume. The data format output by an event camera is also different from that of an ordinary CMOS camera. The data output by an event camera is an address-event data stream.

The binocular camera system composed of an event camera and a common CMOS camera can combine the advantages of the two cameras to obtain image data with high dynamic range, high frame rate and high spatial resolution, and use it to calculate spatial depth information.

The application publication number is CN108038888A, titled "Hybrid Camera System and Its Spatial Calibration Method and Device", which discloses a hybrid camera system and its spatial calibration method. This camera and its spatial calibration method use an event camera, a common CMOS camera and a beam splitter A binocular camera system is formed to achieve spatial calibration by minimizing the spatial projection error. Its shortcoming is that the spatial position between the event camera and the ordinary CMOS camera is roughly calibrated by using a spectroscopic sheet, resulting in that each of the two cameras can only receive insufficient 50% of the light reduces the image quality; the use of spectroscopic sheets results in no parallax between the two cameras, and the loss of spatial depth information.

Contents of the invention

The purpose of the present invention is to overcome the above-mentioned defects in the prior art, and propose a binocular camera system and binocular camera space calibration method that can identify and track high-speed moving targets, retain parallax information and ensure imaging quality.

The present invention is a binocular camera system, which is characterized in that a camera module, a correction module, a calibration module and a storage module are cascaded in sequence to form a binocular camera system; an event camera and an ordinary CMOS camera with the same lens orientation and connected in parallel constitute a camera modules, each of which is described as follows:

Described camera module comprises event camera and CMOS camera, is used for obtaining address-event data flow, image and image shooting time, wherein, event camera is used for obtaining address-event data flow, and CMOS camera is used for obtaining image and image Shooting time, the event camera and the CMOS camera output data at the same time and the output images of the event camera and the CMOS camera have the same size, the lens axes of the event camera and the CMOS camera are parallel, and the sensor planes of the event camera and the CMOS camera are on the same plane;

The correction module is used to respectively perform distortion correction and epipolar correction on the address-event data stream output by the event camera and the image output by the CMOS camera, and output the corrected address-event data stream and the corrected image, wherein, The calibration module stores the camera matrix and distortion matrix of the pre-calibrated event camera, stores the camera matrix and distortion matrix of the CMOS camera, and also stores the rotation matrix and translation matrix between the pre-calibrated event camera and the CMOS camera;

The calibration module is used to calibrate the spatial position of the address-event data stream and image after distortion correction and epipolar line correction, obtain and output a homography matrix representing the corresponding relationship between event coordinates and image pixel coordinates in the address-event data stream ;

The storage module includes a persistent memory for persistent storage of the calibration results, the calibration results include the corrected address-event data stream, the corrected image and the address-event data stream in the event coordinates and image pixel coordinates should matrix.

The present invention is also a spatial calibration method for a binocular camera, which is implemented on a binocular camera system, and is characterized in that it includes the following steps:

(1) The camera module obtains the address-event data stream, images and the shooting time of each image:

The event camera in the camera module acquires the address-event data stream E={e _i |0<i≤N ₁ } and outputs it, while the CMOS camera in the camera module acquires N ₂ images and the moment S={ s _r |0<r≤N ₂ } and output, where e _i represents the i-th event, e _i = (xi _, y _i , g _i , t _i ), and xi and _{y i respectively} represent the triggering of e _i The abscissa and ordinate of the position pixel, g _i represents the gray value of e _i , g _i >0, t _i represents the triggering time of e _i , s _r = (I _r , tp _r ), I _r represents the image sequence For the rth image in S, tp _r represents the moment when image I _r is taken, N ₁ >0, N ₁ represents the number of events in the address-event data stream E, N ₂ >0;

(2) The correction module performs distortion correction and epipolar line correction on each event e _i and image I _r :

The correction module performs distortion correction on each event e _i through the camera matrix and distortion matrix of the event camera to obtain the distortion-corrected event e _1,i , and performs distortion correction on each image I _r through the camera matrix and distortion matrix of the CMOS camera , get the distortion-corrected image I _1,r , and perform epipolar correction on the event e _1,i and image I _1,r through the rotation matrix and translation matrix between the event camera and the CMOS camera, and obtain the epipolar-corrected image event e _2,i and image I _2,r ;

(3) The calibration module performs spatial position calibration on the corrected address-event data stream and image:

For each image I _2,r , the calibration module divides the corrected address-event data stream E ₂ into multiple address-event data stream segments E _2, r according to the shooting time tp _r of the image I ₂ ,r , and accumulate the events in the address-event data flow segment E _2,r into the matrix according to the coordinates of the event, use the SIFT operator to calculate the matrix where the event is accumulated and the feature points of the image I _2,r respectively, and perform feature Point matching obtains many pairs of matching points, and calculates the homography matrix between the matrix and the image I _2,r by the matching points, and the homography matrix represents the correspondence between the event coordinates of the address-event data stream and the image pixel coordinates;

(4) The storage module stores the calibration results: the storage module stores the corrected address-event data stream, the corrected image, and the homography matrix of event coordinates and image pixel coordinates in the address-event data stream in a persistent memory. , to complete the spatial calibration of the event camera and the CMOS camera.

The present invention aims to provide a binocular camera system and perform spatial calibration on the binocular camera system without using a spectroscopic sheet.

Compared with the prior art, the present invention has the following advantages:

Improved imaging quality: Since the present invention does not use a spectroscopic sheet, the event camera and CMOS camera can receive more light, and the imaging quality is higher than that of a binocular camera system using a spectroscopic sheet.

There is a parallax between the cameras, and a depth image can be obtained: due to the parallax between the event camera and the CMOS camera in the present invention, the depth information can be obtained through the obtained address-event data stream and image of the present invention, and the dual There is no parallax between the two cameras of the objective camera system, and it is difficult to obtain a depth image.

The camera can be used to identify and track high-speed moving targets: Since the event camera in the present invention does not produce blur when shooting high-speed moving targets, the trajectory and attitude of the high-speed moving target can be obtained through the event camera. The ordinary CMOS camera in the present invention can complement the background information and color information in the scene that the event camera cannot obtain, and can make up for the defect that the event camera cannot photograph stationary objects. The invention combines an event camera and a common CMOS camera, integrates the advantages of the two cameras, and improves the accuracy of high-speed moving target recognition and trajectory tracking.

Description of drawings

FIG. 1 is a schematic diagram of the overall structure of the binocular camera system of the present invention.

Fig. 2 is a flow chart of the implementation of the binocular camera space calibration method of the present invention.

Detailed ways

Below in conjunction with accompanying drawing and specific embodiment, the present invention is described in further detail:

Example 1

Due to the limitation of the imaging principle of the existing common CMOS camera, the common CMOS camera has a smear phenomenon when shooting a high-speed moving target, which will cause the image to be blurred, and it is difficult to effectively use the image data captured by it. The event camera is a camera with a different imaging principle from the ordinary CMOS camera. The event camera simulates the imaging characteristics of the biological retina. It only images moving targets, so the data volume of the event camera is smaller, and it can shoot high-speed moving targets. Combining a CMOS camera and an event camera to form a binocular camera system can effectively use the advantages of the two cameras to obtain data with the advantages of high frame rate, high dynamic range, and high spatial resolution.

The present invention is a binocular camera system. Referring to FIG. 1 , the binocular camera system of the present invention is sequentially cascaded with a camera module, a correction module, a calibration module and a storage module to form a binocular camera system. The event camera and the ordinary CMOS camera with the same lens facing in parallel constitute a camera module, and each module is described as follows:

The camera module among the present invention comprises event camera and CMOS camera, is used for obtaining address-event data flow, image and image capture time, wherein, event camera is used for obtaining address-event data flow, and CMOS camera is used for obtaining image and The shooting time of the image, the event camera and the CMOS camera output data at the same time and the output images of the event camera and the CMOS camera have the same size, the lens axes of the event camera and the CMOS camera are parallel, and the sensor planes of the event camera and the CMOS camera are on the same plane.

The event camera used in this embodiment is a Celex-V camera, and both the event camera and the common CMOS camera use a lens with a focal length of 16 mm.

The correction module in the present invention is used to respectively perform distortion correction and epipolar line correction on the address-event data stream output by the event camera and the image output by the CMOS camera, and output the corrected address-event data stream and the corrected image, Among them, the correction module stores the camera matrix and distortion matrix of the pre-calibrated event camera, stores the camera matrix and distortion matrix of the pre-calibrated CMOS camera, and also stores the rotation matrix and translation between the pre-calibrated event camera and the CMOS camera matrix.

In this embodiment, the correction module is implemented using Nvidia TX2, and is connected to an event camera or a common CMOS camera through a USB interface.

The calibration module in the present invention is used to calibrate the spatial position of the address-event data stream and image after distortion correction and epipolar line correction, obtain and output the homography matrix of event coordinates and image pixel coordinates in the address-event data stream. The calibration module in this embodiment is implemented using Nvidia TX2.

The storage module in the present invention includes a persistent memory for persistently storing the corrected address-event data stream, the corrected image, and the homography matrix representing event coordinates and image pixel coordinates in the address-event data stream gather.

Since there is no parallax between the two cameras in the binocular camera system using the spectroscopic sheet in the prior art, it is only necessary to calibrate in advance when performing spatial calibration on the binocular camera system using the spectroscopic sheet, but due to the influence of the spectroscopic sheet, this The image quality obtained by this binocular camera system is relatively poor, and the structure of this binocular camera system is complex, requiring high assembly precision. Aiming at this situation, the present invention proposes a binocular camera system without using a spectroscopic sheet. Since the spectroscopic sheet is not used, it is not necessary to align the spectroscopic sheet with the camera lens, which simplifies the structure of the system and reduces the cost. And because no beam splitter is used, the imaging quality of the two cameras is guaranteed. There is a parallax between the two cameras in the present invention, so the data captured by the present invention does not lose spatial depth information and can be used to calculate a depth map.

Since the event camera in the present invention does not produce blur when photographing a high-speed moving target, the track and posture of the high-speed moving target can be obtained through the event camera. The ordinary CMOS camera in the present invention can complement the background information and color information in the scene that the event camera cannot obtain, and can make up for the defect that the event camera cannot photograph stationary objects. The invention combines an event camera and a common CMOS camera, integrates the advantages of the two cameras, and improves the accuracy of high-speed moving target recognition and trajectory tracking.

The binocular camera system of the present invention can be used to obtain data with high temporal resolution and high spatial resolution, and can obtain depth information, and can be applied to visual navigation of unmanned driving systems to improve the safety of unmanned driving systems.

Example 2

The overall composition of a binocular camera system is the same as in Embodiment 1. The camera matrix and distortion matrix of the pre-calibrated event camera stored in the correction module of the present invention are obtained by Zhang Zhengyou's calibration method, and the camera of the pre-calibrated CMOS camera stored in the correction module The matrix and distortion matrix are obtained by Zhang Zhengyou’s calibration method; the rotation matrix and translation matrix between the pre-calibrated event camera and the CMOS camera stored in the correction module are obtained by Bouguet’s epipolar correction method. In this embodiment, when calibrating the event camera and the common CMOS camera, the calibration board used is a checkerboard calibration board. When calibrating the event camera in this camera, a full-scale grayscale image is acquired through the Celex-V camera.

The invention performs distortion correction on each camera in the binocular camera respectively, which can reduce the distortion of the camera image and improve the image quality. The present invention does not use a beam splitter, which not only simplifies the structure of the system, but also eliminates the need for precise alignment of the camera lens and the beam splitter. The invention can align the same object point in the two cameras by correcting the epipolar line of the binocular camera. The image points of imaging are limited to the same horizontal line, which narrows the search range when matching feature points, reduces the amount of calculation, and realizes real-time calibration.

Example 3

Due to the parallax between the two cameras in the binocular camera system in the present invention, the coordinates of the same target on the image planes of the two cameras are related to the relative positions of the target and the binocular camera system. When the relative position of the binocular camera system changes, the coordinates of the target on the respective image planes of the two cameras in the binocular camera system will also change. The camera system is calibrated in real time. The spatial calibration method of the present invention utilizes the binocular camera system of the present invention to capture data in real time, and then corrects and calibrates these data in real time.

The present invention is also a binocular camera space calibration method, which is implemented on the above-mentioned binocular camera system, referring to Fig. 2, including the following steps:

(1) The camera module obtains the address-event data stream, images and the shooting time of each image:

The event camera in the camera module obtains the address-event data stream E and outputs it, E={e _i |0<i≤N ₁ }, i is the serial number of the event, and the CMOS camera in the camera module obtains N ₂ images and each The moment S={s _r |0<r≤N ₂ } of the images taken and output, where e _i represents the i-th event, e _i =(xi _, y _i , g _i , t _i ), xi _and y _i represent the abscissa and ordinate of the trigger position pixel of e _i respectively, g _i represents the gray value of e _i , g _i >0, t _i represents the triggering time of e _i , s _r =(I _r ,tp _r ), s _r includes the image and the shooting time of the image, I _r represents the rth image in the image sequence S, r is the serial number of the image, tp _r represents the time when the image I _r is shot, N ₁ >0, N ₁ represents Address - the total number of events in the event data stream E, N ₂ >0. In this embodiment, the time difference between the shooting moments of two adjacent images is less than 30 ms, so as to reduce the number of events generated by the event camera between the shooting moments of the two adjacent images.

(2) The correction module performs distortion correction and epipolar line correction on each event e _i and each image I _r respectively:

The correction module performs distortion correction on each event e _i through the camera matrix and distortion matrix of the event camera to obtain the distortion-corrected event e _1,i , and performs distortion correction on each image I _r through the camera matrix and distortion matrix of the CMOS camera , get the distortion-corrected image I _1,r , and perform epipolar correction on e _1,i and I _1,r through the rotation matrix and translation matrix between the event camera and the CMOS camera, and obtain the epipolar-corrected event e _2,i and I _2,r . In this embodiment, performing distortion correction and epipolar line correction on event e _i refers to performing distortion correction and epipolar line correction on the coordinates of event e _i , and the time t _i and gray level g _i of event e _i remain unchanged. For CMOS The camera performs distortion correction and epipolar line correction, which refers to correcting the pixel coordinates in the image I _r , and the shooting time tp _r of the image I _r remains unchanged.

(3) The calibration module performs spatial position calibration on the corrected address-event data stream and image:

For each image I _2,r , the calibration module divides the epipolar-corrected address-event data stream E ₂ into multiple address-event data stream segments E ₂ according to the shooting time tp _r of the image I _2, r _,r , and the events in the address-event data flow segment E _2,r are accumulated into the all-zero matrix M according to the coordinates of the event. The matrix M is an all-zero matrix constructed by the calibration module, which is used to calculate the address-event data flow and For the matching points of image I _2,r , use the SIFT operator to match the feature points of the matrix and image I _2,r to obtain multiple matching points, and calculate the homography between the matrix M and image I _2,r through the matching points Matrix, the homography matrix represents the correspondence between address-event data stream event coordinates and image pixel coordinates.

(4) The storage module stores the calibration result: the storage module stores the corrected address-event data stream, the corrected image, and the set H={H _r | 0<r≤N ₂ } is stored in the persistent memory to complete the spatial calibration of the event camera and the CMOS camera. In this embodiment, the data storage format is hdf5 format, and an image, the address-event data flow segment corresponding to the image, and the corresponding relationship between the pixel coordinates of the image and the corresponding address-event data flow segment event coordinates are stored in a file The homography matrix of . It is also possible to use multiple files to respectively store an image, the address-event data flow segment corresponding to the image, and a homography matrix representing the corresponding relationship between the pixel coordinates of the image and the corresponding address-event data flow segment event coordinates, and the data storage format is the same as Not limited to hdf5 format.

The invention uses the shooting time of the image to segment the address-event data stream output by the event camera, realizes the time alignment of the image and the address-event data stream, and reduces the error of spatial calibration of the binocular camera.

Example 4

Since the data output by the event camera in the present invention is an unstructured address-event data stream, it is necessary to convert the address-event data stream into structured data in order to process the address-event data stream. By matching the feature points between the address-event data stream and the image, multiple pairs of matching points are obtained, and the homography matrix between the matching points is calculated through these matching points to complete the calibration of the binocular camera system.

A binocular camera system and its binocular camera space calibration method are the same as those in Embodiment 1-3, and the calibration module described in step (3) performs spatial position calibration on the corrected address-event data stream and image, including the following steps :

(3a) Construct an all-zero matrix M=zeros(H, W), where H and W represent the total number of rows and total columns of the CMOS camera output image respectively, H≥32, W≥32, let each element m in M =0, and let i=1.

(3b) Let r=1, r represents the serial number of the normal CMOS camera image after correction.

(3c) divide the address-event data flow into address-event data flow segments according to the shooting time tp _r of the image I _2,r :

(3c1) Let the address-event data stream event set corresponding to the image I _r be E _2,r , and let E _2,r be an empty set.

(3c2) Judging whether 0<x _2, i ≤W and 0<y _{2,i ≤H} in the event e _2,i =(x 2,i ,y _2,i ,g _i ,t _i ) holds true, if so, Execute step (3c3), otherwise, set i=i+1, and execute step (3c2).

(3c3) Add the event e _2,i =(x _2,i ,y _2,i ,gi , _{t i} ) to the set E _2,r , and execute step (3c4).

(3c4) Judging whether t _i ≤ tp _r is established, if so, set i=i+1, and execute step (3c2), otherwise, obtain the address-event data flow segment E _2,r ={e ₂ , corresponding to the image I _{r ,r,b} |0<b≤B}, where, e _2,r,b ＝(x _2,r,b ,y _2,r,b ,g _2,r,b ,t _2,r,b ) , B represents the total number of events in the set E _2,r .

In step (3c), according to the shooting time tp _r of the image I _2, r and the event information of the event in the address-event data stream, the present invention finds the event generated by the event camera when the image I _2,r is shot, which is conducive to improving the understanding of Accuracy of feature point matching for image and address-event data streams.

(3d) Begin to accumulate the events in the address-event data stream segment into the matrix M, let b=1, b is the event sequence number in an address-event data stream segment E _2,r .

(3e) order Among them, g _{2, r, b} is the gray value of event e _{2, r, b} in the address-event data flow segment, is the element whose coordinates are (x _2,r,b ,y _2,r,b ) in the matrix M, and the matrix M is no longer an all-zero matrix.

(3f) Judging whether b≤B is established, if so, make b=b+1, and execute step (3e), otherwise, when b>B, complete the accumulation of address-event data flow segments, and execute step (3g) .

(3g) Start to match feature points of image I _{2, r} and matrix M, use SIFT operator to extract feature points in matrix M, and get N _{r, 3} feature points HP _r ={p _r,l |0<l ≤N _r,3 }, where p _r,l =(x _r,l ,y _r,l ,F _r,l ), l is the sequence number of the feature point p _r,l , (x _r,l ,y _{r ,l} ) represents the coordinates of feature points, F _r,l represents the feature descriptor of feature points, N _r,3 >8.

(3h) Let l=1, l is the serial number of the feature point in the feature point set of matrix M.

(3i) Start looking for feature points in image I _2,r that match feature points p _r,l , use SIFT operator to calculate the feature points of image I _2,r on the line y _r,l , and get W features Point HP _r,l '={p _r,l,k '|0<k≤W}, where p _r,l,k 'represents the kth feature of image I _2,r on line y _r,l point, p _{r, l, k} '=(x _{r, l, k} ', y _{r, l, k} ', F _{r, l, k} '), k is the line in y _{r, l} in image I _{2, r} The serial number of the feature point above, (x _{r, l, k} ', y _{r, l, k} ') indicates the coordinates of the feature point p _{r, l, k} ', and F _{r, l, k} ' indicates the feature point p _{r, l , the feature descriptor for k} '.

(3j) Let k=1, k represents the kth feature point of image I _2,r on line y _r,l .

(3k) Calculate the Euclidean distance or _{r, l,k} between F _l and F r,l,k '.

(3l) Determine whether k<W is true, if so, set k=k+1, and execute step (3k), otherwise, obtain the feature descriptor distance set O _r,l ={or _r,l,k |0<k ≤W}, and execute step (3m).

(3m) Suppose the minimum value of O _r,l is or _r,l,k' , then the feature point corresponding to p _r,l is p _r,l,k′ ′.

(3n) Determine whether l≤N _r,3 is established, if so, set l=l+1, and execute step (3i), otherwise, get the event and image I _2,r in the address-event data stream set E _2,r The set KP _r ={f _r,l =(p _r,l ,p _r,l,k′ )|0<l≤N _r,3 } of the corresponding points of the pixel, and perform step (3o).

(3o) Calculate the homography matrix H _r of the event coordinates in the address-event data stream set E _2,r and the pixel coordinates in the image I _r by the eight-point method through KP _r .

(3p) Judging whether r≤N ₂ is established, if so, let r=r+1, and execute step (3c), otherwise, for all images and address-event data stream calibration, get the event in the address-event data stream A set of homography matrices H={H _r |0<r<N ₂ } of the corresponding relationship between coordinates and image pixel coordinates.

The present invention divides the address-event data stream into segments through the shooting time of the image and the time information of the events in the address-event data stream, so as to ensure that only images with similar shooting times and address-event data streams are featured matched. When the two cameras shoot the same scene, the accuracy of spatial calibration is improved. After spatial calibration, the parallax of the event camera and the ordinary CMOS camera can be obtained to restore the spatial depth information. After spatial calibration, the data of the event camera and the common CMOS camera can be fused. The common CMOS camera can complement the background information and color information in the scene that the event camera cannot obtain, and the event camera can obtain high-time resolution data. The data fusion of the former can obtain data with the advantages of both high temporal resolution and high spatial resolution.

The present invention solves the problems of poor imaging quality and loss of spatial depth information in the prior art. The binocular camera spatial calibration method includes the following steps: the camera module obtains the address-event data stream, images and the shooting time of each image; The module performs distortion correction and epipolar correction on events and images; the calibration module performs spatial position calibration on the corrected address-event data stream and image, and the storage module stores the corrected address-event data stream, corrected image and address - A collection of homography matrices corresponding to event coordinates and image pixel coordinates in the event data stream is stored in a persistent memory.

Next, the binocular camera system and its calibration method will be combined together to further illustrate the present invention.

Example 5

A binocular camera system and a binocular camera space calibration method thereof are the same as those in Embodiments 1-4.

A binocular camera system of the present invention, referring to FIG. 1 , the binocular camera system of the present invention is sequentially cascaded with a camera module, a correction module, a calibration module and a storage module to form a binocular camera system. The event camera and the ordinary CMOS camera with the same lens facing in parallel constitute a camera module, and each module is described as follows:

The camera module of the present invention includes an event camera and a CMOS camera for obtaining address-event data flow, image and image shooting time, wherein the event camera is used for obtaining address-event data flow, and the CMOS camera is used for obtaining image and image The shooting time of the event camera and the CMOS camera output data at the same time and the output images of the event camera and the CMOS camera have the same size, the lens axes of the event camera and the CMOS camera are parallel, and the sensor planes of the event camera and the CMOS camera are on the same plane.

The correction module of the present invention is used to respectively perform distortion correction and epipolar line correction on the address-event data stream output by the event camera and the image output by the CMOS camera, and output the corrected address-event data stream and the corrected image, wherein , the correction module stores the camera matrix and distortion matrix of the pre-calibrated event camera, stores the camera matrix and distortion matrix of the CMOS camera, and also stores the rotation matrix and translation matrix between the event camera and the CMOS camera.

The calibration module of the present invention is used to calibrate the spatial position of the address-event data stream and image after distortion correction and epipolar line correction, obtain and output the homography matrix of event coordinates and image pixel coordinates in the address-event data stream.

The storage module of the present invention includes a persistent memory for persistently storing the calibration results, the calibration results including the corrected address-event data stream, the corrected image and the event coordinates and image pixel coordinates in the address-event data stream homography matrix.

On the basis of a binocular camera system of the present invention, a binocular camera space calibration method is also designed. Referring to FIG. 2, a binocular camera space calibration method of the present invention can be used for data collection, Identify and track, and obtain the parallax of the event camera and the common CMOS camera for recovering the spatial depth information.

Including the following steps:

(1) The camera module obtains the address-event data stream, images and the shooting time of each image:

The event camera in the camera module acquires the address-event data stream E={e _i |0<i≤N ₁ } and outputs it, while the CMOS camera in the camera module acquires N ₂ images and the moment S={ s _r |0<r≤N ₂ } and output, where e _i represents the i-th event, e _i = (xi _, y _i , g _i , t _i ), and xi and _{y i respectively} represent the triggering of e _i The abscissa and ordinate of the position pixel, g _i represents the gray value of e _i , g _i >0, t _i represents the triggering time of e _i , s _r = (I _r , tp _r ), I _r represents the image sequence For the rth image in S, tp _r represents the time when image I _r is taken, N ₁ >0, N ₁ represents the number of events in the address-event data stream E, N ₂ >0.

(2) The correction module performs distortion correction and epipolar line correction on each event e _i and image I _r :

The correction module performs distortion correction on each event e _i through the camera matrix and distortion matrix of the event camera to obtain the distortion-corrected event e _1,i , and performs distortion correction on each image s _r through the camera matrix and distortion matrix of the CMOS camera , get the distortion-corrected image I _1,r , and perform epipolar correction on e _1,i and I _1,r through the rotation matrix and translation matrix between the event camera and the CMOS camera, and obtain the epipolar-corrected event e _2,r and I _2,r .

(3) The calibration module performs spatial position calibration on the corrected address-event data stream and image:

(3a) Construct an all-zero matrix M=zeros(H, W), where H and W represent the total number of rows and total columns of the CMOS camera output image respectively, H≥32, W≥32, let each element m in M =0, and let i=1.

(3b) Let r=1, r represents the serial number of the normal CMOS camera image after correction.

(3c) divide the address-event data flow into address-event data flow segments according to the shooting time tp _r of the image I _2,r :

(3c1) Let the address-event data stream event set corresponding to the image I _r be E _2,r , and let E _2,r be an empty set.

(3c2) Judging whether 0<x _2, i ≤W and 0<y _{2,i ≤H} in the event e _2,i =(x 2,i ,y _2,i ,g _i ,t _i ) holds true, if so, Execute step (3c3), otherwise, set i=i+1, and execute step (3c2).

(3c3) Add the event e _2,i =(x _2,i ,y _2,i ,gi , _{t i} ) to the set E _2,r , and execute step (3c4).

(3c4) Judging whether t _i ≤ tp _r is established, if so, set i=i+1, and execute step (3c2), otherwise, obtain the address-event data flow segment E _2,r ={e ₂ , corresponding to the image I _{r ,r,b} |0<b≤B}, where, e _2,r,b ＝(x _2,r,b ,y _2,r,b ,g _2,r,b ,t _2,r,b ) , B represents the total number of events in the set E _2,r .

(3d) Begin to accumulate the events in the address-event data stream segment into the matrix M, let b=1, b is the event sequence number in an address-event data stream segment E _2,r .

(3e) order Among them, g _{2, r, b} is the gray value of event e _{2, r, b} in the address-event data flow segment, is the element whose coordinates are (x _2,r,b ,y _2,r,b ) in the matrix M, and the matrix M is no longer an all-zero matrix.

(3f) Judging whether b≤B is established, if so, make b=b+1, and execute step (3e), otherwise, when b>B, complete the accumulation of address-event data flow segments, and execute step (3g) .

(3g) Start to match feature points of image I _{2, r} and matrix M, use SIFT operator to extract feature points in matrix M, and get N _{r, 3} feature points HP _r ={p _r,l |0<l ≤N _r,3 }, where p _r,l =(x _r,l ,y _r,l ,F _r,l ), l is the sequence number of the feature point p _r,l , (x _r,l ,y _{r ,l} ) represents the coordinates of feature points, F _r,l represents the feature descriptor of feature points, N _r,3 >8.

(3h) Let l=1, l is the serial number of the feature point in the feature point set of matrix M.

(3i) Start looking for feature points in image I _2,r that match feature points p _r,l , use SIFT operator to calculate the feature points of image I _2,r on the line y _r,l , and get W features Point HP _r,l '={p _r,l,k '|0<k≤W}, where p _r,l,k 'represents the kth feature of image I _2,r on line y _r,l point, p _{r, l, k} '=(x _{r, l, k} ', y _{r, l, k} ', F _{r, l, k} '), k is the line in y _{r, l} in image I _{2, r} The serial number of the feature point above, (x _{r, l, k} ', y _{r, l, k} ') indicates the coordinates of the feature point p _{r, l, k} ', and F _{r, l, k} ' indicates the feature point p _{r, l , the feature descriptor for k} '.

(3j) Let k=1, k represents the kth feature point of image I _2,r on line y _r,l .

(3k) Calculate the Euclidean distance or _{r, l,k} between F _l and F r,l,k '.

(3l) Determine whether k<W is true, if so, set k=k+1, and execute step (3k), otherwise, obtain the feature descriptor distance set O _r,l ={or _r,l,k |0<k ≤W}, and execute step (3m).

(3m) Suppose the minimum value of O _r,l is or _r,l,k' , then the feature point corresponding to p _r,l is p _r,l,k′ ′.

(3n) Determine whether l≤N _r,3 is established, if so, set l=l+1, and execute step (3i), otherwise, get the event and image I _2,r in the address-event data stream set E _2,r The set KP _r ={f _r,l =(p _r,l ,p _r,l,k′ )|0<l≤N _r,3 } of the corresponding points of the pixel, and perform step (3o).

(3o) Calculate the homography matrix H r of the event coordinates in the address-event data stream set E _{2,r and} the pixel coordinates in the image I _r by the eight-point method through KP _r .

(3p) Judging whether r≤N ₂ is established, if so, let r=r+1, and execute step (3c), otherwise, for all images and address-event data stream calibration, get the event in the address-event data stream A set of homography matrices H={H _r |0<r<N ₂ } of the corresponding relationship between coordinates and image pixel coordinates.

In summary, a binocular camera system and a binocular camera spatial calibration method proposed by the present invention solve the problems of poor imaging quality and loss of spatial depth information existing in the prior art. The binocular camera system is sequentially cascaded with a camera module, a correction module, a calibration module and a storage module, wherein the camera module includes a parallel event camera and a common CMOS camera. The binocular camera space calibration method includes the following steps: the camera module obtains the address-event data stream, image and shooting time of each image; the correction module performs distortion correction and epipolar line correction on the event and image; the calibration module corrects the corrected address- Spatial position calibration of the event data stream and image; the storage module stores the corrected address-event data stream, the corrected image, and the homography matrix set of event coordinates and image pixel coordinates in the address-event data stream in a persistent memory . The binocular camera system of the present invention does not use a beam splitter, which avoids the problem of camera imaging quality degradation caused by the use of a beam splitter. In the present invention, there is a parallax between the two cameras, and spatial depth information can be restored through the parallax between the two cameras. The invention can reconstruct image data with high temporal resolution, high spatial resolution and high dynamic range, and is used for identification and tracking of high-speed moving targets. At the same time, the present invention can also be used to restore spatial depth information, can be used for visual navigation of unmanned driving systems, reconstructs environmental information, and enhances the safety of unmanned driving systems.

Claims (4)

1. A binocular camera system, characterized in that a camera module, a calibration module, a calibration module and a storage module are cascaded in sequence to form a binocular camera system; the camera lens is directed towards the same parallel event camera and a common CMOS camera to form a camera module , where each module is described as follows: Described camera module comprises event camera and CMOS camera, is used for obtaining address-event data flow, image and image shooting time, wherein, event camera is used for obtaining address-event data flow, and CMOS camera is used for obtaining image and image Shooting time, the event camera and the CMOS camera output data at the same time and the output images of the event camera and the CMOS camera have the same size, the lens of the event camera is parallel to the lens axis of the CMOS camera, and the sensor planes of the event camera and the CMOS camera are on the same plane; The correction module is used to respectively perform distortion correction and epipolar correction on the address-event data stream output by the event camera and the image output by the CMOS camera, and output the corrected address-event data stream and the corrected image, wherein, The correction module stores the camera matrix and distortion matrix of the pre-calibrated event camera, stores the camera matrix and distortion matrix of the pre-calibrated CMOS camera, and also stores the rotation matrix and translation matrix between the pre-calibrated event camera and the CMOS camera; The calibration module is used to calibrate the spatial position of the address-event data stream and image after distortion correction and epipolar line correction, obtain and output a homography matrix representing the corresponding relationship between event coordinates and image pixel coordinates in the address-event data stream ; The storage module includes a persistent memory for persistently storing the calibration results, the calibration results include the corrected address-event data stream, the corrected image and the correspondence between event coordinates and image pixel coordinates in the address-event data stream The homography matrix of the relation. 2. The binocular camera system according to claim 1, wherein the camera matrix and distortion matrix of the event camera calibrated in advance stored in the correction module are obtained by Zhang Zhengyou calibration method, and the CMOS calibrated in advance stored in the correction module The camera matrix and distortion matrix of the camera are obtained by Zhang Zhengyou’s calibration method; the rotation matrix and translation matrix between the pre-calibrated event camera and the CMOS camera stored in the correction module are obtained by Bouguet’s epipolar correction method. 3. A binocular camera space calibration method is implemented on the binocular camera system according to claim 1-2, characterized in that it comprises the following steps: (1) The camera module obtains the address-event data stream, images and the shooting time of each image: The event camera in the camera module acquires the address-event data stream E={e _i |0<i≤N ₁ } and outputs it, while the CMOS camera in the camera module acquires N ₂ images and the moment S={ s _r |0<r≤N ₂ } and output, where e _i represents the i-th event, e _i = (xi _, y _i , g _i , t _i ), and xi and _{y i respectively} represent the triggering of e _i The abscissa and ordinate of the position pixel, g _i represents the gray value of e _i , g _i >0, t _i represents the triggering time of e _i , s _r = (I _r , tp _r ), I _r represents the image sequence For the rth image in S, tp _r represents the moment when image I _r is taken, N ₁ >0, N ₁ represents the number of events in the address-event data stream E, N ₂ >0; (2) The correction module performs distortion correction and epipolar line correction on each event e _i and image I _r : The correction module performs distortion correction on each event e _i through the camera matrix and distortion matrix of the event camera to obtain the distortion-corrected event e _1,i , and performs distortion correction on each image I _r through the camera matrix and distortion matrix of the CMOS camera , get the distortion-corrected image I _1,r , and perform epipolar correction on the event e _1,i and image I _1,r through the rotation matrix and translation matrix between the event camera and the CMOS camera, and obtain the epipolar-corrected image event e _2,i and image I _2,r ; (3) The calibration module performs spatial position calibration on the corrected address-event data stream and image: For each image I _2,r , the calibration module divides the corrected address-event data stream E ₂ into multiple address-event data stream segments E _2, r according to the shooting time tp _r of the image I ₂ ,r , and accumulate the events in the address-event data flow segment E _2,r into the matrix according to the coordinates of the event, use the SIFT operator to calculate the feature points of the event accumulation matrix and the image I _2,r respectively, and perform feature point matching , obtain many pairs of matching points, and calculate the homography matrix between the matrix and the image I _2,r by the matching points, the homography matrix represents the corresponding relationship between the event coordinates of the address-event data stream and the image pixel coordinates; (4) The storage module stores the calibration result: the storage module stores the corrected address-event data stream, the corrected image and the homography matrix representing the correspondence between the event coordinates and the image pixel coordinates in the address-event data stream. In the persistent memory, the spatial calibration of the event camera and the CMOS camera is completed. 4. the binocular camera space calibration method of binocular camera system according to claim 3, is characterized in that, described in the step (3) calibration module carries out spatial position calibration to corrected address-event data flow and image, Include the following steps: (3a) Construct all-zero matrix M=zeros(H, W), wherein H and W represent the row number and column number of CMOS camera output image respectively, H≥32, W≥32, make each element m in M= 0, and let i=1; (3b) let r=1; (3c) divide the address-event data flow into address-event data flow segments according to the shooting time tp _r of the image I _2,r : (3c1) Set the address-event data stream event set corresponding to the image I _r as E _2,r , let E _2,r be an empty set; (3c2) Judging whether 0<x _2, i ≤W and 0<y _{2,i ≤H} in the event e _2,i =(x 2,i ,y _2,i ,g _i ,t _i ) holds true, if so, Execute step (3c3), otherwise, let i=i+1, and execute step (3c2); (3c3) Add the event e _2,i = (x _2,i ,y _2,i ,g _i ,t _i ) into the set E _2,r , and perform step (3c4); (3c4) Judging whether t _i ≤ tp _r is established, if so, set i=i+1, and execute step (3c2), otherwise, obtain the address-event data flow segment E _2,r ={e ₂ , corresponding to the image I _{r ,r,b} |0<b≤B}, where, e _2,r,b ＝(x _2,r,b ,y _2,r,b ,g _2,r,b ,t _2,r,b ) , B represents the number of events in the set E _2,r ; (3d) let b=1; (3e) order where, /> Indicates the element whose coordinates are (x _2,r,b ,y _2,r,b ) in the matrix M; (3f) judging whether b≤B is established, if so, make b=b+1, and perform step (3e), otherwise, perform step (3g); (3g) Use the SIFT operator to extract the feature points in the matrix M to obtain N _r,3 feature points HP _r ={p _r,l |0<l≤N _r,3 }, where p _r,l =( x _r,l ,y _r,l ,F _r,l ), (x _r,l ,y _r,l ) represent the coordinates of the feature point, F _r,l represent the feature descriptor of the feature point, N _r,3 >8; (3h) Let l=1; (3i) Use the SIFT operator to calculate the feature points of the pixels of the image I _2,r on the y _r,l line, and obtain W feature points HP _r,l '={p _r,l,k '|0<k≤W }, where p _r,l,k 'represents the kth feature point of image I _2,r on line y _r,l, p _r,l,k '=(x _r,l,k ',y _{r ,l,k} ',F _r,l,k '), (x _r,l,k ',y _r,l,k ') represent the coordinates of the feature point p _r,l,k ', F _{r,l, k} 'represents the feature descriptor of the feature point p _r,l,k '; (3j) let k=1; (3k) calculate the Euclidean distance o r, l _{, k} between F _{r, l} and F r, _{l, k} '; (3l) Determine whether k<W is true, if so, set k=k+1, and execute step (3k), otherwise, obtain the feature descriptor distance set O _r,l ={or _r,l,k |0<k ≤W}, and perform step (3m); (3m) Suppose the minimum value of O _r,l is o _r,l,k' , then the feature point corresponding to p _r,l is p _r,l,k′ ′; (3n) Determine whether l≤Nr _{, 3} is established, if so, set l=l+1, and execute step (3i), otherwise, obtain the address-event data stream set E _2,r and the pixel of the event and image _Ir The set of corresponding points KP _r ={f _r,l =(p _r,l ,p _r,l,k′ ′)|0<l≤N _r,3 }, and perform step (3o); (3o) Calculate the homography matrix H _r representing the corresponding relationship between the event coordinates in the address-event data stream set E _2,r and the pixel coordinates in the image I _r through KP _r by the eight-point method; (3p) judging whether _r≤N2 is established, if so, make r=r+1, and perform step (3c), otherwise, obtain the homography matrix representing the corresponding relationship between event coordinates and image pixel coordinates in the address-event data stream Set H={H _r |0<r<N ₂ }.