CN106618450B - A kind of method of three mesh stereo endoscope self-calibration - Google Patents

Abstract

The invention discloses a kind of three mesh stereo endoscopes, can not be demarcated to stereo endoscope by extrinsic calibration template during normal use；Specifically, initial alignment is carried out first with three scope cameras of scaling board pair, obtains it and corresponds to initial alignment parameter；Secondly, in the case where ensureing that two camera parameters are constant, an other camera is zoomed in and out or focus adjustment, obtain three images that three camera synchronizations take after adjusting, characteristic point detection and matching operation are carried out to the overlapping region of three images, Feature Points Matching result based on three images, three-dimensional coordinate of the characteristic point in specified coordinate system is found out by 3D reconstruct, using gained characteristic point three-dimensional coordinate and gained characteristic point camera correspondence image to be calibrated pixel coordinate information, calibration for cameras is treated again to be demarcated, obtain the new parameter of camera to be calibrated.

Description

A method for self-calibration of trinocular stereoscopic endoscope

technical field

The invention belongs to the technical field of endoscopes, and more specifically relates to a self-calibration method for a trinocular stereoscopic endoscope.

Background technique

Minimally invasive and non-invasive surgery is one of the most important development trends in contemporary clinical medicine. Compared with traditional surgery, it can greatly improve the quality of surgery, reduce the trauma of the surgical site, reduce the pain of patients, and shorten the recovery time after surgery. , which has also become an important support for it as an international research hotspot.

The traditional three-dimensional endoscope is composed of two lenses, which is equivalent to a binocular camera. The general use process is: first, set the focal length according to the requirements before use, and then use the calibration reference object to calibrate the three-dimensional endoscope , and then extend it into the human tissue through a small wound for shooting. Due to the fixed focal length of the endoscope, clear images can only be obtained by changing the distance between the endoscope and the human tissue to be photographed during the shooting process. , and these actions will greatly increase the probability of endoscopic catheter entanglement and unintended contact with important organs of the human body during the intervention process. tissue image, the calibration parameters of the endoscope will be lost, and the calibration parameters are important parameters of the stereoscopic endoscope. The obtained three-dimensional information allows the doctor to have an accurate positioning of the relevant parts of the operation, which is conducive to the correct implementation of the operation; at the same time, the importance of calibration parameters is also indispensable in the detection process, through the calibration parameters of the stereoscopic endoscope and the acquired The image of human tissue can reconstruct the three-dimensional information of the relevant tissue surface, and then can have a more accurate understanding of the actual shape and size of the diseased tissue, which is helpful for doctors to diagnose the patient's condition.

At present, the internal parameters of the camera can be obtained through the self-calibration method without the need to refer to the calibration object. The commonly used self-calibration methods include: self-calibration methods based on absolute quadratic surfaces and infinite planes, but these traditional The calibration accuracy of the calibration method is not high, and the robustness is not enough [1] (Meng Xiaoqiao, Hu Zhanyi, "Research and Progress of Camera Self-calibration Method", Acta Automatica Sinica, 0254-4156), so this method cannot meet the actual needs.

In order to obtain accurate calibration information of the endoscope, the only way is to take out the endoscope and then use the calibration reference object to re-calibrate in the external environment. For this solution, this greatly increases the complexity of the operation. Stereoscopic endoscopes are used during surgical procedures, which are absolutely not permitted. If the more accurate camera parameters can be obtained in real time, some unnecessary operations during the use of the stereoscopic endoscope can be effectively avoided, and the complexity of the operation is also reduced, reducing the additional pain and danger brought to the patient .

Contents of the invention

The object of the present invention is to overcome the deficiencies of the prior art, and provide a method for self-calibration of a trinocular stereoscopic endoscope, so as to realize real-time calibration after parameter changes during real-time use of the stereoscopic endoscope.

In order to achieve the above-mentioned purpose of the invention, a trinocular stereoscopic endoscope of the present invention is characterized in that it includes: three independent electronic endoscopes fixed on the same cylindrical device and a plurality of independent electronic endoscopes evenly surrounding the three independent electronic endoscopes A light source composed of LEDs, and a USB interface for connecting the trinocular stereoscopic endoscope to a computer;

The positions of the three independent electronic endoscopes are relatively fixed, and are evenly distributed in the cylindrical device. The lenses of the three independent electronic endoscopes are located at the front end of the cylindrical device, and together with a light source composed of multiple LEDs, they form an imaging unit. The tail of the independent electronic endoscope is located at the rear end of the cylindrical device, and the tail of each independent electronic endoscope has a knob for independently adjusting the focal length. When a certain electronic endoscope performs zooming or focusing operations, the trinocular stereoscopic endoscope If the parameters are changed, when the trinocular stereoscopic endoscope completes the initial calibration, the other two unadjusted electronic endoscopes can acquire images for self-calibration;

When the patient needs to perform endoscopic examination, put the trinocular stereoscopic endoscope first, then focus through the knob at the tail, then image through the three electronic endoscopes with the cooperation of the light source, and finally pass the signal connected to the USB interface wire to the computer.

Wherein, the method for self-calibration of the electronic endoscope is:

(1) Trinocular stereoscopic endoscope initialization

Before use, use the calibration template to calibrate the trinocular stereoscopic endoscope, and obtain the initial calibration parameter matrices A _a , A _b , A _c and distortion coefficient vectors K ₁ , K ₂ , K ₃ of each endoscope, and each endoscope Conversion relationship between camera coordinate systems [R _ab T _ab ], [R _ac T _ac ], [R _bc T _bc ];

Among them, the general form of the parameter matrix is: α, β are the focal lengths of the camera in the x and y directions respectively, γ is the tilt factor, (u ₀ v ₀ ) is the pixel coordinate of the camera optical center projected on the image plane in the pixel coordinate system; K=[k ₁ k ₂ k ₃ k ₄ ], k ₁ , k ₂ are radial distortion coefficients, k ₃ , k ₄ are centrifugal distortion coefficients; R is the rotation matrix from the right camera coordinate system to the left camera coordinate system, and T is the right camera The translation vector from the coordinate system to the left camera coordinate system;

(2) When a certain electronic endoscope performs a zooming or focusing operation, first obtain the image of the electronic endoscope with known parameters and the electronic endoscope to be calibrated at the same time, and then use the electronic endoscope obtained in step (1) Distortion coefficients k ₁ , k ₂ , k ₃ , k ₄ of the obtained images are corrected for distortion, and then the features of the corrected images are extracted to obtain the feature points in each image, and then the feature points are matched. Finally, from the three images Select n groups of feature points from the feature points that are successfully matched;

(3), according to the conversion relationship between each coordinate system in the step (1) and the n group of feature points selected in the step (2), the solution linear equations is constructed;

Assuming that A electronic endoscope is the electronic endoscope to be calibrated, then the constructed linear equations are:

Among them, [ _ub v _b 1] ^T and [u _c v _c 1] ^T are the homogeneous coordinates of the feature points corresponding to the pixel points in the images corresponding to the electronic endoscopes B and C that have been initially calibrated; R _bc , T _bc , R _ab , T _ab are the rotation matrix and translation amount from the camera coordinate system of electronic endoscope C to the B camera coordinate system, and the rotation matrix and translation amount from the B camera coordinate system to the A camera coordinate system; [X _b Y _b Z _b ] ^T and [X _c Y _c Z _c ] ^T are the three-dimensional coordinates of the feature points corresponding to the B and C camera coordinate systems respectively; [X _a Y _a Z _a 1] ^T , [X _b Y _b Z _b 1] ^T and [X _c Y _c Z _c 1] ^T are the homogeneous coordinates of points in the A, B, and C camera coordinate systems, respectively;

Finally, solve the equations to obtain the three-dimensional coordinates [X _a1 Y _a1 Z _a1 ] ^T ...... [X _an Y _an Z _an ] ^T of n groups of feature points in the coordinate system of the endoscope camera to be calibrated;

(4), using the pixel coordinates [u _a1 v _a1 ] ^T ...... [u _an v _an ] ^T of the feature points in the corresponding images of n electronic endoscopes to be calibrated and n groups of feature points in the endoscope to be calibrated The three-dimensional coordinates in the camera coordinate system [X _a1 Y _a1 Z _a1 ] ^T ...... [X _an Y _an Z _an ] ^T , combined with the relationship between the two Construct the following linear equation:

The least square method is used to solve the linear equation to obtain the internal parameter matrix of the electronic endoscope to be calibrated, and then the nonlinear optimization algorithm is used to introduce the distortion coefficient, and the internal parameter matrix is corrected to obtain the new distortion coefficient of the electronic endoscope to be calibrated, thus Complete the self-calibration of the electronic endoscope to be calibrated.

The purpose of the invention of the present invention is achieved like this:

A trinocular stereoscopic endoscope of the present invention can calibrate the stereoscopic endoscope without the aid of an external calibration template during normal use; specifically, first use the calibration board to initially calibrate the three endoscope cameras, and obtain the corresponding initial calibration parameters; secondly, while keeping the parameters of the two cameras unchanged, adjust the zoom or focus of the other camera, obtain three images captured by the three cameras at the same time after adjustment, and perform feature points on the overlapping areas of the three images Detection and matching operations, based on the matching results of the feature points of the three images, obtain the three-dimensional coordinates of the feature points in the specified coordinate system through 3D reconstruction, and use the obtained three-dimensional coordinates of the feature points and the pixels of the obtained feature points in the corresponding image of the camera to be calibrated Coordinate information, and then calibrate the camera to be calibrated to obtain new parameters of the camera to be calibrated.

Simultaneously, a method for self-calibration of a trinocular stereoscopic endoscope of the present invention also has the following beneficial effects:

(1) During the actual use of the present invention, the function of obtaining the calibration parameters of the endoscope online can be realized without taking the endoscope out of the patient's body for calibration.

(2), the present invention introduces the operation of initial calibration, and because the initial calibration is obtained by calibrating the reference object, that is, the accuracy of the initial calibration parameters is reliable, and the initial calibration parameters of the obtained endoscope are used as a parameter change The basis for new calibration of the posterior endoscope, so the present invention is more accurate than the results of the existing self-calibration method.

Description of drawings

Fig. 1 is a kind of trinocular three-dimensional endoscope structure schematic diagram of the present invention;

Fig. 2 is a schematic diagram of lens distribution of a trinocular stereoscopic endoscope of the present invention;

Fig. 3 is a schematic block diagram of a trinocular stereoscopic endoscope of the present invention;

Fig. 4 is a flow chart of self-calibration of a trinocular stereoscopic endoscope according to the present invention.

Detailed ways

Specific embodiments of the present invention will be described below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

Example

Fig. 1 is a structural schematic diagram of a trinocular stereoscopic endoscope according to the present invention.

In this embodiment, three superocular 5mm medical endoscopes are used as the components of the trinocular stereoscopic endoscope, and the imaging sensor is a CMOS optical sensor.

As shown in Figure 1, a trinocular stereoscopic endoscope of the present invention includes: three independent electronic endoscopes fixed on the same cylindrical device and a light source composed of a plurality of LEDs evenly surrounding the three independent electronic endoscopes , and a USB interface for connecting the trinocular stereoscopic endoscope to a computer;

As shown in Figure 2, the positions of the three independent electronic endoscopes are relatively fixed and evenly distributed in the cylindrical device. The lenses of the three independent electronic endoscopes are located at the front end of the cylindrical device, and together with a light source composed of multiple LEDs, they form an imaging unit. , the tails of the three independent electronic endoscopes are located at the rear end of the cylindrical device, and the tails of each independent electronic endoscope have a knob for independently adjusting the focal length. When a certain electronic endoscope is zooming or focusing, the trinocular stereo When the endoscope parameters are changed, when the trinocular stereoscopic endoscope completes the initial calibration, the other two unadjusted electronic endoscopes can acquire images for self-calibration;

As shown in Figure 1, 1 in the figure is the lens part, which is composed of three independent endoscopes and light sources, and 2, 3, and 4 are the focal length adjustment knobs of the three electronic endoscopes;

When the patient needs to perform an endoscopic examination, first put the trinocular stereoscopic endoscope, then focus through the knob at the tail, and then image through three electronic endoscopes with the cooperation of the light source. The imaging principle is shown in Figure 3. Finally, it is sent to the computer through the signal cable connected to the USB interface.

In the actual diagnosis and treatment process, the camera parameters of the endoscope need to be changed in real time. Therefore, the trinocular stereoscopic endoscope needs to perform real-time self-calibration when the camera parameters change. The following method for self-calibration of the electronic endoscope is as follows:

(1) Trinocular stereoscopic endoscope initialization

Before use, use the calibration template to calibrate the trinocular stereoscopic endoscope, and obtain the initial calibration parameter matrices A _a , A _b , A _c and distortion coefficient vectors K ₁ , K ₂ , K ₃ of each endoscope, and each endoscope The conversion relationship between camera coordinate systems [R _ab T _ab ], [R _ac T _ac ], [R _bc T _bc ], in this embodiment, the calibration template is a checkerboard calibration board;

Among them, the general form of the parameter matrix is: α, β are the focal lengths of the camera in the x and y directions respectively, γ is the tilt factor, (u ₀ v ₀ ) is the pixel coordinate of the camera optical center projected on the image plane in the pixel coordinate system; K=[k ₁ k ₂ k ₃ k ₄ ], k ₁ , k ₂ are radial distortion coefficients, k ₃ , k ₄ are centrifugal distortion coefficients; R is the rotation matrix from the right camera coordinate system to the left camera coordinate system, and T is the right camera The translation vector from the coordinate system to the left camera coordinate system;

Among them, the corresponding model of the distortion coefficient is:

x _u -x＝(k ₁ x(x ² +y ² )+k ₂ x(x ² +y ² ) ² )+(2k ₃ xy+k ₄ (3x ² +y ² ))

y _u -y＝(k ₁ y(x ² +y ² )+k ₂ y(x ² +y ² ) ² )+(k ₃ (x ² +3y ² )+2k ₄ xy)

Among them, (x _u y _u ) and (xy) represent ideal without distortion and actual distortion respectively.

(2) When a certain electronic endoscope performs a zooming or focusing operation, first obtain the images of the electronic endoscope with known parameters and the electronic endoscope to be calibrated at the same time, and then use the images of each endoscope obtained in step (1) Distortion coefficients k ₁ , k ₂ , k ₃ , k ₄ perform distortion correction on the obtained image, and then perform feature extraction on the corrected image to obtain feature points in each image, then perform feature point matching, and finally extract the features from the three images Select n groups of feature points from the feature points that are all successfully matched;

In this embodiment, the scale-invariant feature transformation algorithm SIFT is used to obtain the feature matching point set X _ab of the corresponding image a of the electronic endoscope to be calibrated and the corresponding image b of the electronic endoscope with any known parameter, and the SIFT algorithm is used again to obtain The matching point set X ac of the corresponding image a of the electronic endoscope to be calibrated and the corresponding image c of the electronic endoscope with known parameters and the matching point set X _ac of the corresponding image b of the single endoscope with known parameters and the corresponding image c of the electronic endoscope with known parameters X _bc , if a point in X _ab , X _ac and X _bc is the same point, it is determined that this point is the matching point of the three images, that is, the matching point set of the three images is obtained, and the number of matching point groups in the matching point set n≥3.

(3), according to the conversion relationship between each coordinate system in the step (1) and the n group of feature points selected in the step (2), the solution linear equations is constructed;

Assuming that A electronic endoscope is the electronic endoscope to be calibrated, then the constructed linear equations are:

Among them, [ _ub v _b 1] ^T and [u _c v _c 1] ^T are the homogeneous coordinates of the feature points corresponding to the pixel points in the images corresponding to the electronic endoscopes B and C that have been initially calibrated; R _bc , T _bc , R _ab , T _ab are the rotation matrix and translation amount from the camera coordinate system of electronic endoscope C to the B camera coordinate system, and the rotation matrix and translation amount from the B camera coordinate system to the A camera coordinate system; [X _b Y _b Z _b ] ^T and [X _c Y _c Z _c ] ^T are the three-dimensional coordinates of the feature points corresponding to the B and C camera coordinate systems respectively; [X _a Y _a Z _a 1] ^T , [X _b Y _b Z _b 1] ^T and [X _c Y _c Z _c 1] ^T are the homogeneous coordinates of points in the A, B, and C camera coordinate systems, respectively;

Finally, solve the equations to obtain the three-dimensional coordinates [X _a1 Y _a1 Z _a1 ] ^T ...... [X _an Y _an Z _an ] ^T of n groups of feature points in the coordinate system of the endoscope camera to be calibrated;

(4), using the pixel coordinates [u _a1 v _a1 ] ^T ...... [u _an v _an ] ^T of the feature points in the corresponding images of n electronic endoscopes to be calibrated and n groups of feature points in the endoscope to be calibrated The three-dimensional coordinates in the camera coordinate system [X _a1 Y _a1 Z _a1 ] ^T ...... [X _an Y _an Z _an ] ^T , combined with the relationship between the two From the above, it can be known that one matching point and its corresponding three-dimensional coordinates can obtain two linear equations, but there are five unknown parameters, that is, at least three matching points are needed to obtain a unique solution, so the corresponding images of the three endoscope units are implemented The number of matching point groups obtained by the matching operation needs to meet the condition of n≥3. Therefore, when there are n matching points, the following equation can be obtained to construct the following linear equation:

The least square method is used to solve the linear equation to obtain the internal parameter matrix of the electronic endoscope to be calibrated, and then the nonlinear optimization algorithm is used to introduce the distortion coefficient, and the internal parameter matrix is corrected to obtain the new distortion coefficient of the electronic endoscope to be calibrated, thus Complete the self-calibration of the electronic endoscope to be calibrated.

Among them, the model adopted by the nonlinear optimization algorithm is:

Among them, K is the distortion coefficient, A is the internal parameter matrix, m _i indicates that the i-th feature point in the n groups of feature points corresponds to the actual pixel coordinates of the corresponding image of the electronic endoscope to be calibrated, and M _i is the i-th feature point to be matched The three-dimensional coordinates in the endoscope camera coordinate system, m(K,A,M _i ) is the pixel coordinate of the projection point of point M _i in the corresponding image of the electronic endoscope to be calibrated.

Although the illustrative specific embodiments of the present invention have been described above, so that those skilled in the art can understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, As long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are included in the protection list.

Claims (3)

1. a method for trinocular stereoscopic endoscope self-calibration, is characterized in that, comprises the following steps: (1) Trinocular stereoscopic endoscope initialization Before use, use the calibration template to calibrate the trinocular stereoscopic endoscope, and obtain the initial calibration parameter matrices A _a , A _b , A _c and distortion coefficient vectors K ₁ , K ₂ , K ₃ of each endoscope, and each endoscope Conversion relationship between camera coordinate systems [R _ab T _ab ], [R _ac T _ac ], [R _bc T _bc ]; Among them, the general form of the parameter matrix is: α, β are the focal lengths of the camera in the x and y directions respectively, γ is the tilt factor, (u ₀ v ₀ ) is the pixel coordinate of the camera optical center projected on the image plane in the pixel coordinate system; K=[k ₁ k ₂ k ₃ k ₄ ], k ₁ , k ₂ are radial distortion coefficients, k ₃ , k ₄ are centrifugal distortion coefficients; R is the rotation matrix from the right camera coordinate system to the left camera coordinate system, and T is the right camera The translation vector from the coordinate system to the left camera coordinate system; (2) When a certain electronic endoscope performs a zooming or focusing operation, first obtain the image of the electronic endoscope with known parameters and the electronic endoscope to be calibrated at the same time, and then use the electronic endoscope obtained in step (1) Distortion coefficients k ₁ , k ₂ , k ₃ , k ₄ of the obtained images are corrected for distortion, and then the features of the corrected images are extracted to obtain the feature points in each image, and then the feature points are matched. Finally, from the three images Select n groups of feature points from the feature points that are successfully matched; (3), according to the conversion relationship between each coordinate system in the step (1) and the n group of feature points selected in the step (2), the solution linear equations is constructed; Assuming that A electronic endoscope is the electronic endoscope to be calibrated, then the constructed linear equations are: Among them, [ _ub v _b 1] ^T and [u _c v _c 1] ^T are the homogeneous coordinates of the feature points corresponding to the pixel points in the images corresponding to the electronic endoscopes B and C that have been initially calibrated; R _bc , T _bc , R _ab , T _ab are the rotation matrix and translation amount from the camera coordinate system of electronic endoscope C to the B camera coordinate system, and the rotation matrix and translation amount from the B camera coordinate system to the A camera coordinate system; [X _b Y _b Z _b ] ^T and [X _c Y _c Z _c ] ^T are the three-dimensional coordinates of the feature points corresponding to the B and C camera coordinate systems respectively; [X _a Y _a Z _a 1] ^T , [X _b Y _b Z _b 1] ^T and [X _c Y _c Z _c 1] ^T are the homogeneous coordinates of points in the A, B, and C camera coordinate systems, respectively; Finally, solve the equations to obtain the three-dimensional coordinates [X _a1 Y _a1 Z _a1 ] ^T ...... [X _an Y _an Z _an ] ^T of n groups of feature points in the coordinate system of the endoscope camera to be calibrated; (4), using the pixel coordinates [u _a1 v _a1 ] ^T ...... [u _an v _an ] ^T of the feature points in the corresponding image of n electronic endoscopes to be calibrated and n groups of feature points in the endoscope to be calibrated The three-dimensional coordinates in the camera coordinate system [X _a1 Y _a1 Z _a1 ] ^T ...... [X _an Y _an Z _an ] ^T , combined with the relationship between the two Construct the following linear equation: The least square method is used to solve the linear equation to obtain the internal parameter matrix of the electronic endoscope to be calibrated, and then the nonlinear optimization algorithm is used to introduce the distortion coefficient, and the internal parameter matrix is corrected to obtain the new distortion coefficient of the electronic endoscope to be calibrated, thus Complete the self-calibration of the electronic endoscope to be calibrated; Wherein, the structure of the trinocular stereoscopic endoscope includes: three independent electronic endoscopes fixed on the same cylindrical device and a light source composed of multiple LEDs uniformly surrounding the three independent electronic endoscopes, and a The USB interface used to connect the trinocular stereoscopic endoscope to the computer; The positions of the three independent electronic endoscopes are relatively fixed, and are evenly distributed in the cylindrical device. The lenses of the three independent electronic endoscopes are located at the front end of the cylindrical device, and together with a light source composed of multiple LEDs, they form an imaging unit. The tail of the independent electronic endoscope is located at the rear end of the cylindrical device, and the tail of each independent electronic endoscope has a knob for independently adjusting the focal length. When a certain electronic endoscope performs zooming or focusing operations, the trinocular stereoscopic endoscope If the parameters are changed, when the trinocular stereoscopic endoscope completes the initial calibration, the other two unadjusted electronic endoscopes can acquire images for self-calibration; When the patient needs to perform endoscopic examination, put the trinocular stereoscopic endoscope first, then focus through the knob at the tail, then image through the three electronic endoscopes with the cooperation of the light source, and finally pass the signal connected to the USB interface wire to the computer. 2. A method for self-calibration of a trinocular stereoscopic endoscope according to claim 1, characterized in that, in the step (2), the value of n satisfies n≥3. 3. the method for a kind of trinocular stereoscopic endoscope self-calibration according to claim 1, is characterized in that, in described step (4), the model that nonlinear optimization algorithm adopts is: Among them, K is the distortion coefficient, A is the internal parameter matrix, m _i indicates that the i-th feature point in the n groups of feature points corresponds to the actual pixel coordinates of the corresponding image of the electronic endoscope to be calibrated, and M _i is the i-th feature point to be matched The three-dimensional coordinates in the endoscopic camera coordinate system, m(K,A,M _i ) is the pixel coordinate of the projection point of point M _i in the corresponding image of the camera to be calibrated.