CN110033465A - A kind of real-time three-dimensional method for reconstructing applied to binocular endoscope medical image - Google Patents

Abstract

The invention relates to a real-time three-dimensional reconstruction method applied to binocular endoscopic medical images. The method firstly divides complex color regions through superpixel segmentation, and the boundaries of different regions constitute a three-dimensional skeleton of an organ. Then, according to the principle of outer epipolar line constraint, the corresponding epipolar lines on the right angle of view are sequentially searched for the contour information photographed from the left angle of view. In order to obtain accurate matching point pairs, it is combined with the ORB feature description operator to quickly and accurately locate the position of the intersection of the corresponding area in the camera, and calculate the three-dimensional data of the boundary skeleton through the corresponding position relationship. Finally, the SFS method is used in the divided sub-regions to obtain the relative relationship of coordinates, and the three-dimensional coordinate information between each region is calculated based on the difference of different color gradients, and all the three-dimensional topographic coordinates of the organs in the scene are obtained. Compared with the existing three-dimensional reconstruction method, the invention solves the difficult problem of high-precision three-dimensional reconstruction of the endoscope, and has the advantages of simple operation, high reliability, low operation risk and alleviation of the pain of the patient.

Description

A real-time three-dimensional reconstruction method applied to binocular endoscopic medical images

technical field

The present invention relates to a real-time three-dimensional reconstruction and acquisition method applied to binocular endoscopic medical images, more particularly, the present invention relates to a method that can be used to display accurate three-dimensional topographic coordinates of organs in endoscopic images.

Background technique

Today, about 17.5 million people worldwide die of heart disease every year, accounting for 30% of all deaths. The number of cardiovascular patients in my country has reached 290 million, and the mortality rate is much higher than that of other diseases, which shows that it has a great impact on people's health. The traditional surgical method requires opening the thoracic cavity and sawing the sternum, which has a huge impact on the patient's respiratory function. Due to the high tension of the sternum incision, it is very difficult for patients with poor constitution to recover after surgery.

Minimally invasive surgery can not only reduce the risk of surgery, but also reduce the pain of patient treatment. Endoscopy is an important signal acquisition method for minimally invasive surgery. Doctors no longer need to open the chest, but only need to make three small holes in the chest wall, and place the thoracoscope imaging device, ultrasonic scalpel and surgical waste absorption device respectively. After the operation, the epidermal wound can heal itself, which greatly reduces the trauma and pain of the patient, and also shortens the postoperative recovery time.

Because traditional two-dimensional endoscopes cannot generate an accurate correspondence of intuitive three-dimensional position information in the surgeon's brain, physicians who have undergone long-term training are required to skillfully use it to perform operations on key parts. The existing two-dimensional endoscope still has the following risks during use:

(1) The two-dimensional endoscope lacks a sense of image depth, which causes doctors to visually misjudge important anatomical structures and their relative positions during surgery. And due to the lack of depth sense, the doctor cannot accurately judge the depth of the incision position, and it is very easy to cause accidental bleeding due to operational errors.

(2) The image distortion of the two-dimensional endoscope is large, and the human tissue structure is very complex, which will affect the fluency and progress of the operation and prolong the operation time.

The 3D space manipulation of the human body brought by 3D endoscope has set off a revolutionary change in the medical field, and minimally invasive surgery using 3D technology will become the mainstream of surgery. The use of three-dimensional endoscope greatly reduces the pain of the patient during the operation and shortens the postoperative healing time. If three-dimensional data of key parts can be obtained while acquiring three-dimensional images, the operation time can be greatly shortened and the operation risk can be reduced. The real-time three-dimensional reconstruction method of endoscopic medical images proposed by the present invention is proposed to solve the above problems.

The present invention designs a fast three-dimensional reconstruction method that combines superpixel segmentation to generate three-dimensional skeleton and SFS (Shape From Shading). Due to the different structures of internal organs and the different distribution of color shades, it is difficult for a single three-dimensional reconstruction method to obtain the overall three-dimensional shape. appearance information. The present invention firstly uses the superpixel segmentation method to segment the complex color area, and the boundaries of different areas constitute the three-dimensional skeleton of the organ. Then, according to the principle of outer epipolar line constraint, the corresponding epipolar lines on the right angle of view are sequentially searched for the contour information photographed from the left angle of view. In order to obtain accurate left and right matching point pairs, it is necessary to combine with the ORB (Oriented FAST and Rotated BRIEF) feature description operator to quickly and accurately locate the position of the intersection of the corresponding areas in the left and right cameras, and calculate the three-dimensional data of the boundary skeleton through the corresponding position relationship of the left and right cameras. . Finally, the 3D skeleton is fused with SFS. The reconstruction accuracy of the existing SFS algorithm is highly dependent on the light source model of its imaging, and only has a good 3D effect for areas with the same color, but for areas with different colors, there is a large deviation in the 3D data. In the present invention, the SFS method is used in the sub-regions after the superpixel segmentation to obtain the relative relationship of coordinates in the regions, and the generated three-dimensional skeleton coordinates are used as the benchmark, combined with the gradient difference of different colors, the three-dimensional coordinate information between the regions is sequentially calculated. Then, all the three-dimensional topographic coordinates of the organs in the scene are obtained. Real-time 3D reconstruction of the endoscope in the surgical scene is realized, providing accurate and effective navigation information for doctors.

SUMMARY OF THE INVENTION

The present invention designs a real-time three-dimensional reconstruction and acquisition method applied to endoscopic medical images. The method can be applied to operations using three-dimensional endoscopes. The three-dimensional coordinates of key parts can be obtained while acquiring three-dimensional images, which can greatly Shorten the operation time and reduce the operation risk.

The hardware device for real-time three-dimensional reconstruction of endoscopic medical images includes:

One LED cold light source;

Two optical rigid endoscopes;

Calibration platform for establishing high-precision coordinate datum;

Two 1200*1600 industrial color cameras for image acquisition;

Computers for precision control, image acquisition and data processing;

A scanning platform for placing the light source and the camera.

The real-time three-dimensional reconstruction and acquisition method of endoscopic medical images designed by the present invention has the following specific operation steps:

Step 1: Calibrate the binocular camera, set the coordinate system where the left camera A is located as O _a X _a Y _a Z _a , set the coordinate system where the right camera B is located as O _b X _b Y _b Z _b , and the two cameras are The rotation matrix between is R, the translation matrix is T, the calibration formula is shown in formula (1), r ₁₁ -r ₃₃ are the rotation matrix components of the right camera B relative to the left camera A, t _x , _ty , t _z are the translations of the right camera B relative to the left camera A matrix component;

Step 2: Extend the detection lens of the binocular endoscope into the patient's body to obtain the organ surface image, and use the median filter method to denoise and smooth the image to protect the details of the image. information;

Step 3: Use the SLIC (Simple Linear Iterative Clustering) superpixel segmentation method to segment the organ surface image obtained in step 2. First, the organ surface image is subdivided into multiple sub-regions through the color, brightness, and texture features of adjacent pixels, and then the Each sub-region image is converted from RGB color space to CIE-Lab color space, according to the number of superpixels, the seed points are evenly distributed in the image, and the three-dimensional color information and two-dimensional spatial position information are used in the neighborhood of the seed points. , calculate the distance from each searched pixel point to the seed point to cluster the pixel points, and control the size of the segmented area through the target number of the superpixel segmentation area, and finally perform iterative optimization and enhance connectivity to get For the segmented organ surface image, the distance calculation is shown in formula (2), _dc represents the color distance, _ds represents the spatial distance, _Ns represents the maximum spatial distance within the class, and _Nc represents the maximum color distance;

Step 4: According to the outer limit matching principle, select points on the segmentation boundary of the segmented organ surface image collected by the left camera, and determine the epipolar line on the segmented organ surface image collected by the right camera. And the intersection of the epipolar line and the segmentation boundary to obtain accurate left and right matching point pairs, and then combined with the ORB (Oriented FAST and Rotated BRIEF) feature description operator to locate the position of the corresponding area intersection in the left and right cameras, and select the point with the highest matching degree in the intersection point as Matching point, let the slope of the point selected by the left camera be ka , then the slope k _b of the relative matching point can be obtained by formula (3 ₎ , ka is the slope of _a certain point P in the image skeleton collected by the left camera, and k _b is The slope of the point corresponding to the point P in the image skeleton collected by the right camera, and repeating this step, the three-dimensional coordinates of the positions of all the skeletons can be obtained, forming the three-dimensional coordinates of the boundary skeleton, and recording the three-dimensional coordinate information of each sub-region boundary;

Step 5: Inside each superpixel segmented sub-region obtained in Step 4, first use SFS (ShapeFrom Shading) to carry out 3D reconstruction, select the linear method for 3D modeling, and use the finite difference method to carry out surface gradients p and q. Discrete approximation, then linearize in the direction of height Z(x, y) according to formula (4), and finally obtain the coordinate change relationship of local points;

In formula (4):

Step 6: Integrate the coordinate change relationship obtained in step 5 with the boundary skeleton coordinate information obtained in step 4 to calculate the global three-dimensional coordinates of the organ surface; the operation is completed.

The beneficial effects of the present invention are: through the method for generating superpixel segmentation and ORB fusion 3D skeleton under the constraint of epipolar line, and the fast 3D reconstruction method for fusion of 3D skeleton and SFS, the feature points of 3D reconstruction can be reduced. The matching time can also improve the number and accuracy of feature point matching. Based on the generated three-dimensional skeleton coordinates, combined with the gradient differences of different colors, the three-dimensional coordinate information between each area can be calculated in turn, and then all the three-dimensional topographic coordinates of the organs in the scene can be obtained.

Description of drawings

Figure 1: Flow chart of 3D reconstruction method;

Figure 2: Comparison before and after the SLIC superpixel segmentation method;

(a) the original image before segmentation;

(b) Pictures after segmentation;

(c) After segmentation, a boundary 3D skeleton picture is generated;

Figure 3: Image of each sub-region after SFS processing;

(a) overall area image;

(b) image of the border area;

(c) Image of borderless region.

Detailed ways

The real-time three-dimensional reconstruction method for endoscopic medical images designed by the present invention is shown in Figure 1, and the specific operations are as follows:

Complete the construction of the binocular endoscope and calibrate the binocular camera. Let the coordinate system of the left camera A be O _a X _a Y _a Z _a and the coordinate system of the right camera B to be O _b X _b Y _b Z _b , the rotation matrix between the two cameras is R, and the translation matrix is T. The calibration formula is shown in formula (5), r ₁₁ -r ₃₃ are the rotation matrix components of the right camera B relative to the left camera A, t _x , _ty , t _z are the translations of the right camera B relative to the left camera A matrix components.

The detection lens of the binocular endoscope is inserted into the body of the patient to obtain the organ surface image, and the organ surface image collected by the endoscope is denoised and image smoothed by median filtering to protect the detailed information of the image.

The SLIC superpixel segmentation method is used to segment the organ surface image, and the comparison diagrams before and after superpixel segmentation are shown in Figure 2(a) and Figure 2(b).

(1) Initialize the seed point. According to the set number of superpixels, the seed points are evenly distributed in the image. Suppose the picture has a total of N pixels and is pre-segmented into K superpixels of the same size, then the size of each superpixel block is N/K, and the distance between adjacent seed points is approximately A;

(2) Reselect the seed point within the n×n neighborhood of the seed point. The specific method is to calculate the gradient value of all pixel points in the neighborhood, and move the seed point to the place with the smallest gradient in the neighborhood;

(3) Assign a class label to each pixel in the neighborhood around each seed point. The search range of SLIC is limited to 2S×2S, which can speed up the algorithm convergence;

(4) Distance metrics, including color distance and space distance. For each searched pixel, calculate its distance from the seed point separately. The distance calculation method is shown in formula (6) and formula (7), where d _c represents the color distance, d _s represents the spatial distance, and N _S is the maximum spatial distance within the class, which is defined as N _S =S=sqrt(N/ K), for each cluster. For the maximum color distance N _c , we take a fixed constant m instead, and the value of m takes 10. The final distance metric is shown in formula (8). Since each pixel point will be searched by multiple seed points, each pixel point has a distance from the surrounding seed points, and the seed point corresponding to the minimum value is taken as the The cluster center of the pixel point;

(5) Iterative optimization. In practice, it is found that the pictures can get ideal results after 10 iterations, so the number of iterations is 10;

(6) Enhance connectivity. The flaws that appear after the above iterative optimization: multi-connectivity, too small superpixel size, and a single superpixel being cut into multiple discontinuous superpixels can be solved by enhancing connectivity.

In binocular stereo vision measurement, stereo matching is the key technology, and epipolar constraints play an important role. Two camera centers C ₀ and C ₁ observing the scene point trace the line connecting the three-dimensional point X in space to the center of the camera, and the point p of the space point X in an image can be found. Instead, the corresponding point q in the other image can be found through this point p. Searching on another image plane along this line forms an imaginary straight line L on the other image plane, which is called the epipolar line of point p. One endpoint of the epipolar line is bounded by the projection of infinity on the original observation line, and the other endpoint is bounded by the projection of the original camera center on the second image plane, which is the pole e. The fundamental matrix F maps two-dimensional image points p in one view to epipolar lines in another view. Assuming that the slope of the point selected by the left camera is ka , the slope k _b of the corresponding matching point can be obtained by formula ₍ 9). k _a is the slope of a certain point P in the skeleton of the image collected by the left camera, k _b is the slope of the point corresponding to point P in the image collected by the right camera, and r ₁₁ -r ₃₃ are the relative values of the right camera B to the point P. The rotation matrix components of the left camera A, t _x , _ty , and t _z are the translation matrix components of the right camera B relative to the left camera A.

When processing organ surface images with insignificant features, the combination of epipolar constraint and ORB matching algorithm can effectively make up for the matching error of the ORB matching algorithm, improve the matching accuracy, and can effectively exert the combination of epipolar constraint and ORB matching. Algorithmic advantages. Extract feature points through ORB features and perform matching and screening to extract the position of the intersection point of a certain area in the left camera. Through the camera calibration parameters, the epipolar line of its position in another camera can be obtained, and all the boundary points of the area intersecting with the epipolar line can be calculated. The ORB feature of the position of , compares it with the ORB feature of the point to be matched in the left camera, and finds the point with the highest similarity in the right camera as the feature to be matched. By repeating this step, the three-dimensional coordinates of the positions of all the skeletons can be obtained to form the three-dimensional coordinates of the boundary skeleton, and the three-dimensional coordinate information of the boundary of each sub-region is recorded.

The detection process of ORB features is as follows:

(1) Select pixel p in the image, and set its brightness to be _Ip ;

(2) set a threshold value T, the value is 20% of _Ip ;

(3) With p as the center, select 16 pixels on a circle with a radius of 3 pixels;

(4) If the brightness of 12 consecutive points on the selected circle is greater than IP + _T or less than IP -T, _p is regarded as a feature point.

The 3D imaging method using the fusion of 3D skeleton and SFS. The SFS algorithm is a fast and effective 3D reconstruction method, but the current reconstruction accuracy of this method is highly dependent on the light source model for imaging, and due to the different structures of the internal organs, the color distribution is also different, using a light source model is easy to cause Recovery bias of 3D data. The area after superpixel segmentation reduces the influence of complex color background, and then performs SFS 3D reconstruction on different areas after segmentation, which can improve the reconstruction accuracy. In order to achieve the unification of coordinates in different regions, the 3D coordinates of different regions are fused based on the 3D skeleton, so as to obtain accurate 3D point clouds of objects with complex color changes.

The three-dimensional modeling of the linear method is selected. The SFS used in this application is to use the finite difference method to discretely approximate the surface gradients p and q, and then perform linearization in the height Z direction. functions are applicable. Discrete approximations are used for p and q, as shown in equations (10) and (11):

For a certain pixel point (x, y) and a given image gray level E(x, y), the linear approximation of the function f in formula (12) with respect to the height map Z ^n-1 can be expanded by Taylor series , and then use Jacobian iteration to solve, after simplification, we can get:

Then, for Z(x, y) = Z ⁿ (x, y), the height map of the n-th iteration can be solved directly by formula (13):

In formula (13):

Now, set the initial estimated value of all pixels as Z ⁰ (x, y)=0, the height Z can be obtained by iterative formula (13). The image of each sub-region after SFS processing is shown in Figure 3.

The coordinate change relationship obtained after SFS is fused with the boundary skeleton coordinate information obtained after superpixel segmentation, and the global 3D coordinates are calculated. Completion of operation.

The biggest difference between the present invention and the existing three-dimensional reconstruction method has the following three points:

(1) The superpixel segmentation and ORB fusion 3D skeleton generation method under the constraint of epipolar line is proposed. This method divides the complex color regions, and the boundaries of different regions constitute the 3D skeleton of the organ. The method can not only reduce the feature point matching time of three-dimensional reconstruction, but also improve the feature point matching accuracy.

(2) A fast 3D imaging method based on the fusion of 3D skeleton and SFS is proposed. The method takes the generated 3D skeleton coordinates as a benchmark, combines the gradient differences of different colors, and sequentially calculates the 3D coordinate information between each area, and then obtains the scene. The full 3D topographic coordinates of the organ.

(3) Solve the problem of high-precision 3D reconstruction of 3D endoscopes, and promote the development and application of 3D reconstruction in the medical and industrial fields. Compared with the existing traditional surgery, this study can reduce the pain of the patient, shorten the operation time, and reduce the operation risk. In addition, this study provides a new algorithm and theoretical research basis for high-precision endoscopic 3D reconstruction in other fields.

To sum up, the advantages of the three-dimensional reconstruction method of the present invention are:

(1) Accurate three-dimensional coordinate data can be obtained;

(2) The influence of complex color background is reduced, the 3D reconstruction speed of different regions after segmentation is fast, and the reconstruction accuracy is greatly improved.

Three-dimensional medical endoscopy can not only shorten the training time of doctors and reduce the operation time, but also solve the key technical problems in the promotion of minimally invasive surgery in my country. At the same time, the promotion of 3D endoscopic surgery technology constructed by this 3D stereoscopic restoration method can promote the development of precision medicine and virtual reality medical methods, and promote the progress of my country's medical instrument industry.

The present invention and its embodiments have been described above schematically, and the description is not limited, and what is shown in the accompanying drawings is only one of the embodiments of the present invention. Therefore, if those of ordinary skill in the art are inspired by it, without departing from the purpose of the present invention, use other forms of similar parts or other forms of the layout of each part, and design a design similar to this technical solution without creativity. The technical solutions and embodiments of the invention shall belong to the protection scope of the present invention.

Claims (1)

1. the present invention devises a kind of real-time three-dimensional method for reconstructing applied to binocular endoscope medical image, characterized in that packet Containing steps are as follows:

Step 1: binocular camera being demarcated, if the coordinate system where left camera A is O_aX_aY_aZ_aIf the seat where right camera B Mark system is O_bX_bY_bZ_b, the spin matrix between two cameras is R, translation matrix T, shown in the formula of calibration such as formula (1),r₁₁-r₃₃Spin matrix point for the right camera B relative to the left camera A Amount, t_x, t_y, t_zTranslation matrix component for the right camera B relative to the left camera A；

Step 2: the detecting lenses of binocular endoscope being protruded into patient body, to obtain organ surface image, and endoscope are adopted The organ surface image of collection carries out denoising and picture smooth treatment using median filtering method, protects the detailed information of image；

Step 3: utilizing SLIC (Simple Linear Iterative Clustering) superpixel segmentation method segmentation step 2 Organ surface image is subdivided by obtained organ surface image first by the color of adjacent pixel, brightness, textural characteristics Multiple subregions, then all subregion image is transformed into CIE-Lab color space from RGB color, according to super-pixel number, Seed point is evenly distributed in image, and three-dimensional colouring information and two-dimensional space bit are utilized in the contiguous range of seed point Confidence breath calculates the pixel each searched to the distance of the seed point, to cluster to pixel, and passes through super-pixel The destination number of cut zone, controls the size of cut zone, optimization and enhancing connectivity is finally iterated, after obtaining segmentation Organ surface image, distance calculates as shown in formula (2), d_cRepresent color distance, d_sRepresent space length, N_sIt represents in class Maximum space distance, N_cRepresent maximum color distance；

Step 4: by the organ surface image after step 3 segmentation, according to outer limit matching principle, after left camera collects segmentation Organ surface image segmentation boundary selected point, right camera collect segmentation after organ surface image on determine polar curve and Polar curve and partitioning boundary intersection point, obtain accurately left and right matching double points, then with ORB (Oriented FAST and Rotated BRIEF) feature describes operator combination, positions the position of corresponding region intersection point in the camera of left and right, chooses matching degree highest in intersection point Point be match point, if the slope of left camera selected point be k_a, then the slope k of opposite match point_bIt can be obtained by formula (3), k_a For the slope of certain point P in left camera acquired image skeleton, k_bFor in right camera acquired image skeleton with the P The slope of point corresponding points, repeats this step, can obtain the three-dimensional coordinate of whole skeleton positions, constitutes boundary skeleton Three-dimensional coordinate records the three-dimensional coordinate information on each subregion boundary；

Step 5: the inside of subregion after each obtained super-pixel segmentation of step 4, first with SFS (Shape From Shading) three-dimensional reconstruction is carried out, linear approach three-dimensional modeling is chosen, surface graded p and q is carried out using finite-difference method Then discrete approximation carries out linearization process according to formula (4) on the direction height Z (x, y), finally obtain the coordinate of partial points Variation relation；

In formula (4):

Step 6: step 5 changes in coordinates relationship obtained is merged with the obtained boundary skeleton coordinate information of step 4, Calculate organ surface global three-dimensional coordinate；Operation finishes.