CN119991423B - Image registration method, device and equipment for ODT data based on coarse calibration system - Google Patents
Image registration method, device and equipment for ODT data based on coarse calibration systemInfo
- Publication number
- CN119991423B CN119991423B CN202411792089.XA CN202411792089A CN119991423B CN 119991423 B CN119991423 B CN 119991423B CN 202411792089 A CN202411792089 A CN 202411792089A CN 119991423 B CN119991423 B CN 119991423B
- Authority
- CN
- China
- Prior art keywords
- odt
- image
- wide
- point set
- transformation matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Investigating, Analyzing Materials By Fluorescence Or Luminescence (AREA)
Abstract
本发明提供一种基于粗校准系统针对ODT数据的图像配准方法、装置及设备，方法包括：从三维折射率数据中提取与宽场荧光图像匹配的单层二维数据；对单层二维数据和宽场荧光图像进行特征点提取，得到ODT特征点点集和宽场荧光特征点点集；基于聚类算法改进的适应性迭代算法，对ODT特征点点集和宽场荧光特征点点集进行匹配并迭代生成仿射变换矩阵；将仿射变换矩阵作为初始解，构建差分优化器，将仿射变换矩阵输入至差分优化器中进行优化，得到透视变换矩阵；基于透视变换矩阵，确定图像配准结果。专门针对ODT三维折射率数据与多通道宽场荧光数据实现图像配准，进而实现同一样本同一时刻下无标记非特异性成像和特异性成像方式的图像融合。
The present invention provides an image registration method, device, and apparatus for ODT data based on a coarse calibration system. The method comprises: extracting single-layer two-dimensional data matching a widefield fluorescence image from three-dimensional refractive index data; extracting feature points from the single-layer two-dimensional data and the widefield fluorescence image to obtain an ODT feature point set and a widefield fluorescence feature point set; matching the ODT feature point set and the widefield fluorescence feature point set using an adaptive iterative algorithm improved by a clustering algorithm and iteratively generating an affine transformation matrix; constructing a differential optimizer using the affine transformation matrix as an initial solution, inputting the affine transformation matrix into the differential optimizer for optimization to obtain a perspective transformation matrix; and determining the image registration result based on the perspective transformation matrix. The method specifically implements image registration for ODT three-dimensional refractive index data and multi-channel widefield fluorescence data, thereby achieving image fusion of label-free nonspecific imaging and specific imaging methods for the same sample at the same time.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image registration method, apparatus, and device for ODT data based on a coarse calibration system.
Background
Optical diffraction tomography (Optical Diffraction Tomography, ODT) is a technology for three-dimensional microscopic imaging, the incident angle is changed by matching laser with a scanning galvanometer, the multi-angle data of a sample is obtained, and the three-dimensional refractive index distribution of the sample is reconstructed according to a reconstruction algorithm, so that the method has the advantages of non-invasiveness and high resolution. Wide-field fluorescence imaging (Wide-field Fluorescence Imaging) is a technique that utilizes fluorescent markers and Wide-angle illumination to obtain images of a sample. In order to realize the complementation of label-free nonspecific imaging and specific imaging modes so as to achieve comprehensive, integral and dynamic observation and research on a sample, it is necessary to integrate two imaging technologies in a multi-mode microscopic imaging system and realize the image fusion of multi-mode imaging of the same sample under the same state through an image registration method.
For the requirements of traditional natural image registration, the following method is generally adopted, namely, feature points are extracted by adopting a scale invariant algorithm, then feature matching is carried out by adopting a violent matching or k-nearest neighbor algorithm, a transformation model is selected according to possible transformation, and finally parameter estimation is carried out by adopting a random sampling consistency algorithm. The general registration method is used for registering the ODT data, and firstly, the ODT data reflects three-dimensional refractive index information of a sample rather than intensity or gray scale information due to imaging characteristics, and almost no correct result can be generated when feature points are extracted due to specific noise and errors of imaging, and secondly, even if the data is preprocessed into binary gray scale images with similar image characteristics as wide field fluorescence as far as possible and then registered, the registration accuracy is low, the registration effect is poor, even error information affecting judgment and decision is caused, and to avoid the problem, a hardware system is required to be adjusted to be accurate and even ideal, and a globally optimal violence registration algorithm which takes quite long time is required to be used, so that a great amount of economic and time cost is increased.
Disclosure of Invention
The invention provides an image registration method, device and equipment for ODT data based on a coarse calibration system, which are used for solving the defect that the demand of ODT data registration cannot be processed in the prior art, and ensuring high accuracy and high processing efficiency so as to realize the image registration and fusion of the ODT data.
The invention provides an image registration method for ODT data based on a coarse calibration system, which comprises the following steps:
Acquiring three-dimensional refractive index data of an ODT image to be registered, and extracting single-layer two-dimensional data matched with a wide-field fluorescent image from the three-dimensional refractive index data, wherein the ODT image to be registered and the wide-field fluorescent image are acquired by the same sample at the same time, and the ODT image to be registered and the wide-field fluorescent image have overlapping areas;
respectively extracting characteristic points of the single-layer two-dimensional data and the wide-field fluorescent image to obtain an ODT characteristic point set and a wide-field fluorescent characteristic point set;
Matching the ODT characteristic point set and the wide-field fluorescence characteristic point set based on an adaptive iterative algorithm improved by a clustering algorithm, and iterating to generate an affine transformation matrix;
Taking the affine transformation matrix as an initial solution, constructing a differential optimizer, and inputting the affine transformation matrix into the differential optimizer for optimization to obtain a perspective transformation matrix;
And determining an image registration result of the three-dimensional refractive index data and the wide-field fluorescent image based on the perspective transformation matrix.
According to the image registration method for ODT data based on the coarse calibration system, the fitness function of the differential optimizer comprises any one of mean square error and structural similarity.
According to the image registration method for ODT data based on the coarse calibration system provided by the invention, the characteristic point extraction is respectively carried out on the single-layer two-dimensional data and the wide-field fluorescent image to obtain an ODT characteristic point set and a wide-field fluorescent characteristic point set, and the image registration method comprises the following steps:
and respectively extracting characteristic points from the single-layer two-dimensional data and the wide-field fluorescent image by using a scale space extremum detection algorithm to obtain the ODT characteristic point set and the wide-field fluorescent characteristic point set.
According to the image registration method for ODT data based on a coarse calibration system, the acquisition steps of the single-layer two-dimensional data and the wide-field fluorescent image comprise the following steps:
acquiring original single-layer two-dimensional data and an original wide-field fluorescent image;
sequentially performing image binarization, image color reversal and edge feature extraction on the original single-layer two-dimensional data, and filling holes in edge information to obtain the single-layer two-dimensional data;
and respectively carrying out image binarization and image smoothing on the original wide-field fluorescent image to obtain the wide-field fluorescent image.
According to the image registration method for ODT data based on the coarse calibration system, which is provided by the invention, the adaptive iterative algorithm improved based on the clustering algorithm is used for matching the ODT characteristic point set and the wide-field fluorescence characteristic point set and generating an affine transformation matrix in an iterative manner, and the method further comprises the following steps:
And reversely screening the ODT characteristic point set and the wide-field fluorescence characteristic point set which meet the error requirement in the iteration process.
According to the image registration method for ODT data based on the coarse calibration system, which is provided by the invention, the ODT characteristic point set and the wide-field fluorescence characteristic point set which meet the error requirement are reversely screened in the iterative process, and the image registration method comprises the following steps:
Step s1, determining an initial affine transformation matrix based on the ODT characteristic point set and the wide-field fluorescence characteristic point set, and determining a transformation ODT image based on the initial affine transformation matrix and the ODT image to be registered;
Step s2, calculating Manhattan distance of the corresponding point set after transformation based on the transformation ODT image and the wide-field fluorescent image, taking the Manhattan distance as an evaluation index, and screening out the ODT characteristic point set and the wide-field fluorescent characteristic point set with errors larger than a set threshold based on the evaluation index;
and repeating the iterative step s1 and the iterative step s2 to obtain the ODT characteristic point set, the wide-field fluorescence characteristic point set and the affine transformation matrix which meet the error requirement.
The invention also provides an image registration system for ODT data based on the coarse calibration system, which comprises the following modules:
The device comprises an acquisition unit, a detection unit and a detection unit, wherein the acquisition unit is used for acquiring three-dimensional refractive index data of an ODT image to be registered and extracting single-layer two-dimensional data matched with a wide-field fluorescent image from the three-dimensional refractive index data, the ODT image to be registered and the wide-field fluorescent image are acquired by the same sample at the same time, and the ODT image to be registered and the wide-field fluorescent image have an overlapping area;
The characteristic point extraction unit is used for extracting characteristic points of the single-layer two-dimensional data and the wide-field fluorescent image respectively to obtain an ODT characteristic point set and a wide-field fluorescent characteristic point set;
the matching unit is used for matching the ODT characteristic point set and the wide-field fluorescence characteristic point set based on an adaptive iterative algorithm and generating an affine transformation matrix in an iterative mode;
the perspective transformation matrix solving unit is used for taking the affine transformation matrix as an initial solution, constructing a differential optimizer, inputting the affine transformation matrix into the differential optimizer for optimization, and obtaining a perspective transformation matrix;
and the determining unit is used for determining an image registration result of the three-dimensional refractive index data and the wide-field fluorescent image based on the perspective transformation matrix.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the image registration method for ODT data based on the coarse calibration system as described in any one of the above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the coarse calibration system-based image registration method for ODT data as described in any of the above.
The present invention also provides a computer program product comprising a computer program which, when executed by a processor, implements the coarse calibration system based image registration method for ODT data as described in any of the above.
The image registration method, the device and the equipment based on the coarse calibration system for the ODT data, provided by the invention, on one hand, realize image registration specifically for the ODT three-dimensional refractive index data and the multi-channel wide-field fluorescence data, further realize image fusion of a label-free nonspecific imaging mode and a specific imaging mode under the same sample at the same time, capture the integral structure and the morphological change of the sample, highlight interaction and process of specific biomolecules or chemical components, and provide new visual angles and clues for scientific research, and on the other hand, provide an image registration method with lower time cost, higher precision and faster time efficiency for a microscopic imaging system with larger coarse calibration or error.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of an image registration method for ODT data based on a coarse calibration system according to the present invention.
FIG. 2 is a schematic diagram of an ODT three-dimensional refractive index volume image of a fluorescent microsphere sample provided by the invention.
FIG. 3 is a schematic diagram of a wide field fluorescence image of a fluorescent microsphere sample provided by the present invention.
Fig. 4 is a schematic diagram of a pretreated ODT image provided by the present invention.
FIG. 5 is a graph showing the fusion effect of ODT three-dimensional data and multichannel wide-field fluorescence data of a cell sample provided by the invention.
Fig. 6 is a second flow chart of the image registration method for ODT data based on the coarse calibration system provided by the invention.
Fig. 7 is a schematic structural diagram of an image registration system for ODT data based on a coarse calibration system provided by the present invention.
Fig. 8 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a schematic flow chart of an image registration method for ODT data based on a coarse calibration system according to the present invention, and as shown in fig. 1, the method includes steps 110, 120, 130, 140 and 150.
Step 110, acquiring three-dimensional refractive index data of an ODT image to be registered, and extracting single-layer two-dimensional data matched with a wide-field fluorescent image from the three-dimensional refractive index data, wherein the ODT image to be registered and the wide-field fluorescent image are acquired by the same sample at the same time, and the ODT image to be registered and the wide-field fluorescent image have an overlapping area.
Specifically, the ODT image to be registered and the wide-field fluorescent image are obtained by a multi-mode microscopic imaging system at the same time by the same sample, wherein the system state of the multi-mode microscopic imaging system is rough calibration, and the system errors comprise different angles of illumination light sources of two mode light sources, inclination of sample or displacement table levelness, insufficient reset precision of a mirror bracket turntable and the like.
The multimode microscopic imaging system comprises a laser, a displacement table, two cameras, a microscope stand, a scanning galvanometer, an electric diaphragm, a fluorescent lamp box and optical elements, wherein the microscope stand is arranged on the bottom surface, other hardware devices are arranged on the microscope stand, the system is used for imaging through a specific light path and matched with hardware, and ODT data are obtained by acquiring data from the cameras and calculating and imaging through a calculation module.
Accordingly, standard fluorescent stained microspheres are first prepared as registration samples because imaging of standard microspheres typically has more distinct and sharp-edged image features, and for the specificity of a wide-field fluorescent image, specific-band specific microspheres can be fully stimulated to emit light. And then acquiring ODT images and wide-field fluorescent images of the microspheres with different field sizes and overlapping areas at the same time through a multi-mode microscopic imaging system.
Fig. 2 is a schematic diagram of an ODT three-dimensional refractive index volume image of a fluorescent microsphere sample provided by the present invention, and fig. 3 is a schematic diagram of a wide-field fluorescent image of a fluorescent microsphere sample provided by the present invention, as shown in fig. 2 and 3. For the three-dimensional refractive index volume data of the ODT image to be registered, which is shown in fig. 2, and reflects the refractive index volume data of a sample in a certain z-axis interval, different layer characteristics of the image such as gray scale, gradient, edge information and large noise change, the specific layer truly corresponding to the focal plane of the wide-field fluorescence imaging data, namely the actual z-axis height, needs to be screened out through algorithm calculation.
For the ODT data of the screened specific layer, the data reflects the sample refractive index information instead of intensity or gray scale information due to its calculated imaging characteristics, and the accurate result is hardly generated when the feature point extraction is performed due to the calculated imaging specific noise and error. Fig. 4 is a schematic diagram of a pretreated ODT image provided by the present invention, as shown in fig. 4, a series of pretreatment methods can be performed on ODT data of a specific layer to make it exhibit and enhance image characteristics similar to those of wide-field fluorescent microspheres. For example, the refractive index of the screened original single-layer two-dimensional data is mapped to a gray space by a design algorithm, then image binarization, image color reversal and edge feature extraction are sequentially carried out, and then holes are filled in edge information, so that the single-layer two-dimensional data is obtained. And respectively carrying out image binarization and image smoothing on the original wide-field fluorescent image to obtain a wide-field fluorescent image.
Gray scale mapping of images, namely mapping floating point number range of original refractive index image values from about-2.000000 to 1.7000000 to integer values of 0 to 255, and properly cutting off the upper and lower limits of 3 to 5 percent of values according to the integral distribution of refractive index values to ensure that the mapped images show contrast which is easier to register
The binarization of the image, namely, the gray value of the pixel point on the image is set to 0 or 255, so that the whole image presents obvious visual effects of only black and white. Binarization is the simplest method of image segmentation, which converts a gray-scale image into a binary image.
Image inversion refers to inverting the color value of each pixel in an image. In the gray color space, this typically means subtracting the original value from the gray value of each pixel by 255.
Edge feature extraction refers to extracting pixels or regions with edge features from an image. The edges generally represent the boundaries of objects or discontinuously changing portions, and have important significance for tasks such as segmentation, identification, reconstruction and the like of images.
The purpose of filling the holes is typically to improve the visual effect of the image, to increase the accuracy of subsequent image analysis or recognition.
Image smoothing is effectively a low-pass filter that smoothes the image gray scale, reducing abrupt gradients, and thus improving image quality.
And 120, extracting characteristic points of the single-layer two-dimensional data and the wide-field fluorescent image respectively to obtain an ODT characteristic point set and a wide-field fluorescent characteristic point set.
Specifically, feature point extraction is respectively carried out on the single-layer two-dimensional data and the wide-field fluorescent image, and an ODT feature point set and a wide-field fluorescent feature point set are obtained.
The feature point extraction can be performed on the single-layer two-dimensional data and the wide-field fluorescent image respectively by using a scale space extremum detection algorithm, so that an ODT feature point set and a wide-field fluorescent feature point set are obtained.
In order to detect feature points on different scales, a scale space needs to be first constructed for the image by performing a gaussian blur operation on the image and repeating at different scales. Each blurred image represents a different scale version of the image so that the algorithm can detect features of different sizes. In the constructed scale space, extreme points are detected by comparison of neighboring pixels. Specifically, each pixel point is compared to its pixels at adjacent scales and adjacent spatial locations. If a point appears to be a local maximum (or minimum) in all comparisons, it is considered a potential feature point.
And 130, matching the ODT characteristic point set and the wide-field fluorescence characteristic point set based on an adaptive iterative algorithm improved by a clustering algorithm, and iterating to generate an affine transformation matrix.
Specifically, considering that the detected potential feature points may further contain some unreliable points, further screening is needed, and the ODT feature point set and the broad-field fluorescence feature point set can be matched and an affine transformation matrix is iteratively generated based on an adaptive iterative algorithm improved by a clustering algorithm.
And 140, taking the affine transformation matrix generated in the last time as an initial solution, constructing a differential optimizer, and inputting the affine transformation matrix into the differential optimizer for optimization to obtain a perspective transformation matrix.
Specifically, taking an affine transformation matrix as an initial solution, simplifying parameters and an estimation interval thereof, and adopting a differential optimization method to globally estimate the perspective transformation matrix so as to compensate perspective transformation possibly existing and ensure accuracy and rapid convergence of a registration result.
Namely, the affine transformation matrix is input into a differential optimizer for optimization, and a perspective transformation matrix is obtained.
Here, the fitness function of the differential optimizer includes any one of mean square error (Mean Squared Error, MSE), structural similarity (Structural Similarity, SSIM).
Here, the affine transformation matrix generated in the last iteration is taken as the initial solution for estimating the perspective transformation matrix next, and the affine transformation matrix is:
Wherein, the Is the coordinates of the original image, and the coordinate of the original image,,Is the transformed coordinates; In order to be able to rotate the disc in an amount, ,,Is the translation amount.
The perspective transformation matrix is:
Wherein, the Is the coordinates of the original image, and the coordinate of the original image,,Is the transformed coordinates; In order to be able to rotate the disc in an amount, Is the translation amount.
In order to ensure registration accuracy, since the coarse calibration system mainly has linear transformation such as rotation, translation, scaling and the like, but some nonlinear transformation caused by light paths, camera levelness or displacement platform levelness, namely perspective transformation, is not ignored, iterative calculation is carried out on 9 parameters of a perspective transformation matrix by directly designing a global optimizer, so that the steps firstly estimate 6 parameters of the affine transformation matrix by adopting a local rapid iteration method, and screening characteristic point pairs in an iteration process to further ensure point set high matching degree and registration accuracy, and taking a characteristic point set generated by the last iteration and the affine transformation matrix as inputs to globally estimate the perspective transformation matrix. The specific method for screening the characteristic point pairs is that the Manhattan distance is carried out on the transformed ODT point pairs and the wide-field fluorescent point pairs, an error feedback function is established by taking the Manhattan distance as a main evaluation index, and the point pairs with the error larger than a set threshold value are screened and removed.
Step 150, determining an image registration result of the three-dimensional refractive index data and the wide-field fluorescent image based on the perspective transformation matrix.
Specifically, after obtaining the perspective transformation matrix, an image registration result of the three-dimensional refractive index data and the wide-field fluorescent image may be determined based on the perspective transformation matrix.
Based on the obtained perspective transformation matrix, the perspective transformation matrix is applied to the actual sample data acquired subsequently, the image registration of optical diffraction tomography and wide-field fluorescence imaging is completed, the fusion display of two imaging mode data is realized based on the registration result, and fig. 5 is an ODT three-dimensional data and multi-channel wide-field fluorescence data fusion effect diagram of a cell sample, wherein the ODT image presents a super-resolution panoramic image of the cell sample, and the multi-channel wide-field fluorescence data specifically presents cell membranes and lipid droplets, as shown in fig. 5.
The method provided by the embodiment of the invention realizes image registration aiming at ODT three-dimensional refractive index data and multichannel wide-field fluorescence data, further realizes image fusion of label-free nonspecific imaging and specific imaging modes of the same sample at the same time, can capture the integral structure and morphological change of the sample, can highlight interaction and process of specific biomolecules or chemical components, provides new visual angles and clues for scientific research, and provides an image registration method with lower time cost, higher precision and faster time efficiency aiming at a microscopic imaging system with coarse calibration or larger error.
Based on the above embodiment, step 130 further includes:
And reversely screening the ODT characteristic point set and the wide-field fluorescence characteristic point set which meet the error requirement in the iteration process.
In particular, the detected potential feature points may also contain some unreliable points, so further screening is required, and rapid estimation of affine transformation matrix is performed by using adaptive iterative algorithm improved by clustering algorithm. And setting a threshold value of the number of ideal characteristic point pairs according to the information density of the calibration image, substituting the threshold value into an affine transformation matrix, and reversely screening out the characteristic point pairs meeting the conditions by setting an error evaluation index. Repeating the steps until the ideal characteristic point pair meets the set threshold value.
The method comprises the steps of reversely screening an ODT characteristic point set and a wide-field fluorescence characteristic point set which meet the error requirement in the iteration process, wherein the specific steps are as follows:
Step s1, determining an initial affine transformation matrix based on the ODT characteristic point set and the wide-field fluorescence characteristic point set, and determining a transformation ODT image based on the initial affine transformation matrix and the ODT image to be registered.
Step s2, calculating Manhattan distance of the corresponding point set after transformation based on the transformation ODT image and the wide-field fluorescent image, taking the Manhattan distance as an evaluation index, and screening out the ODT characteristic point set and the wide-field fluorescent characteristic point set with errors larger than a set threshold based on the evaluation index;
And repeating the iterative step s1 and the iterative step s2 to obtain an ODT characteristic point set, a wide-field fluorescence characteristic point set and an affine transformation matrix which meet the error requirement.
That is, the ODT feature point set and the wide-field fluorescent feature point set are matched using a feature point matching algorithm. And calculating an initial affine transformation matrix according to the matched characteristic point pairs. And applying the initial affine transformation matrix to the ODT image to be registered to obtain a transformed ODT image. And finding corresponding characteristic point pairs between the transformed ODT image and the wide-field fluorescent image. The manhattan distance between the pairs of corresponding feature points is calculated. An error threshold is set, and characteristic point pairs with Manhattan distance larger than the set threshold are screened out, namely the characteristic point pairs are considered to introduce larger errors in the registration process.
And performing the next iteration by using the filtered characteristic point set, recalculating an affine transformation matrix by using the updated characteristic point set, and applying the new affine transformation matrix to the ODT image to obtain a new transformation ODT image. And calculating a new Manhattan distance, and screening the characteristic point set again. The iteration is terminated when the average or maximum value of the manhattan distance is less than a set error requirement. The ODT characteristic point set, the wide-field fluorescence characteristic point set and the affine transformation matrix obtained at the moment are the final results.
Based on any of the above embodiments, fig. 6 is a second flowchart of an image registration method for ODT data based on a coarse calibration system according to the present invention, as shown in fig. 6, the method includes:
Step S1, standard fluorescent staining microspheres are selected as registration samples.
Step S2, acquiring an ODT image and a wide-field fluorescent image through a multi-mode microscopic imaging system, wherein the ODT image comprises a visual field area and a focal plane layer of the wide-field fluorescent image.
And S3, screening out the specific layer two-dimensional data corresponding to the focal plane of the wide-field fluorescence imaging data from the ODT three-dimensional image through an algorithm.
And S4, preprocessing such as gray mapping, binarization, edge detection, hole filling and the like is carried out on the screened specific layer ODT data, so that the specific layer ODT data represent and enhance the image characteristics similar to the wide-field fluorescence.
And S5, extracting characteristic points of the preprocessed ODT data and the preprocessed wide-field fluorescence imaging data by using a scale space extremum detection algorithm.
And S6, estimating an affine transformation matrix for the characteristic point pairs by using an adaptive iterative algorithm improved by a clustering algorithm.
And S7, setting a threshold value of the number of ideal characteristic point pairs according to the information density of the calibration image, substituting the threshold value into an affine transformation matrix, and reversely screening out the characteristic point pairs meeting the conditions by setting an error evaluation index.
And S8, repeating the steps S6-S7 until the ideal characteristic point pair meets a preset threshold.
Step S9, setting initial values of the perspective transformation matrix based on the affine transformation matrix finally obtained in the step S8.
Step S10, designing a global optimizer, and globally estimating a perspective transformation matrix by adopting a differential optimization method.
Step S11, a global estimated perspective transformation matrix is applied to obtain the transformation of the ODT image to the registered image.
And S12, selecting different channels to fuse the ODT image and the wide-field fluorescent image.
The image registration system for ODT data based on the coarse calibration system provided by the invention is described below, and the image registration system for ODT data based on the coarse calibration system described below and the image registration method for ODT data based on the coarse calibration system described above can be referred to correspondingly with each other.
Based on any of the above embodiments, the present invention provides an image registration system for ODT data based on a coarse calibration system, and fig. 7 is a schematic structural diagram of the image registration system for ODT data based on a coarse calibration system provided by the present invention, as shown in fig. 7, the system includes:
An obtaining unit 710, configured to obtain three-dimensional refractive index data of an ODT image to be registered, and extract single-layer two-dimensional data matching a wide-field fluorescent image from the three-dimensional refractive index data, where the ODT image to be registered and the wide-field fluorescent image are obtained by the same sample at the same time, and the ODT image to be registered and the wide-field fluorescent image have an overlapping area;
The feature point extracting unit 720 is configured to extract feature points of the single-layer two-dimensional data and the wide-field fluorescent image respectively, so as to obtain an ODT feature point set and a wide-field fluorescent feature point set;
A matching unit 730, configured to match the ODT feature point set and the broad-field fluorescent feature point set based on an adaptive iterative algorithm, and iterate to generate an affine transformation matrix;
The perspective transformation matrix solving unit 740 is configured to take the affine transformation matrix as an initial solution, construct a differential optimizer, and input the affine transformation matrix into the differential optimizer for optimization to obtain a perspective transformation matrix;
A determining unit 750 for determining an image registration result of the three-dimensional refractive index data and the wide-field fluorescent image based on the perspective transformation matrix.
The system judges according to the registration requirement, if the registration result is formed, the registration matrix recorded in the computer storage medium is only required to be directly applied to the data, the calibration flows of preprocessing, feature extraction, matching, screening and the like are not required to be carried out, and if the registration result is not formed, the registration method is executed.
The method also comprises the steps of selecting and requiring a sample, wherein the sample needs standard fluorescent dye microspheres to ensure that the imaging quality meets the requirement.
The system provided by the embodiment of the invention realizes image registration aiming at ODT three-dimensional refractive index data and multichannel wide-field fluorescence data, further realizes image fusion of label-free nonspecific imaging and specific imaging modes of the same sample at the same time, can capture the integral structure and morphological change of the sample, can highlight interaction and process of specific biomolecules or chemical components, provides new visual angles and clues for scientific research, and provides an image registration method with lower time cost, higher precision and faster time efficiency aiming at a microscopic imaging system with coarse calibration or larger error.
Based on any of the above embodiments, the fitness function of the differential optimizer includes any one of a mean square error and a structural similarity.
Based on any of the above embodiments, the feature point extraction unit 720 is specifically configured to:
and respectively extracting characteristic points from the single-layer two-dimensional data and the wide-field fluorescent image by using a scale space extremum detection algorithm to obtain the ODT characteristic point set and the wide-field fluorescent characteristic point set.
Based on any one of the above embodiments, the system further includes a data acquisition unit, where the data acquisition unit is specifically configured to:
acquiring original single-layer two-dimensional data and an original wide-field fluorescent image;
sequentially performing image binarization, image color reversal and edge feature extraction on the original single-layer two-dimensional data, and filling holes in edge information to obtain the single-layer two-dimensional data;
and respectively carrying out image binarization and image smoothing on the original wide-field fluorescent image to obtain the wide-field fluorescent image.
Based on any of the above embodiments, the apparatus further comprises a reverse screening unit, specifically configured to:
And the inverse screen unit is used for inverse screening the ODT characteristic point set and the wide-field fluorescence characteristic point set which meet the error requirement in the iterative process.
Based on any of the above embodiments, the reverse sifter unit is specifically configured to:
Step s1, determining an initial affine transformation matrix based on the ODT characteristic point set and the wide-field fluorescence characteristic point set, and determining a transformation ODT image based on the initial affine transformation matrix and the ODT image to be registered;
Step s2, calculating Manhattan distance of the corresponding point set after transformation based on the transformation ODT image and the wide-field fluorescent image, taking the Manhattan distance as an evaluation index, and screening out the ODT characteristic point set and the wide-field fluorescent characteristic point set with errors larger than a set threshold based on the evaluation index;
and repeating the iterative step s1 and the iterative step s2 to obtain the ODT characteristic point set, the wide-field fluorescence characteristic point set and the affine transformation matrix which meet the error requirement.
Fig. 8 is a schematic structural diagram of an electronic device according to the present invention, as shown in fig. 8, the electronic device may include a processor 810, a communication interface (Communications Interface) 820, a memory 830, and a communication bus 840, where the processor 810, the communication interface 820, and the memory 830 complete communication with each other through the communication bus 840. The processor 810 can call logic instructions in the memory 830 to execute an image registration method for ODT data based on a coarse calibration system, wherein the method comprises the steps of obtaining three-dimensional refractive index data of an ODT image to be registered, extracting single-layer two-dimensional data matched with a wide-field fluorescent image from the three-dimensional refractive index data, obtaining the ODT image to be registered and the wide-field fluorescent image which are obtained by the same sample at the same time, and have overlapping areas, extracting characteristic points of the single-layer two-dimensional data and the wide-field fluorescent image respectively to obtain an ODT characteristic point set and a wide-field fluorescent characteristic point set, carrying out matching and iterative generation of an affine transformation matrix based on an adaptive iterative algorithm improved by a clustering algorithm, taking the affine transformation matrix as an initial solution, constructing a differential optimizer, inputting the affine transformation into the differential optimizer to be optimized, and obtaining a transformation matrix, and determining a perspective transformation matrix of the ODT characteristic point set and the wide-field fluorescent image registration result based on the transformation matrix.
Further, the logic instructions in the memory 830 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. The storage medium includes a U disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, an optical disk, or other various media capable of storing program codes.
On the other hand, the invention also provides a computer program product, which comprises a computer program, wherein the computer program can be stored on a non-transitory computer readable storage medium, when the computer program is executed by a processor, the computer can execute the image registration method based on the ODT data provided by the above methods, and the image registration method based on the coarse calibration system comprises the steps of acquiring three-dimensional refractive index data of an ODT image to be registered, extracting single-layer two-dimensional data matched with a wide-field fluorescent image from the three-dimensional refractive index data, acquiring the same sample of the ODT image to be registered and the wide-field fluorescent image at the same time, and extracting characteristic points of the ODT image to be registered and the wide-field fluorescent image to obtain an ODT characteristic point set and a wide-field fluorescent characteristic point set, carrying out matching on the ODT characteristic point set and the wide-field fluorescent characteristic point set based on an adaptive iterative algorithm improved based on a clustering algorithm, carrying out iterative generation of an affine transformation matrix, taking the initial affine transformation matrix and the differential transformation matrix as an optimal transformation matrix, and carrying out optimization transformation to the optimal transformation result of the optimal transformation matrix to the three-dimensional transformation matrix, and obtaining the optimal transformation result based on the optimal transformation matrix.
In still another aspect, the present invention further provides a non-transitory computer readable storage medium, on which a computer program is stored, the computer program being implemented when executed by a processor to perform the method for image registration of ODT data based on a coarse calibration system provided by the above methods, the method comprising obtaining three-dimensional refractive index data of an ODT image to be registered, and extracting single-layer two-dimensional data matched with a wide-field fluorescent image from the three-dimensional refractive index data, the ODT image to be registered and the wide-field fluorescent image being obtained at the same time and having overlapping areas, respectively performing feature point extraction on the single-layer two-dimensional data and the wide-field fluorescent image to obtain an ODT feature point set and a wide-field fluorescent feature point set, performing matching and iterative generation of an affine transformation matrix based on an improved adaptive iterative algorithm of a clustering algorithm, taking the affine transformation matrix as an initial solution, and constructing a differential optimizer, inputting the affine transformation matrix into the differential optimizer, and performing optimization of the perspective transformation based on the differential transformation matrix, and obtaining a three-dimensional perspective transformation result of the perspective transformation matrix.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
It should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention, and not for limiting the same, and although the present invention has been described in detail with reference to the above-mentioned embodiments, it should be understood by those skilled in the art that the technical solution described in the above-mentioned embodiments may be modified or some technical features may be equivalently replaced, and these modifications or substitutions do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solution of the embodiments of the present invention.
Claims (10)
1. An image registration method for ODT data based on a coarse calibration system, comprising:
Acquiring three-dimensional refractive index data of an ODT image to be registered, and extracting single-layer two-dimensional data matched with a wide-field fluorescent image from the three-dimensional refractive index data, wherein the ODT image to be registered and the wide-field fluorescent image are acquired by the same sample at the same time, and the ODT image to be registered and the wide-field fluorescent image have overlapping areas;
respectively extracting characteristic points of the single-layer two-dimensional data and the wide-field fluorescent image to obtain an ODT characteristic point set and a wide-field fluorescent characteristic point set;
Matching the ODT characteristic point set and the wide-field fluorescence characteristic point set based on an adaptive iterative algorithm improved by a clustering algorithm, and iterating to generate an affine transformation matrix;
Taking the affine transformation matrix as an initial solution, constructing a differential optimizer, and inputting the affine transformation matrix into the differential optimizer for optimization to obtain a perspective transformation matrix;
And determining an image registration result of the three-dimensional refractive index data and the wide-field fluorescent image based on the perspective transformation matrix.
2. The method of image registration for ODT data based on a coarse calibration system according to claim 1, wherein the fitness function of the differential optimizer includes any one of mean square error, structural similarity.
3. The method for image registration of ODT data based on a coarse calibration system according to claim 1, wherein the performing feature point extraction on the single-layer two-dimensional data and the wide-field fluorescent image to obtain an ODT feature point set and a wide-field fluorescent feature point set includes:
and respectively extracting characteristic points from the single-layer two-dimensional data and the wide-field fluorescent image by using a scale space extremum detection algorithm to obtain the ODT characteristic point set and the wide-field fluorescent characteristic point set.
4. The image registration method for ODT data based on the coarse calibration system according to any one of claims 1 to 3, characterized by the acquisition step of the single-layer two-dimensional data and the wide-field fluorescent image comprising:
acquiring original single-layer two-dimensional data and an original wide-field fluorescent image;
sequentially performing image binarization, image color reversal and edge feature extraction on the original single-layer two-dimensional data, and filling holes in edge information to obtain the single-layer two-dimensional data;
and respectively carrying out image binarization and image smoothing on the original wide-field fluorescent image to obtain the wide-field fluorescent image.
5. The image registration method for ODT data based on a coarse calibration system according to any one of claims 1 to 3, wherein the adaptive iterative algorithm modified based on a clustering algorithm matches the ODT feature point set and the broad field fluorescence feature point set and iteratively generates an affine transformation matrix, further comprising:
And reversely screening the ODT characteristic point set and the wide-field fluorescence characteristic point set which meet the error requirement in the iteration process.
6. The method for image registration for ODT data based on a coarse calibration system according to claim 5, wherein the inverse screening of the ODT feature point set and the broad field fluorescence feature point set meeting error requirements in the iterative process includes:
Step s1, determining an initial affine transformation matrix based on the ODT characteristic point set and the wide-field fluorescence characteristic point set, and determining a transformation ODT image based on the initial affine transformation matrix and the ODT image to be registered;
Step s2, calculating Manhattan distance of the corresponding point set after transformation based on the transformation ODT image and the wide-field fluorescent image, taking the Manhattan distance as an evaluation index, and screening out the ODT characteristic point set and the wide-field fluorescent characteristic point set with errors larger than a set threshold based on the evaluation index;
and repeating the iterative step s1 and the iterative step s2 to obtain the ODT characteristic point set, the wide-field fluorescence characteristic point set and the affine transformation matrix which meet the error requirement.
7. An image registration system for ODT data based on a coarse calibration system, comprising:
The device comprises an acquisition unit, a detection unit and a detection unit, wherein the acquisition unit is used for acquiring three-dimensional refractive index data of an ODT image to be registered and extracting single-layer two-dimensional data matched with a wide-field fluorescent image from the three-dimensional refractive index data, the ODT image to be registered and the wide-field fluorescent image are acquired by the same sample at the same time, and the ODT image to be registered and the wide-field fluorescent image have an overlapping area;
The characteristic point extraction unit is used for extracting characteristic points of the single-layer two-dimensional data and the wide-field fluorescent image respectively to obtain an ODT characteristic point set and a wide-field fluorescent characteristic point set;
the matching unit is used for matching the ODT characteristic point set and the wide-field fluorescence characteristic point set based on an adaptive iterative algorithm and generating an affine transformation matrix in an iterative mode;
the perspective transformation matrix solving unit is used for taking the affine transformation matrix as an initial solution, constructing a differential optimizer, inputting the affine transformation matrix into the differential optimizer for optimization, and obtaining a perspective transformation matrix;
and the determining unit is used for determining an image registration result of the three-dimensional refractive index data and the wide-field fluorescent image based on the perspective transformation matrix.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the coarse calibration system based image registration method for ODT data as claimed in any one of claims 1 to 6 when executing the computer program.
9. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the coarse calibration system based image registration method for ODT data according to any of claims 1 to 6.
10. A computer program product comprising a computer program which, when executed by a processor, implements the method of image registration for ODT data based on a coarse calibration system according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411792089.XA CN119991423B (en) | 2024-12-06 | 2024-12-06 | Image registration method, device and equipment for ODT data based on coarse calibration system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411792089.XA CN119991423B (en) | 2024-12-06 | 2024-12-06 | Image registration method, device and equipment for ODT data based on coarse calibration system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN119991423A CN119991423A (en) | 2025-05-13 |
| CN119991423B true CN119991423B (en) | 2025-09-05 |
Family
ID=95638940
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411792089.XA Active CN119991423B (en) | 2024-12-06 | 2024-12-06 | Image registration method, device and equipment for ODT data based on coarse calibration system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119991423B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118314175A (en) * | 2024-03-12 | 2024-07-09 | 武汉科技大学 | Unsupervised deformable three-dimensional medical image registration method and device |
| WO2024183245A1 (en) * | 2023-03-07 | 2024-09-12 | 云南电网有限责任公司玉溪供电局 | Infrared-visible light image fusion-based integrated management and control method for grid field operation |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20240233187A1 (en) * | 2021-10-18 | 2024-07-11 | Sun's Arrow Research, Inc. | Color Calibration Systems and Pipelines for Digital Images |
-
2024
- 2024-12-06 CN CN202411792089.XA patent/CN119991423B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024183245A1 (en) * | 2023-03-07 | 2024-09-12 | 云南电网有限责任公司玉溪供电局 | Infrared-visible light image fusion-based integrated management and control method for grid field operation |
| CN118314175A (en) * | 2024-03-12 | 2024-07-09 | 武汉科技大学 | Unsupervised deformable three-dimensional medical image registration method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN119991423A (en) | 2025-05-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7422825B2 (en) | Focus-weighted machine learning classifier error prediction for microscope slide images | |
| CN112308832B (en) | Bearing quality detection method based on machine vision | |
| CN109580630B (en) | Visual inspection method for defects of mechanical parts | |
| Roeder et al. | A computational image analysis glossary for biologists | |
| US10430933B2 (en) | Imaging systems and methods of using the same | |
| Ali et al. | Robust focus volume regularization in shape from focus | |
| Thomann et al. | Automatic fluorescent tag localization II: Improvement in super‐resolution by relative tracking | |
| EP3937120B1 (en) | Computer-implemented method, computer program product and system for processing images | |
| Piccinini et al. | Extended depth of focus in optical microscopy: Assessment of existing methods and a new proposal | |
| Iwasokun et al. | Image enhancement methods: a review | |
| WO2021067507A1 (en) | Building computational transfer functions on 3d light microscopy images using deep learning | |
| Fortun et al. | Reconstruction from multiple particles for 3D isotropic resolution in fluorescence microscopy | |
| CN113284158A (en) | Image edge extraction method and system based on structural constraint clustering | |
| Cao et al. | Universal high spatial resolution hyperspectral imaging using hybrid-resolution image fusion | |
| Liang et al. | Scale-invariant structure saliency selection for fast image fusion | |
| CN112258480A (en) | A weakly supervised gland instance segmentation method based on point annotation | |
| Šroubek et al. | Fusion of blurred images | |
| Song et al. | SRPS–deep-learning-based photometric stereo using superresolution images | |
| CN119991423B (en) | Image registration method, device and equipment for ODT data based on coarse calibration system | |
| Taylor | Implementing high resolution structured light by exploiting projector blur | |
| Salvi et al. | Computational Synthesis of Histological Stains: A Step Toward Virtual Enhanced Digital Pathology | |
| CN117236462A (en) | Methods for evaluating consistency of virtual processing mappings, methods for training machine learning systems having processing models, machine learning systems, computer program products and image processing systems | |
| Zhang | In situ image segmentation using the convexity of illumination distribution of the light sources | |
| Yang et al. | Structured light residual channel attention network for super-resolution enhancement<? TeX\break?> of 3D measurement images | |
| CN120278999B (en) | A wood detection method and system based on quantum image enhancement algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |