CN116579958B

CN116579958B - Multi-focus image fusion method of depth neural network guided by regional difference priori

Info

Publication number: CN116579958B
Application number: CN202310233028.9A
Authority: CN
Inventors: 肖斌; 房嘉敏
Original assignee: Chongqing University of Post and Telecommunications
Current assignee: Chongqing University of Post and Telecommunications
Priority date: 2023-03-10
Filing date: 2023-03-10
Publication date: 2025-05-30
Anticipated expiration: 2043-03-10
Also published as: CN116579958A

Abstract

The invention discloses a multi-focus image fusion method of a region difference priori guided deep neural network, and relates to the technical fields of digital image processing, computer vision, deep learning and the like. The method comprises the specific steps of 1) manufacturing a disclosed multi-focus image data set, 2) preprocessing the disclosed multi-focus image data set, including technologies such as image denoising, image enhancement and image registration, 3) obtaining region difference priori information by utilizing morphological operation-expansion and corrosion enhancement of differences between pairs of multi-focus images, 4) designing a depth neural network guided by the region difference priori, and 5) fusing the multi-focus images in a test set by utilizing a model obtained through training to obtain a final fusion result. The method utilizes the proposed region difference priori, and combines the model obtained by the prior deep neural network training to improve the accuracy of focusing measurement and obtain a fusion image with higher quality.

Description

Multi-focus image fusion method of depth neural network guided by regional difference priori

Technical Field

The invention relates to a multi-focus image fusion method of a region difference priori guided deep neural network, belonging to the technical fields of digital image processing, computer vision, deep learning and the like.

Background

Because the existing optical camera system is limited by the depth of field of the lens, the image obtained when shooting the same scene is usually an image with clear partial area and blurred or defocused partial area, and an ideal full-focus image is difficult to generate. However, this can present a significant challenge for tasks requiring accurate analysis of the entire scene, such as robot vision, medical imaging, and the like. And many other popular computer vision tasks often require the full focus image to be further processed, such as detection and segmentation. In view of the above problems, a current common approach is a multi-focus image fusion algorithm. With the rapid development of deep learning and fusion techniques, the field of multi-focus image fusion is continually explored and developed with new methods and techniques. So far, research into multi-focus image fusion has been continued for more than 30 years, during which numerous algorithms have been published. These algorithms can be broadly divided into two main categories, reconstruction policy-based methods and decision policy-based methods, based on the differences in fusion policies employed in the fusion process. The reconstruction strategy based method generally consists of three steps, image decomposition, decomposition coefficient fusion and reconstruction. Such algorithms treat defocused images in different focused scenes as degraded images of an ideal fully focused image. In this mode, multi-focus image fusion actually becomes an image enhancement task to obtain a fully focused image from a degraded image. Similar to the additional enhancement task, such reconstruction strategy-based methods inevitably generate pixel defects, such as luminance and color distortions, in the generated fusion image. Thus, decision strategy-based approaches have begun to prevail.

Unlike the reconstruction strategy-based approach, decision strategy-based work is mainly focused on how to generate an accurate decision graph. And then selecting focusing parts from the defocused images through the decision graph to combine to form a full-focusing image. In such a method, a Focus Measurement (FM) step is used to obtain a decision map, and Focus measurement can be accomplished by manual design as well as deep learning. Depending on how focus measurements are implemented, decision strategy based methods can continue to be classified into methods based on manually designed FM and methods based on learnable FM. Methods based on manually designed FM are popular, but such methods have difficulty in handling complex scenes and limit applicable space, greatly limiting the performance of multi-focus image fusion.

With the continuous development of deep learning technology, the method based on the learning FM mainly uses a convolutional neural network as a main framework to develop research. In such methods, the deep neural network is considered a tool for implementing FM, and can learn a mapping from defocused images to decision graphs directly. The method based on the learnable FM can automatically learn a more accurate decision chart, which is difficult to achieve by the method based on the manual design FM. Because CNN possesses strong learning ability, the performance of multi-focus image fusion task is further improved compared with the method based on artificial design FM. However, although current methods of learning FM achieve some degree of boosting, decision graphs generated in such methods still suffer from the phenomenon of holes. Therefore, how to effectively improve the accuracy of focusing measurement and effectively improve the hole phenomenon in the decision diagram is more and more important.

CN113313663A, a multi-focus image fusion method based on zero sample learning, a multi-focus image fusion network structure IM-Net is used for fusing information contained in an input multi-focus image, the IM-Net comprises two combined sub-networks I-Net and M-Net, the I-Net models the depth priori of the fusion image, the M-Net models the depth priori of the focus image, zero sample learning is realized through extracted priori information, reconstruction constraint is applied to the IM-Net, so that information of a source image pair can be transmitted to the fusion image better, high-layer semantic information can keep brightness consistency of adjacent pixels, guiding loss provides guiding information for finding a clear area for the IM-Net, and experimental results indicate the effectiveness of the method.

The method in this patent still achieves their objective by designing a complex network structure, but the improvement of the fusion effect is limited and increases the computational burden. The invention considers how to improve the fusion efficiency from a brand new angle, namely, the difference between the focusing part and the defocusing part in the defocusing image pair is strengthened to provide the region difference priori, so that a simple neural network is designed in a priori, and a good fusion effect can be achieved. Further, the use of region difference priors in neural networks is simple and efficient and has versatility.

Disclosure of Invention

The invention aims to solve the problems existing in the existing learning-based FM method. A multi-focus image fusion method of a depth neural network guided by regional difference priori is provided. The technical scheme of the invention is as follows:

A multi-focus image fusion method of a region difference prior guided deep neural network, comprising the following steps:

(1) Collecting original image samples and making a multi-focus image training set for training;

(2) Performing image preprocessing operations including image denoising, image enhancement and image registration on the multi-focus image to realize data enhancement;

(3) Obtaining regional difference prior information using morphological operations, i.e., differences between pairs of multi-focused images of dilation and erosion enhancement;

(4) Designing a depth neural network guided by the regional difference priori based on the regional difference priori information obtained in the step (3), and performing model training;

(5) And (3) testing the multi-focus images in the test set by using the model obtained by training in the step (4) to obtain a final fusion result.

Further, the step (2) performs image preprocessing operations including image denoising, image enhancement and image registration on the multi-focus image to enhance data, and specifically includes:

image denoising is specifically to remove image noise by using certain artificially designed low-pass filters, such as median filtering and wiener filtering;

the image enhancement specifically comprises the steps of directly carrying out various linear or nonlinear operations on the image, and carrying out enhancement treatment on the pixel gray value of the image;

The image registration method comprises the steps of extracting features of two images to obtain feature points, finding matched feature point pairs through similarity measurement, obtaining image space coordinate transformation parameters through the matched feature point pairs, and finally carrying out image registration through the coordinate transformation parameters.

Further, the step (3) of obtaining the region difference priori information by using the morphological operation, that is, the difference between the expansion and corrosion enhancement paired multi-focus images, specifically includes:

the expansion of a structural element SE at a position (x, y) of an image f is defined as follows:

wherein SE is a structural element, f is a gray scale image; s, t are moving steps;

the erosion of an image f at the location (x, y) by the structural element SE is defined as follows:

Is a corrosion operation;

Given a pair of grayscale images I _a and I _b, performing a global expansion operation on I _a and I _b results in a pair of images D _a and D _b, which are defined as follows:

Wherein dilate is a global expansion operation, since the expansion of the gray image by SE at any position (x, y) is defined as the maximum of the overlapping area, whereby the gray values of the pixels of images D _a and D _b will rise, subtracting I from D, while the variation of the in-focus and out-of-focus portions defined between D and I is:

wherein D _focus、I_focus represents the focusing part in D and I;

Δ _focus↑、Δ_defocus↑ represents the amount of change in the focusing and defocusing portions between D and I, respectively;

D _defocus、I_defocus represents defocused portions in D and I, and likewise, by performing a global erosion operation on images I _a and I _b, a pair of images E _a and E _b are obtained, which can be described as:

Where erode is a global erosion operation, the erosion of the gray image by SE at any location (x, y) is defined as the minimum of the overlapping area, so the gray values of the pixels in images E _a and E _b will drop, subtracting E from I, and defining the variation of the in-focus and out-of-focus portions between I and E, which can be described as:

Considering the purpose of region difference reinforcement, i.e. both Δ _focus↑Δ_defocus↑ and Δ _focus↓Δ_defocus↓ are valid, the best way to obtain region difference priors is given by:

dilate(I_a),erode(I_a);dilate(I_b),erode(I_b)

Wherein I _a,dilate(I_a) represents that I _a and dilate (I _a) are connected together in the channel dimension.

Further, in the step (4), a deep neural network guided by a regional difference priori is designed and model training is performed, and the training process is as follows:

Where p is the generated focus map, For the reference image to be a reference image,

I represents the i-th batch;

j represents the j-th channel;

a reference image representing the jth channel of the ith lot;

A focus map representing the jth channel of the ith lot;

N represents the batch size, C represents the number of channels, the output of a two-channel focusing graph p is used as a neural network by training, each output value is the focusing score of a corresponding pixel in a pair of source images, then an initial decision graph is generated by comparing the focusing scores of two channels in p, the initial decision graph is perfected by adopting a fully connected conditional random field CRF, and then a final decision graph W is obtained.

Further, the method comprises the steps of generating an initial decision graph T by comparing the focus scores of two channels in p, perfecting the initial decision graph by adopting a fully connected conditional random field CRF, and then obtaining a final decision graph W, wherein the method specifically comprises the following steps:

obtaining an initial decision diagram;

Each pixel i is provided with a class label x _i and a corresponding observed value y _i, so that each pixel point is used as a node, the relation between pixels is used as an edge, namely, a conditional random field is formed, the class label x _i corresponding to the pixel i is estimated through the observed variable y _i, and the initial decision diagram can be perfected through the technology.

Further, in the step (5), final image fusion is performed, which is expressed as follows:

F_fusion(x,y)=A(x,y)W(x,y)+B(x,y)(1-W(x,y))

Wherein A and B are source images, W is a final decision graph, pixel multiplication is performed, and F _fusion is a final fusion result.

The invention has the advantages and beneficial effects as follows:

The multi-focus image fusion method of the depth neural network guided by the regional difference priori provided by the invention is an effective method which is generated based on the regional difference priori information and can effectively improve the focus measurement precision. The method improves the problems existing in the prior learning-based FM method, can effectively fuse to obtain high-quality fused images when facing various fused scenes at the same time, and can be applied to other neural network models by regional difference priori, thereby having universality.

The invention realizes the multi-focus image fusion task by utilizing the technologies of digital image processing, computer vision, deep learning and the like. The invention relates to a method based on regional difference priori information, which utilizes the difference between defocused images of morphological operation expansion and corrosion reinforcement pairs to obtain regional difference priori, and designs a novel deep neural network to generate a better decision diagram so as to obtain a final fusion result. The invention has the following advantages:

(1) The pytorch platform is utilized for training and testing, so that the efficiency is high;

(2) The method is a method for improving the focusing measurement precision based on the region difference priori, and can achieve the purpose of strengthening the region difference only by using simple morphological operation, so as to obtain the region difference priori. The use of the region difference priori information is simple and efficient;

(3) The regional difference priori obtained by the invention can effectively improve the quality of the fusion image obtained under various fusion scenes;

(4) The invention has good lifting effect on other neural network methods;

(5) Aiming at the difficult-to-determine area in the multi-focus image, the area difference priori can bring about the improvement of the precision of focus measurement to a certain extent;

(6) The method can assist related image detection and image segmentation work, has practical significance and achieves better effects.

The innovation of the invention is mainly the steps of claims 3, 4.

Taking into account the differences between the enhanced defocused image versus the focused and defocused portions to obtain region difference prior information is a significant innovation, from which other approaches have never seen multi-focused image fusion.

The method has the advantage that the acquired region difference priori is due to the fact that the sensitivity degree of the focusing part and the defocusing part to morphological operation, namely expansion and corrosion, is completely different, namely the focusing part has a more intense response to the expansion and corrosion operation, so that after the operation is applied, the focusing part and the defocusing part can generate completely different changes, the purpose of difference strengthening is achieved, and the method meets the aim of us. Meanwhile, the region difference priori provided by the invention has universality and can be used in other multi-focus image fusion methods to improve the focus measurement precision of the multi-focus image fusion methods. For any different fusion scene, the quality of the fusion image obtained by using the region difference priori is very high, and the use of the region difference priori is simple and efficient.

Drawings

FIG. 1 is a system flow diagram of a preferred embodiment of the present invention;

fig. 2 (a 1) to (a 6) are original images and images after the region difference enhancing operation;

FIGS. 2 (b 1) - (b 4) are decision graphs obtained in a scene without regional differential prior participation and decision graphs obtained in a scene with regional differential prior participation;

fig. 2 (b 5) - (b 6) are examples of two final fused images.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and specifically described below with reference to the drawings in the embodiments of the present invention. The described embodiments are only a few embodiments of the present invention.

The technical scheme for solving the technical problems is as follows:

as shown in fig. 1, a multi-focus image fusion method of a region difference prior guided deep neural network includes the following steps:

Collecting original image samples and manufacturing a multi-focus image training set for training;

secondly, the common image preprocessing operation is used for the manufactured multi-focus image training set to realize data enhancement;

The third step, the difference between the multi-focus images in the pair of expansion and corrosion enhancement is utilized to obtain the prior information of the regional difference, the step comprises the steps of carrying out expansion operation and corrosion operation on the multi-focus images and effectively combining the multi-focus images to realize the purpose of regional difference enhancement, and the specific steps are as follows:

Since the focusing and defocusing portions are sensitive to different degrees for the expansion and corrosion operations, i.e. the focusing portion is more sensitive to the morphological operations, the focusing and defocusing portions may change significantly when the morphological operations are performed on the two portions. Image processing of an image by dilation and erosion operations relies on a Structural Element (SE) which is defined as dilation of an image f at locations (x, y) as follows:

wherein SE is a structural element, f is a gray scale image; Is an expansion operation.

wherein SE is a structural element, f is a gray scale image; is an etching operation.

Wherein dilate is a global expansion operation. The expansion of the gray image due to SE at any position (x, y) is defined as the maximum value of the overlapping area. Thus, the gray values of the pixels of the images D _a and D _b rise. Subtracting I from D, while defining the variation of the in-focus and out-of-focus portions between D and I as:

where D _focus、I_focus represents the focused portions in D and I, and D _defocus、I_defocus represents the defocused portions in D and I. Likewise, by performing a global erosion operation on images I _a and I _b, a pair of images E _a and E _b is obtained. It can be described as:

Wherein erode is a global etching operation. The erosion of the gray scale image by SE at any location (x, y) is defined as the minimum of overlapping areas. Thus, the gray values of the pixels in the images E _a and E _b will decrease. Subtracting E from I and defining the variation of the in-focus and out-of-focus parts between I and E, it can be described as:

It can be noted that the existing learning FM-based methods are all designed with complex network structures to achieve the purpose of improving the focus measurement accuracy, and can achieve the effect of improvement to a certain extent but still have the hole phenomenon, and additionally increase the calculation burden. Therefore, the invention provides the region difference priori as the guiding priori of the deep neural network, and considers the purpose of region difference reinforcement, namely delta _focus↑Δ_defocus↑ and delta _focus↓Δ_defocus↓ can be effectively established, and the best mode for obtaining the region difference priori is provided as follows:

dilate(I_a),erode(I_a);dilate(I_b),erode(I_b)

wherein I _a,dilate(I_a) represents that I _a and dilate (I _a) are connected together in the channel dimension. The purpose of the above formula is to significantly improve the accuracy of focus measurement by maximizing the difference between the focused and defocused portions.

Designing a depth neural network guided by the regional difference priori for the regional difference priori obtained in the step (3), and performing model training, wherein the training process is as follows:

Where p is the generated focus map, For reference pictures, N represents the batch size and C represents the number of channels. By effectively training, a two-channel focus map p is used as the output of the neural network, where each output value is the focus score of a corresponding pixel in the paired source images. An initial decision map is then generated by comparing the focus scores of the two channels in p. Considering that there may be some imperfections in the initial decision diagram, a fully connected Conditional Random Field (CRF) is used to refine the initial decision diagram, and then the final decision diagram W is obtained.

And fifthly, performing fusion test on the input multi-focus images by using the model trained in the step (4) to obtain a final fusion result.

The experimental method comprises the following steps:

in the experimental process, we made a disclosed multi-focus image dataset, and trained the neural network with 90% of it as the training set, and another 10% as the validation set to verify the fusion quality of the proposed region difference a priori guided multi-focus image fusion method of the deep neural network.

The first step is to perform data enhancement operation on the manufactured multi-focus image training set by utilizing image preprocessing operation, wherein the data enhancement operation comprises image denoising, image enhancement and image registration.

And secondly, running a python program, inputting the training set picture and the corresponding label into a deep neural network, and obtaining a final trained model after tuning the training parameters.

And thirdly, testing the images in the test set by using the trained model and calculating the quality index of the fused image.

Experiments prove that the method provided by the invention can effectively improve the precision of focusing measurement after training, and the quality of the fused image of the multi-focusing image under various fused scenes is effectively improved.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.

The above examples should be understood as illustrative only and not limiting the scope of the invention. Various changes and modifications to the present invention may be made by one skilled in the art after reading the teachings herein, and such equivalent changes and modifications are intended to fall within the scope of the invention as defined in the appended claims.

Claims

1. A multi-focus image fusion method of a deep neural network guided by regional difference priors, characterized in that it includes the following steps:

(1) Collect original image samples and create a multi-focus image training set for training;

(2) Perform image preprocessing operations including image denoising, image enhancement, and image registration on multi-focus images to achieve data enhancement;

(3) Using morphological operations, namely dilation and erosion, the differences between paired multi-focus images are enhanced to obtain regional difference prior information. Given a pair of grayscale images _Ia and _Ib , a global dilation operation is performed on _Ia and _Ib to obtain a pair of images _Da and _Db , which are defined as follows:

Where dilate is a global dilation operation. Since the dilation of the grayscale image by SE at any position (x, y) is defined as the maximum value of the overlapping area, the grayscale values of the pixels of images _Da and _Db will increase. Subtract I from D, and define the changes in the focused and defocused parts between D and I as:

Where D _focus and I _focus represent the focused parts of D and I;

Δ _focus↑ and Δ _defocus↑ represent the changes in the focused and defocused parts between D and I, respectively;

D _defocus and I _defocus represent the defocused parts in D and I; similarly, by performing a global erosion operation on images I _a and I _b , a pair of images E _a and E _b are obtained, which can be described as:

Where erode is a global erosion operation, and the erosion of the grayscale image by SE at any position (x, y) is defined as the minimum value of the overlapping area; therefore, the grayscale value of the pixels in images _Ea and _Eb will decrease; subtract E from I, and define the change in the focused and defocused parts between I and E, which can be described as:

Considering the purpose of regional difference enhancement, that is, Δ _focus↑ >> Δ _defocus↑ and Δ _focus↓ >> Δ _defocus↓ are both valid, the best way to obtain regional difference priors is given as follows:

dilate(I _a ),erode(I _a ); dilate(I _b ),erode(I _b )

Among them, erode(I _a ), dilate(I _a ) represents connecting erode(I _a ) and dilate(I _a ) together in the channel dimension;

(4) Based on the regional difference prior information obtained in step (3), a deep neural network guided by regional difference prior is designed and model training is performed;

(5) Use the model trained in step (4) to test the multi-focus images in the test set to obtain the final fusion result.

2. The multi-focus image fusion method of the deep neural network guided by regional difference prior according to claim 1 is characterized in that the step (2) performs image preprocessing operations including image denoising, image enhancement, and image registration on the multi-focus image to achieve data enhancement, specifically comprising:

Image denoising specifically involves: using some artificially designed low-pass filters, such as median filtering and Wiener filtering, to remove image noise;

Image enhancement specifically includes: performing various linear or nonlinear operations directly on the image and enhancing the pixel grayscale value of the image;

Image registration is specifically as follows: extract features from two images to obtain feature points; find matching feature point pairs by performing similarity measurement; then obtain image space coordinate transformation parameters through the matching feature point pairs; and finally perform image registration based on the coordinate transformation parameters.

3. The multi-focus image fusion method of deep neural network guided by regional difference prior according to claim 1 is characterized in that the step (3) uses morphological operations, i.e., dilation and erosion, to strengthen the difference between paired multi-focus images to obtain regional difference prior information, specifically comprising:

The structural element SE is the expansion of an image f at position (x, y), which is defined as follows:

Where SE is the structural element; f is the grayscale image; is the expansion operation; s, t are the moving steps;

The structural element SE erodes an image f at position (x, y) and is defined as follows:

For corrosion operation.

4. The multi-focus image fusion method of the deep neural network guided by regional difference prior according to claim 1 is characterized in that in the step (4), a deep neural network guided by regional difference prior is designed and model training is performed, and the training process is as follows:

Where p is the generated focus map, is the reference image,

i represents the i-th batch;

j represents the jth channel;

represents the reference image of the jth channel of the i-th batch;

Represents the focus map of the jth channel of the i-th batch;

N represents the batch size; C represents the number of channels. A two-channel focus map p is trained as the output of the neural network, where each output value is the focus score of the corresponding pixel in the paired source image; then, an initial decision map is generated by comparing the focus scores of the two channels in p; the fully connected conditional random field CRF is used to improve the initial decision map, and then the final decision map W is obtained.

5. The multi-focus image fusion method of the deep neural network guided by regional difference prior according to claim 4 is characterized in that an initial decision graph T is generated by comparing the focus scores of the two channels in p; the fully connected conditional random field CRF is used to improve the initial decision graph, and then the final decision graph W is obtained, which specifically includes:

Get the initial decision graph;

For each pixel i, there is a category label _xi and a corresponding observation value _yi . In this way, each pixel point is used as a node, and the relationship between pixels is used as an edge, which constitutes a conditional random field. The category label _xi corresponding to pixel i is inferred through the observed variable _yi . Through the above technology, the initial decision graph can be improved.

6. The multi-focus image fusion method of deep neural network guided by regional difference prior according to claim 5, characterized in that the final image fusion is performed in step (5), which is expressed as follows:

F _fusion (x,y)＝A(x,y)⊙W(x,y)+B(x,y)⊙(1-W(x,y))

Where A and B are source images, W is the final decision map, ⊙ is the pixel multiplication, and F _fusion is the final fusion result.