Background
The remote sensing satellite platform has the advantages of full time, high maneuverability, wide field of view and the like, however, when the illumination condition is weak, such as the condition of the morning and evening, the image effect obtained by visible light remote sensing is poor, along with the development of a low-light night vision technology, the luminous remote sensing load capable of realizing global night imaging is also possible, and the low-light night is also called night light, and is a collective term for weak visible light such as moon light, starlight, atmosphere glow and the like existing at night. The low-light night vision technology is a photoelectric technology for researching the enhancement, transmission, storage, reproduction and application of the acquired image under the condition of lower illumination, and is an important component of the modern optoelectronic technology. Due to the inherent characteristics of human eyes, when the illuminance of the environment is low, the human eyes can only observe the outline of an object, and detailed features cannot be accurately identified. The enhanced CCD/CMOS (INTENSIFIED CCD/CMOS, ICCD/ICMOS) is a solid micro-light imaging device which has wide application and can work under the condition of low illumination and is formed by coupling an image enhancer and the CCD/CMOS. Although ICMOS can image under low-light night vision condition, the image intensifier amplifies the intensity of noise at the same time of enhancing the signal, so that the random noise of the obtained image is obvious, and the noise characteristic is more complex than that of the traditional CMOS imaging.
Image denoising is a hotspot problem in computer vision field research. The existing denoising algorithm can be divided into a spatial domain denoising algorithm, a transform domain denoising algorithm, a denoising algorithm based on sparse representation and a denoising algorithm based on deep learning. The spatial domain denoising algorithm mainly aims at the characteristic that natural image noise is independent and distributed in space, noise is removed in a filtering mode, the transformation domain denoising algorithm firstly carries out specific transformation on a noise image, then processes transformation coefficients in the transformation domain according to the characteristics of the transformation domain and the properties of the noise, removes noise components and retains signal components, the sparse representation method is to carry out sparse representation on the noise image through a certain overcomplete atom library, a plurality of large signals are used for representing an original signal, sparsity is used for separating the image from the noise, and the deep learning method is used for separating the noise from an image signal by learning the distribution characteristics of the noise from a large number of noise data samples.
The low-light-level image, especially the image obtained by ICMOS detector, has larger difference with the natural image obtained under normal illumination, the signal-to-noise ratio is low, the random noise is obvious, the overall look and feel of the image is greatly reduced, the observation and recognition of human eyes are not facilitated, the multi-aperture camera has a certain multi-frame advantage, and how to use the advantage to realize the quality improvement of the low-light-level image is the problem to be solved.
Disclosure of Invention
In order to solve the technical problems in the prior art, the invention aims to provide a micro-light image denoising method for a multi-aperture camera, which can improve the cloud removal effect, reduce the cloud amount residue, reduce the cloud shielding and greatly improve the usability of remote sensing data.
In order to achieve the above object, the present invention provides a method for denoising a micro-image for a multi-aperture camera, comprising the steps of:
Step S1, acquiring N target scene images S n at the same moment by using a multi-aperture camera;
Step S2, registering the target scene image S n based on the registration matrix to obtain a registered image S' n;
s3, constructing a multi-frame micro-light image denoising network and training;
and step S4, sending the N registered images S' n obtained in the step S2 into the trained network in the step S3 for reasoning.
According to one aspect of the present invention, in the step S1, the multi-space camera is arranged in a "straight" shape.
According to one technical scheme of the invention, before the step S2, pixel dislocation among sub-apertures of the multi-aperture camera is initially corrected, five target images are registered by taking one image as a reference frame through shooting targets and adopting a registration method, and a registration matrix M j among the sub-apertures is obtained, j=1.
According to an embodiment of the present invention, in the step S2, an ORB method is adopted to obtain a registered image S n' from the target scene image S n by using the registration matrix M j.
According to one embodiment of the present invention, the step S3 specifically includes:
step S31, constructing a data set comprising micro-light noise data and true value data;
S32, constructing a multi-aperture micro-light image denoising network;
And step S33, inputting the data set into the multi-aperture micro-light image denoising network to obtain the trained multi-aperture micro-light image denoising network.
According to one embodiment of the present invention, in the step S31, the micro-noise data is formed by using images captured by a multi-aperture micro-camera laboratory and a micro-light data set together, and the truth data is obtained by overlapping continuously captured multi-frame images.
According to one technical scheme of the invention, in the step S32, the multi-aperture micro-light image denoising network comprises an input module, a feature extraction module, a fusion module and an output module;
The input module comprises N paths of input sub-modules, each path of input sub-module comprises a convolution layer and ReLu activation units, a single-channel image is mapped to a feature space with higher dimensionality, and then the single-channel image enters an N path feature extraction module to extract features of each image respectively;
The feature extraction module comprises N feature extraction sub-modules, the number of feature images of each feature extraction sub-module is gradually reduced, the feature extraction sub-modules consist of residual errors and convolution layers, the feature extraction module outputs denoised image feature images of each image respectively, and N paths of feature images enter the fusion module;
The fusion module introduces a channel attention mechanism, learns feature weight distribution among each image, generates two dimensional vectors through maximum pooling and average pooling of feature images input by each image, obtains channel attention vectors of the input feature images through linear optimization of a full-connection layer respectively, and finally multiplies the channel attention vectors with the input feature images to obtain an output feature image of the fusion module;
The output feature map is mapped to a denoised image with the same input dimension through the output module.
According to one aspect of the invention, there is provided an electronic device comprising one or more processors, one or more memories, and one or more computer programs, wherein the processors are connected to the memories, the one or more computer programs are stored in the memories, and when the electronic device is operated, the processors execute the one or more computer programs stored in the memories, so that the electronic device executes a micro-image denoising method for a multi-aperture camera according to any of the above technical solutions.
According to an aspect of the present invention, there is provided a computer readable storage medium storing computer instructions which, when executed by a processor, implement a method for denoising a microimage for a multi-aperture camera according to any one of the above-mentioned aspects.
Compared with the prior art, the invention has the following beneficial effects:
the invention provides a low-light-level image denoising method for a multi-aperture camera, which utilizes the advantages of the multi-aperture camera for simultaneously acquiring multi-frame images of the same scene/target image, acquires multi-frame image information at the same moment, extracts additional information generated by sub-pixel level dislocation, refers to a traditional multi-frame superposition denoising method, combines a deep learning technical means, sends registered multi-frame images into a constructed deep learning network, realizes the denoising of the low-light-level image, improves the signal-to-noise ratio of the image, is beneficial to further improving the imaging performance of a low-light detector, and widens the application field of the low-light-level image denoising method.
Detailed Description
The description of the embodiments of this specification should be taken in conjunction with the accompanying drawings, which are a complete description of the embodiments. In the drawings, the shape or thickness of the embodiments may be enlarged and indicated simply or conveniently. Furthermore, portions of the structures in the drawings will be described in terms of separate descriptions, and it should be noted that elements not shown or described in the drawings are in a form known to those of ordinary skill in the art.
Any references to directions and orientations in the description of the embodiments herein are for convenience only and should not be construed as limiting the scope of the invention in any way. The following description of the preferred embodiments will refer to combinations of features, which may be present alone or in combination, and the invention is not particularly limited to the preferred embodiments. The scope of the invention is defined by the claims.
As shown in fig. 1, the method for denoising a micro-light image for a multi-aperture camera of the present invention includes the following steps:
Step S1, acquiring N target scene images S n at the same moment by using a multi-aperture camera;
in the embodiment, n=5, the sub-apertures of the multi-aperture camera are arranged in a line, and as shown in fig. 3, five images are acquired S 1,S2,S3,S4,S5 respectively.
Step S2, registering the target scene image S n based on the registration matrix to obtain a registered image S' n;
Before shooting images, carrying out initial correction on pixel dislocation among sub-apertures of the multi-aperture camera, taking one image as a reference frame by adopting a registration method through shooting targets, and registering the five target images to obtain a registration matrix M j among the sub-apertures, j=1. Introducing a registration matrix M j into the step S1 by adopting a ORB (Oriented FAST and Rotated BRIEF) method, and registering the target scene image S n to obtain a registered image S n'
S3, constructing a multi-frame glimmer image denoising network and training, wherein the method specifically comprises the following steps of:
step S31, constructing a data set comprising micro-light noise data and true value data;
the micro-light noise data is formed by jointly using images shot by a multi-aperture micro-light camera laboratory and micro-light data sets, and the truth value data is obtained by overlapping continuously shot multi-frame images.
Step S32, constructing a multi-aperture micro-light image denoising network, as shown in FIG. 4;
the multi-aperture micro-light image denoising network comprises an input module, a feature extraction module, a fusion module and an output module;
The input module comprises 5 paths of input sub-modules, each path of input sub-module comprises a convolution layer and ReLu activation units, a single-channel image is mapped to a feature space with higher dimensionality, and then the single-channel image enters into the 5 paths of feature extraction modules to extract features of each image respectively;
the feature extraction module comprises 5 feature extraction sub-modules, features of each image are extracted respectively, the number of feature graphs of each feature extraction sub-module is gradually reduced, the feature extraction sub-modules consist of residual errors and convolution layers, and the mapping function of the residual errors is as follows:
RES(x)=ReLU(f(x)+x)
Wherein RES (x) represents a mapping function of the whole residual, x represents a characteristic diagram of residual input, reLU represents a nonlinear activation unit, f (x) represents an intermediate layer output result, and the residual is realized through jump connection, so that the difficulty of network training can be reduced, and the problems of gradient degradation and the like are solved. The feature image dimension of each feature extraction sub-module is 256, 128, 64 and 32 dimensions from front to back respectively, the convolution layer is a convolution kernel of 3 multiplied by 3, the feature extraction module outputs the denoised image feature image of each image respectively, and 5 paths of feature images enter the fusion module;
The fusion module introduces a channel attention mechanism, performs global maximum pooling and global average pooling of space dimensions on an input feature map to obtain two feature maps of 1 multiplied by 5C, wherein C is a single-path feature map dimension, sends the results of the global maximum pooling and the global average pooling into a multi-layer perceptron for learning respectively, finally performs fusion and activation function mapping on the results output by the multi-layer perceptron to obtain a channel attention weight matrix M c,
Mc(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))
Multiplying the weight matrix M c by the input feature map to obtain an output feature map of the fusion module;
The output feature map is mapped into a denoised image with the same input dimension through an output module;
The output feature map is mapped to a denoised image with the same input dimension through the output module.
And step S33, inputting the data set into the multi-aperture micro-light image denoising network to obtain the trained multi-aperture micro-light image denoising network.
And S4, sending the N registered images S' n obtained in the step S2 into the network trained in the step S3 for reasoning to obtain a denoised image, and denoising the multi-aperture micro-light image.
According to one aspect of the invention, there is provided an electronic device comprising one or more processors, one or more memories, and one or more computer programs, wherein the processors are connected to the memories, the one or more computer programs are stored in the memories, and when the electronic device is operated, the processors execute the one or more computer programs stored in the memories, so that the electronic device executes a micro-image denoising method for a multi-aperture camera according to any of the above technical solutions.
According to an aspect of the present invention, there is provided a computer readable storage medium storing computer instructions which, when executed by a processor, implement a method for denoising a micro-optical image for a multi-aperture camera according to any one of the above technical solutions.
Computer-readable storage media may include any medium that can store or transfer information. Examples of a computer readable storage medium include an electronic circuit, a semiconductor memory device, a ROM, a flash memory, an Erasable ROM (EROM), a floppy disk, a CD-ROM, an optical disk, a hard disk, a fiber optic medium, a Radio Frequency (RF) link, and the like. The code segments may be downloaded via computer networks such as the internet, intranets, etc.
The invention discloses a micro-light image denoising method for a multi-aperture camera, which comprises the following steps of S1, obtaining N target scene images S n at the same moment by using the multi-aperture camera, S2, registering the target scene images S n based on a registration matrix to obtain registered images S 'n, S3, constructing a multi-frame micro-light image denoising network and training, and S4, sending the N registered images S' n obtained in the S2 to the training network in the S3 for reasoning. According to the invention, the advantages of multiple frames of images of the same scene/target are simultaneously acquired by utilizing the multi-aperture camera, multiple frames of image information at the same moment are acquired, additional information generated by sub-pixel level dislocation is extracted, the traditional multiple frame superposition denoising method is used as a reference, the registered multiple frames of images are sent into a constructed deep learning network by combining a deep learning technical means, denoising of the low-light-level images is realized, the signal-to-noise ratio of the images is improved, the imaging performance of the low-light detector is further improved, and the application field of the low-light-level image is widened.
Furthermore, it should be noted that the present invention can be provided as a method, an apparatus, or a computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the invention may take the form of a computer program product on one or more computer-usable storage media having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should also be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
It is finally pointed out that the above description of the preferred embodiments of the invention, it being understood that although preferred embodiments of the invention have been described, it will be obvious to those skilled in the art that, once the basic inventive concepts of the invention are known, several modifications and adaptations can be made without departing from the principles of the invention, and these modifications and adaptations are intended to be within the scope of the invention. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the invention.