CN112508786B - Arbitrary-scale super-resolution reconstruction method and system for satellite imagery - Google Patents

Abstract

The invention relates to a satellite image-oriented arbitrary-scale super-resolution reconstruction method and system, which improve the expression capacity of a satellite image by improving a meta-amplification module and edge enhancement based on the characteristics of the satellite image, realize reconstruction and comprise low-resolution satellite image shallow layer feature extraction, adopt a residual error dense network to extract image deep layer features and fuse the low-resolution satellite image features; a weight prediction network in the element amplification module is adopted to obtain filter weights, and a more accurate weight is obtained by optimizing a projection mapping function, so that a more accurate reconstruction result of any resolution of the satellite image is obtained and is used as an intermediate reconstructed satellite image; clear expression of satellite image texture is achieved through satellite image edge enhancement, structural information of the satellite image is fully mined, and a final high-resolution satellite image is obtained. The method can improve the expression capability of satellite images with different resolutions, and the information such as edge details and the like is clearer, so that the method can be widely applied to various fields such as earth observation and the like.

Description

Arbitrary-scale super-resolution reconstruction method and system for satellite imagery

technical field

The invention belongs to the field of image super-resolution, in particular to an arbitrary-scale super-resolution reconstruction scheme oriented to satellite images.

Background technique

In recent years, satellite imagery has been widely used in urban planning, disaster monitoring, sea area monitoring, moving target monitoring and other fields of earth observation. Restricted by hardware conditions such as the power consumption of spaceborne equipment, the spatial resolution of satellite images obtained on the ground is much lower than that of natural images. The super-resolution reconstruction technology of satellite images infers and restores clearer high-resolution images from one or more frames of low-resolution images by learning the mapping relationship between low-resolution and high-resolution images. It is of great significance to enhance the spatial resolution of images by designing new technology solutions, which is of great significance to strengthen the ability to express satellite image data and expand the application scenarios of satellite images.

The existing super-resolution methods of satellite images (Document 1, Document 2) mainly use deep learning-based methods to model complex data, and introduce skip connections and dense connections to fully extract the input information, which can express more refined information. Feature texture, so as to better improve the performance of model image reconstruction. However, this type of method treats super-resolution with different scale factors as independent tasks, and only for integer scale factors (such as X2, X3, X4), each scale factor trains a specific model, which has limitations in practical applications sex.

To address image super-resolution reconstruction at arbitrary scales, an arbitrary upscaling network for super-resolution is proposed (Reference 3). The method trains a single model to solve super-resolution for arbitrary scale factors including non-integer scale factors, replacing the traditional upscaling module with a meta-upscaling module that takes the scale factor as the weights of the input dynamic prediction filter, through these weights Low-resolution features are mapped to high-resolution images of different sizes. However, due to the weak texture and low resolution characteristics of satellite images, it is difficult for this method to recover fine image content and clear edge contour information for satellite images.

Documents related to the present invention:

[1]K.Jiang,Z.Wang,P.Yi,J.Jiang,J.Xiao,and Y.Yao,"Deep distillation recursive network for remote sensing imagery super-resolution,"RemoteSensing,vol.10,no.11 , p.1700, 2018.

[2]T.Lu,J.Wang,Y.Zhang,Z.Wang,and J.Jiang,"Satellite image super-resolution via multi-scale residual deep neural network,"Remote Sensing,vol.11,no.13 , p.1588, 2019.

[3]X.Hu,H.Mu,X.Zhang,Z.Wang,T.Tan,and J.Sun,"Meta-sr:A magnification-arbitrary network for super-resolution,"in Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, 2019, pp.1575-1584.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the present invention provides an arbitrary-scale super-resolution reconstruction technology for satellite images. The method is based on two characteristics of satellite images: "weak texture" and "low resolution". Enlarge module and edge enhancement module to improve the expressiveness of satellite imagery.

The technical solution of the present invention is an arbitrary-scale super-resolution reconstruction method oriented to satellite images. Based on the characteristics of satellite images, the expression ability of satellite images is improved by improving the meta-amplification module and edge enhancement. The reconstruction implementation process is as follows:

First, extract the shallow features of the low-resolution satellite image, use the residual dense network to extract the deep features of the image, and fuse the features of the final low-resolution satellite image;

Then, the satellite image is enlarged at any scale, including using the weight prediction network in the meta-enlargement module to obtain filter weights, and by optimizing the projection mapping function to obtain more accurate weights, thereby obtaining more accurate satellite image reconstruction results at any resolution, as intermediate reconstructions satellite imagery;

Finally, the satellite image texture is clearly expressed through edge enhancement of the satellite image, and the structural information of the satellite image is fully exploited to obtain the final high-resolution satellite image.

Furthermore, the optimized projection mapping function is set as follows,

其中T(i，j)表示投影映射函数，[]表示四舍五入函数。Define the scaling factor of magnification as r, and the corresponding point (i', j') of a pixel point (i, i) on the high-resolution image on the low-resolution image is described as

where T(i, j) represents the projection mapping function, and [] represents the rounding function.

Moreover, the implementation of satellite image edge enhancement is as follows:

输出为边缘特征图I_edge；Extract the edge feature map from the intermediate reconstructed satellite image, and denote the input intermediate reconstructed satellite image as

The output is an edge feature map I _edge ;

中提取结构张量，归一化后获得图像掩膜MASK；Reconstruct the satellite image from the middle

Extract the structure tensor from , and get the image mask MASK after normalization;

The edge feature map is constrained by the image mask MASK to obtain enhanced edge information I _sharp , where I _sharp = I _edge · I _mask ;

和增强的边缘信息I_sharp进行叠加，根据

得到最终的超分辨率重建卫星图像I_SR。Intermediate results for high-resolution reconstructed images

Superimposed with enhanced edge information I _sharp , according to

The final super-resolution reconstructed satellite image I _SR is obtained.

Also, when extracting edge feature maps from intermediate reconstructed satellite images, it is preferable to use 11 convolutional layers cascaded to achieve edge extraction.

On the other hand, the present invention also provides a satellite image-oriented arbitrary-scale super-resolution reconstruction system, which is used to realize the above-mentioned satellite image-oriented arbitrary-scale super-resolution reconstruction method.

Also, the following modules are included,

The first module is used to extract the shallow features of the low-resolution satellite image, using the residual dense network to extract the deep features of the image, and fuse to obtain the features of the final low-resolution satellite image;

The second module is used for satellite image enlargement at any scale, including using the weight prediction network in the meta-enlargement module to obtain filter weights, and obtaining more accurate weights by optimizing the projection mapping function, thereby obtaining more accurate satellite image reconstruction results at any resolution , as an intermediate reconstructed satellite image;

The third module is used to realize the clear expression of satellite image texture through satellite image edge enhancement, fully excavate the structural information of satellite image, and obtain the final high-resolution satellite image.

Alternatively, it includes a processor and a memory, where the memory is used for storing program instructions, and the processor is used for calling the stored instructions in the memory to execute the above-mentioned method for satellite image-oriented super-resolution reconstruction at any scale.

Alternatively, a readable storage medium is included, and a computer program is stored on the readable storage medium, and when the computer program is executed, the above-mentioned method for super-resolution reconstruction of satellite images at any scale is implemented.

With the above technical solutions, the present invention can be used to generate satellite images of any resolution, improve the data expression capability of the satellite images, and improve the viewing experience of users of mobile terminals with different resolutions. Compared with the prior art, the present invention has the following advantages and beneficial effects:

1) Compared with the prior art, the present invention solves a new problem, that is, the problem of super-resolution reconstruction of satellite images at any scale.

2) Compared with the prior art, the present invention proposes a satellite image-oriented super-resolution reconstruction framework based on an enhanced meta-amplification module.

3) Compared with the prior art, the present invention excavates the characteristics of satellite images, and uses its characteristics to design an edge enhancement algorithm, realizes the super-resolution reconstruction technology of satellite images at any scale, strengthens the expression ability of satellite images, and can be widely used for all fields of Earth observation.

Description of drawings

FIG. 1 is a main flowchart of an embodiment of the present invention.

FIG. 2 is a network structure diagram of an embodiment of the present invention.

FIG. 3 is a schematic diagram of a residual dense block according to an embodiment of the present invention.

Detailed ways

In order to facilitate the understanding and implementation of the present invention by those of ordinary skill in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention, but not to limit it. this invention.

The invention provides an arbitrary scale super-resolution reconstruction technology oriented to satellite images. The method is based on two characteristics of satellite images: "weak texture" and "low resolution", and an improved meta-enlargement module and edge enhancement module are designed to Improve the expressiveness of satellite images. First, two convolutional layers are used to obtain the shallow features of the image, and then the features of the low-resolution image are obtained based on the residual dense network. Through the improved meta-enlargement module, the satellite image can be enlarged at any scale to obtain intermediate results. Finally, the satellite image texture is clearly expressed through the edge enhancement module, and the final high-resolution satellite image is obtained.

In this example, four NVIDIA GTX2080Ti GPUs are used to run the experiments in parallel, and the training data comes from the public satellite image dataset WHU-RS19. WHU-RS19 is a dataset of remote sensing images collected from Google Earth, the dataset covers 19 categories, including: airports, bridges, farmland, football fields, industrial areas, residential areas, rivers, etc., each category has 50 images image. The size of its HR image is 600 × 600 pixels. The examples randomly select 900 images, and use 800, 50, and 50 images as training set, validation set, and test sample, respectively. Since the performance of the super-resolution reconstruction method is related to the similarity of the test and training images, the embodiment conducts additional tests on the public satellite image dataset NWPU-RESISC45 to ensure the robustness and generalization of the present invention .

Referring to FIG. 1 , a process of a satellite image-oriented super-resolution reconstruction method at any scale provided by an embodiment of the present invention includes the following steps:

Step 1. Feature extraction of low-resolution satellite images, using residual dense network to extract features of low-resolution satellite images, including the following sub-steps:

Step 1.1. Collect public data sets of satellite images taken by different satellites, select image sequences of different scenes, and after cropping and screening, set training sets, validation sets, and test sets respectively;

Step 1.2. For the training set image in step 1.1, first use bi-trilinear interpolation to obtain a total of 30 low-resolution images of different scales from 1.1 to 4.0 (that is, the scale step size is 0.1), forming a high-resolution-low-resolution image The rate image pair is used as the input of the deep network. Similar to most existing deep learning based super-resolution reconstruction algorithms, the embodiment also uses two cascaded convolutional layers to extract the shallow feature F ¹ of the image.

Step 1.3. Use the residual dense module to extract the deep features of the image, fuse the local features and global features of the satellite image, and obtain the final feature ^FLR of the low-resolution satellite image;

Referring to Figure 3, the residual dense module in this step can be implemented by a dense connected block composed of multiple layers of convolution layers with activation functions, as well as feature fusion layers and convolution layers, such as reference [3].

Referring to FIG. 2 , in the embodiment, the feature extraction module includes two cascaded convolutional layers, multiple cascaded residual dense blocks, feature fusion layers and convolutional layers, a final convolutional layer, and multiple cascaded convolutional layers. The output of the residual dense block is input to the feature fusion layer for fusion.

Step 2. The satellite image is arbitrarily enlarged to realize the mapping of the features of the low-resolution satellite image to the high-resolution image of any resolution.

T(i，j)表示投影映射函数，

表示向下取整函数。这一操作会导致图像像素值的大量丢失，而对卫星图像来说，由于其弱纹理特性，这些丢失的像素值极为重要。为了减少像素值的丢失，本发明提出采用四舍五入函数代替向下取整函数，也即改进后的投影映射函数为

[]表示四舍五入函数。The meta-enlargement module in the prior art includes three parts: position projection, weight prediction and feature mapping. The position projection is realized by the rounding down function, and the corresponding point (i', j') of a pixel point (i, j) on the high-resolution image on the low-resolution image can be described as

T(i, j) represents the projection mapping function,

Represents a round-down function. This operation results in a large loss of image pixel values, which are extremely important for satellite images due to their weak texture properties. In order to reduce the loss of pixel values, the present invention proposes to use a rounding function to replace the rounding down function, that is, the improved projection mapping function is

[] represents a rounding function.

Referring to Figure 2, the specific implementation of this step includes the following sub-steps:

其中T(i，j)表示投影映射函数，[]表示四舍五入函数。Step 2.1. Find the projection mapping of the pixel on the high-resolution image to the corresponding pixel on the low-resolution image, and define the zoom factor as r, then a certain pixel (i, j) on the high-resolution image is in the low-resolution image. The corresponding point (i', j') on the resolution image can be described as

where T(i, j) represents the projection mapping function, and [] represents the rounding function.

Step 2.2. The weight prediction network is composed of two fully connected layers and one activation function layer to predict the filter weights from low-resolution features to high-resolution images, see Figure 2, and set the fully connected layer, activation function layer, full connection layer;

r表示比例因子，θ为权重网络的参数。The weight prediction network can be expressed as W(i,j)=ψ(V(i,j);θ), where W() represents the weight of the filter, ψ() represents the weight prediction operation, and V(i,j) is the weight prediction network input, defined as

r represents the scale factor, and θ is the parameter of the weight network.

可表示为

其中Φ表示矩阵相乘操作，因此中间重建图像可表示为

Step 2.3. Multiply the low-resolution satellite image feature F ^LR (i, j) obtained from step 1.2 and the weight obtained in step 2.2 to obtain the intermediate result of the high-resolution reconstructed image

can be expressed as

where Φ represents the matrix multiplication operation, so the intermediate reconstructed image can be expressed as

Step 3. The edge of the satellite image is enhanced, the edge features of the satellite image are extracted, and the high-frequency information in the satellite image is recovered. In the embodiment, the preferred implementation process includes the following sub-steps,

输出为边缘特征图I_edge；Step 3.1. Extract its edge feature map from the intermediate reconstructed satellite image, preferably using 11-layer convolution layer cascade to achieve edge extraction, and record the input intermediate reconstructed satellite image as

The output is an edge feature map I _edge ;

中提取结构张量，归一化后获得图像掩膜MASK。首先计算图像的结构张量

其中I_x和I_y是指图像的水平、垂直梯度，I_xy和I_yx分别表示对图像依次计算水平梯度、垂直梯度和依次计算垂直梯度、水平梯度。然后，采用矩阵的迹判断图像像素中高频信息的多少。图像掩膜MASK求取如下：Step 3.2. Reconstruct the satellite image from the middle

The structure tensor is extracted from , and the image mask MASK is obtained after normalization. First compute the structure tensor of the image

Wherein I _x and I _y refer to the horizontal and vertical gradients of the image, and I _xy and I _yx respectively represent the sequential calculation of the horizontal gradient and the vertical gradient for the image and the sequential calculation of the vertical gradient and the horizontal gradient. Then, the trace of the matrix is used to determine the amount of high-frequency information in the image pixels. The image mask MASK is calculated as follows:

其中max()、min()分别表示图像像素的最大值和最小值，(x，y)表示图像掩膜上的像素点。Among them, m and n represent the height and width of the image, respectively, S(m, n) is the structure tensor, tr(S(m, n)) is the trace of the structure tensor, and M _c is the mask of the image. In order to adjust the dynamic range of pixel values of the mask to 0-255 and maintain the relative relationship between matrix elements, the embodiment normalizes the image mask to obtain I _mask . The specific operation is

where max() and min() represent the maximum and minimum values of the image pixels, respectively, and (x, y) represent the pixels on the image mask.

Step 3.3. In order to avoid introducing noise, the embodiment uses the image mask I _mask to constrain the edge feature map to obtain enhanced edge information I _sharp , which can be expressed as the dot product of the image edge information I _edge and the image mask I _mask : I _sharp = I _edge · I _mask .

和增强的边缘信息I_sharp进行叠加，根据

得到最终的超分辨率重建卫星图像I_SR。Step 3.4. Intermediate reconstructed satellite imagery

Superimposed with enhanced edge information I _sharp , according to

The final super-resolution reconstructed satellite image I _SR is obtained.

Step 4. Evaluation of the performance of super-resolution reconstructed images of satellite images at any scale

Step 4.1. Use the validation set and test set obtained in Step 1.1 to verify and test the performance of the super-resolution reconstructed image of satellite images at any scale, and use the image peak signal-to-noise ratio (PSNR) as the objective evaluation index of image quality

Step 4.2. The comparison algorithm adopts the classical bi-triple nonlinear interpolation algorithm Bicubic and the image super-resolution arbitrary enlargement network Meta-SR. The method proposed in the present invention is called Ours.

During specific implementation, the automatic operation of the above process may be realized by software. Through experiments using the above process, it can be seen that the edge-enhanced satellite image arbitrary-scale super-resolution technology can obtain clear edge information of the image, and the effect is significantly better than the existing image arbitrary super-resolution algorithm.

Based on the results obtained by performing steps 1 to 3 of the present invention, it can be seen in Table 1 that the method of the present invention is better than other comparison algorithms under different non-integer scale factors.

Table 1. Results of arbitrary scale factors on different methods, the test dataset is WHU-RS19, and the best results are shown in bold.

It can be seen that the present invention proposes a satellite image-oriented super-resolution reconstruction framework based on an improved meta-enlargement model. Under this model, the structural features of satellite images can be explicitly expressed, and satellite image reconstruction of any resolution can be achieved.

During specific implementation, the method proposed by the technical solution of the present invention can be realized by those skilled in the art by using computer software technology to realize the automatic running process. The system device for implementing the method is, for example, a computer-readable storage medium storing a computer program corresponding to the technical solution of the present invention, and a computer that runs the corresponding computer program. The computer equipment of the program should also be within the protection scope of the present invention.

In some possible embodiments, an arbitrary-scale super-resolution reconstruction system for satellite images is provided, including the following modules:

The first module is used to extract the shallow features of the low-resolution satellite image, using the residual dense network to extract the deep features of the image, and fuse to obtain the features of the final low-resolution satellite image;

The second module is used for satellite image enlargement at any scale, including using the weight prediction network in the meta-enlargement module to obtain filter weights, and obtaining more accurate weights by optimizing the projection mapping function, thereby obtaining more accurate satellite image reconstruction results at any resolution , as an intermediate reconstructed satellite image;

The third module is used to realize the clear expression of satellite image texture through satellite image edge enhancement, fully excavate the structural information of satellite image, and obtain the final high-resolution satellite image.

In some possible embodiments, an arbitrary-scale super-resolution reconstruction system for satellite images is provided, comprising a processor and a memory, the memory is used for storing program instructions, and the processor is used for calling the stored instructions in the memory to execute the above-mentioned An arbitrary-scale super-resolution reconstruction method for satellite imagery.

In some possible embodiments, an arbitrary-scale super-resolution reconstruction system for satellite images is provided, including a readable storage medium, where a computer program is stored on the readable storage medium, and when the computer program is executed, the above-mentioned implementation is achieved Described is an arbitrary-scale super-resolution reconstruction method oriented to satellite images.

It should be understood that the parts not described in detail in this specification belong to the prior art.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which the present invention pertains can make various modifications or additions to the described specific embodiments or substitute in similar manners, but will not deviate from the spirit of the present invention or go beyond the definitions of the appended claims range.

Claims (6)

1. an arbitrary scale super-resolution reconstruction method oriented to satellite images, it is characterized in that: based on satellite image characteristics, improve the expressive power of satellite images by improving the meta-enlarging module and edge enhancement, and the reconstruction realization process is as follows, First, extract the shallow features of the low-resolution satellite image, use the residual dense network to extract the deep features of the image, and fuse the features of the final low-resolution satellite image; Then, the satellite image is enlarged at any scale, including using the weight prediction network in the meta-enlargement module to obtain filter weights, and by optimizing the projection mapping function to obtain more accurate weights, thereby obtaining more accurate satellite image reconstruction results at any resolution, as intermediate reconstructions satellite imagery; Finally, the satellite image texture is clearly expressed by the edge enhancement of the satellite image, and the structural information of the satellite image is fully excavated to obtain the final high-resolution satellite image; The optimized projection mapping function is set as follows, Define the scaling factor of magnification as r, and the corresponding point (i', j') of a pixel point (i, j) on the high-resolution image on the low-resolution image is described as

Where T(i, j) represents the projection mapping function, [] represents the rounding function; The implementation of the satellite image edge enhancement is as follows: Extract the edge feature map from the intermediate reconstructed satellite image, and denote the input intermediate reconstructed satellite image as

The output is an edge feature map I _edge ; Reconstruct the satellite image from the middle

Extract the structure tensor from , and get the image mask MASK after normalization; The edge feature map is constrained by the image mask MASK to obtain enhanced edge information I _sharp , where I _sharp = I _edge · I _mask ; Intermediate results for high-resolution reconstructed images

Superimposed with enhanced edge information I _sharp , according to

The final super-resolution reconstructed satellite image I _SR is obtained. 2 . The satellite image-oriented super-resolution reconstruction method according to claim 1 , wherein: when extracting the edge feature map from the intermediate reconstructed satellite image, it is preferable to use 11 layers of convolution layers cascade to realize edge extraction. 3 . 3 . An arbitrary-scale super-resolution reconstruction system oriented to satellite images, characterized in that: it is used to realize the arbitrary-scale super-resolution reconstruction method oriented to satellite images according to any one of claims 1-2. 4 . 4. The arbitrary scale super-resolution reconstruction system oriented to satellite images according to claim 3, is characterized in that: comprising the following modules, The first module is used to extract the shallow features of the low-resolution satellite image, using the residual dense network to extract the deep features of the image, and fuse to obtain the features of the final low-resolution satellite image; The second module is used for satellite image enlargement at any scale, including using the weight prediction network in the meta-enlargement module to obtain filter weights, and obtaining more accurate weights by optimizing the projection mapping function, thereby obtaining more accurate satellite image reconstruction results at any resolution , as an intermediate reconstructed satellite image; The third module is used to realize the clear expression of satellite image texture through satellite image edge enhancement, fully excavate the structural information of satellite image, and obtain the final high-resolution satellite image. 5. The arbitrary scale super-resolution reconstruction system oriented to satellite images according to claim 3, is characterized in that: comprising a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute as claimed in the claims A satellite image-oriented arbitrary-scale super-resolution reconstruction method according to any one of 1-2. 6. The arbitrary-scale super-resolution reconstruction system oriented to satellite images according to claim 3, characterized in that: comprising a readable storage medium on which a computer program is stored, and when the computer program is executed, realizes A satellite image-oriented arbitrary-scale super-resolution reconstruction method according to any one of claims 1-2.

Images