[go: up one dir, main page]

CN118469818B - Image super-resolution reconstruction method, electronic device and storage medium based on FSRCNN and OPE - Google Patents

Image super-resolution reconstruction method, electronic device and storage medium based on FSRCNN and OPE Download PDF

Info

Publication number
CN118469818B
CN118469818B CN202410671523.2A CN202410671523A CN118469818B CN 118469818 B CN118469818 B CN 118469818B CN 202410671523 A CN202410671523 A CN 202410671523A CN 118469818 B CN118469818 B CN 118469818B
Authority
CN
China
Prior art keywords
image
resolution
super
ope
fsrcnn
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202410671523.2A
Other languages
Chinese (zh)
Other versions
CN118469818A (en
Inventor
刘晓东
郭旬涛
占梓源
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Harbin Institute of Technology Shenzhen
Original Assignee
Harbin Institute of Technology Shenzhen
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Harbin Institute of Technology Shenzhen filed Critical Harbin Institute of Technology Shenzhen
Priority to CN202410671523.2A priority Critical patent/CN118469818B/en
Publication of CN118469818A publication Critical patent/CN118469818A/en
Application granted granted Critical
Publication of CN118469818B publication Critical patent/CN118469818B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

一种基于FSRCNN和OPE的图像超分辨率重建方法、电子设备及存储介质，属于图像超分辨率重建技术领域。为解决图像超分辨率算法在不同上采样率下独立训练模型的问题，本发明将数据集中的低分辨率图像输入到FSRCNN网络中，通过卷积层提取图像的特征，输出特征映射输入到OPE模块中，对特征映射进行编码，得到正交位置编码，然后利用得到的正交位置编码对特征映射进行上采样处理，通过OPE模块的线性组合，计算超分辨率图像SR的像素值，生成初步的SR图像进行后处理，得到超分辨率处理后的图像；进行性能评估并计算损失函数，根据损失函数的结果，通过反向传播调整FSRCNN网络结构和OPE模块的参数。本发明节省训练时间。

A method, electronic device and storage medium for image super-resolution reconstruction based on FSRCNN and OPE, belonging to the technical field of image super-resolution reconstruction. In order to solve the problem of independent training models of image super-resolution algorithms at different upsampling rates, the present invention inputs low-resolution images in a data set into an FSRCNN network, extracts features of the image through a convolutional layer, outputs feature maps and inputs them into an OPE module, encodes the feature maps to obtain orthogonal position codes, and then uses the obtained orthogonal position codes to upsample the feature maps, calculates pixel values of super-resolution images SR through linear combinations of OPE modules, generates preliminary SR images for post-processing, and obtains images after super-resolution processing; performs performance evaluation and calculates loss functions, and adjusts the FSRCNN network structure and parameters of the OPE module through back propagation according to the results of the loss function. The present invention saves training time.

Description

Image super-resolution reconstruction method based on FSRCNN and OPE, electronic equipment and storage medium
Technical Field
The invention belongs to the technical field of image super-resolution reconstruction, and particularly relates to an image super-resolution reconstruction method based on FSRCNN and OPE, electronic equipment and a storage medium.
Background
Image resolution is an important indicator for measuring image detail and definition, and high-resolution images can provide more information. In military, medical, industrial and everyday life, there is an increasing demand for high resolution images. However, limited by the technology and conditions, the actual acquired images often have blurring and noise problems, and it is difficult to achieve the desired resolution. The traditional sensor technology method is high in cost and lacks flexibility, is only suitable for capturing new images, and cannot improve the resolution of the existing low-resolution images. Therefore, it is important to explore other methods for improving the resolution of images. The image super-resolution technology can recover a high-resolution image from a low-resolution image, improves the image definition, reduces the blurring and noise, and is widely applied to the fields of high-definition televisions, medical imaging, satellite imaging, monitoring, biological information identification and the like.
The technique utilizes image processing techniques to mine hidden information to generate high resolution images. At present, the technology is mainly divided into three types, namely an interpolation-based method, a reconstruction-based method and a depth learning-based method. Interpolation-based methods infer unknown pixel values from known pixel values, including nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. These methods are simple and fast, but have limitations in high frequency detail recovery. The reconstruction-based method takes image priori knowledge as constraint, and improves the linear model performance and the edge blurring problem through iterative back projection, a maximum posterior probability method, a convex set projection method and the like. The method based on the deep learning realizes the super-resolution reconstruction of the image by learning the mapping relation between the low-resolution image and the high-resolution image and adopting the methods of neighborhood embedding, sparse representation, anchoring neighbor regression and deep learning. The deep learning method, such as SRCNN, learns image features through a convolutional neural network to realize efficient super-resolution reconstruction, but faces challenges such as model size, training time and memory consumption, and the existing image super-resolution algorithm based on deep learning generally needs to train an independent model aiming at a specific up-sampling rate, which limits the universality of the algorithm in image processing of different scales.
Disclosure of Invention
The invention aims to solve the problem that an existing image super-resolution algorithm needs to independently train a model under different upsampling rates, and provides an image super-resolution reconstruction method, electronic equipment and a storage medium based on FSRCNN and OPE.
In order to achieve the above purpose, the present invention is realized by the following technical scheme:
A FSRCNN and OPE-based image super-resolution reconstruction method comprises the following steps:
S1, acquiring a high-resolution image or acquiring a high-resolution image based on a DIV2K Dataset open source data set, preprocessing to acquire a high-resolution image HR and a corresponding low-resolution image LR, and constructing a data set for training based on FSRCNN and OPE image super-resolution reconstruction methods;
s2, constructing FSRCNN network structures and OPE modules;
S3, inputting the low-resolution image in the dataset obtained in the step S1 into a FSRCNN network, extracting the features of the image through a convolution layer, and outputting a feature map F;
S4, inputting the feature map obtained in the step S3 into an OPE module, coding the feature map to obtain an orthogonal position code, then carrying out up-sampling processing on the feature map by utilizing the obtained orthogonal position code, and calculating a pixel value of a super-resolution image SR through linear combination of the OPE module to generate a preliminary SR image;
S5, performing post-processing on the preliminary SR image obtained in the step S4 to obtain an image subjected to super-resolution processing;
s6, performing performance evaluation on the super-resolution processed image obtained in the step S5, calculating a loss function, adjusting FSRCNN parameters of a network structure and an OPE module through back propagation according to the result of the loss function, performing iterative training by using a data set, and outputting a reconstruction result from an LR image to an SR image.
Further, the specific implementation method of the step S1 includes the following steps:
s1.1, downsampling a high-resolution image by a bicubic interpolation method to obtain a low-resolution image LR, adjusting the reduction ratio of the bicubic interpolation method to obtain a low-resolution image LR with three ratios of 1/2, 1/4 and 1/8, and respectively placing the HR and the LR under different folders;
S1.2, carrying out normalization processing on the HR and LR obtained in the step S1.1, wherein the pixel value range is [0,1], and obtaining a dataset used for training an image super-resolution reconstruction method based on FSRCNN and OPE.
Further, the specific implementation method of the step S2 includes the following steps:
S2.1, constructing FSRCNN a network structure, wherein the first layer is a 3x3 convolution layer, 32 convolution kernels are used, a ReLU activation function is connected to the first layer, the second layer is a 3x3 maximum pooling layer, the third layer is a 3x3 convolution layer, 64 convolution kernels are used, a ReLU activation function is connected to the second layer, and the fourth layer is a 1x1 convolution layer used for compressing feature mapping;
S2.2, constructing an OPE module, namely selecting sine and cosine functions as orthogonal bases, setting maximum frequency and coding length, selecting Adam as an optimizer for model training, and setting initial learning rate lr to be 0.001.
Further, the expression of the feature of the extracted image in step S3 is:
Where x m,n is the pixel value in the input image, B i,j is the weight of the convolution kernel, f is the ReLU activation function, y i,j is the pixel value of the output feature image, M×N is the size of the convolution kernel, wherein M and N represent the position coordinates of the pixels in the image, M is the coordinates in the horizontal direction, N is the coordinates in the vertical direction, M is the width of the image, and N is the height of the image;
The output of FSRCNN network is the feature map F, the dimension of F is h×w×c, where H is the height of feature map, W is the width of feature map, and C is the number of channels of feature map.
Further, the specific implementation method of the step S4 includes the following steps:
s4.1, defining a group of orthogonal basis functions, wherein the group of orthogonal basis functions comprises an orthogonal basis function in the horizontal direction and an orthogonal basis function in the vertical direction, and the expression is as follows:
Wherein phi p (x) is a cosine base function in the horizontal direction, phi q (y) is a cosine base function in the vertical direction, phi p (x) is a sine base function in the horizontal direction, phi q (y) is a sine base function in the vertical direction, p and q are frequency indexes of orthogonal base functions respectively, and x and y are positions of feature maps in the horizontal direction and the vertical direction respectively;
S4.2, for each pair of orthogonal basis functions, calculating projection coefficients of characteristic mapping on the orthogonal basis functions, wherein the expression is as follows:
Wherein F m,n,c is the pixel value of the feature map at positions m, n and channel c, phi p (m) and phi q (n) are the values of the sine base function at positions m, n, respectively, and phi p (m) and phi q (n) are the values of the cosine base function at positions m, n, respectively;
s4.3, carrying out up-sampling processing on the feature mapping by utilizing the projection coefficient of the obtained feature mapping on the orthogonal basis function, calculating the pixel value of the super-resolution image SR through the linear combination of the OPE module, generating a preliminary SR image, and calculating the expression as follows:
Where I i,j is the pixel value of the high-resolution image at position I, j, K and L are the number of orthogonal basis functions in the horizontal and vertical directions, and P p,q is the coefficient of the P-th and q-th basis functions in the encoded feature map.
Further, step S5 performs post-processing on the preliminary SR image, denoising by using a non-local mean technique, performing high-pass filtering by using a Sobel operator, and finally obtaining the super-resolution processed image through histogram equalization.
Further, the specific implementation method of the step S6 includes the following steps:
S6.1, performance evaluation, namely performing performance evaluation on the reconstructed super-resolution image by using peak signal-to-noise ratio (PSNR) and Structural Similarity (SSIM) indexes, wherein the PSNR has a calculation expression as follows:
Wherein MAX I is the maximum pixel value of the image, MSE is the mean square error, and the calculation formula of MSE is:
Wherein, I HR is the original high-resolution image, I SR is the image processed by super resolution, M and N are the number of rows and columns of the image respectively;
The calculated expression for SSIM is:
Wherein: And The average of the original high resolution image and the super resolution processed image,AndThe variances of the original high resolution image and the super resolution processed image respectively,Is the covariance of the original high-resolution image and the super-resolution processed image, C 1 and C 2 are constants for stabilizing the denominator of SSIM;
s6.2, setting MSE and SSIM to jointly construct a loss function of model training, wherein the expression is as follows:
Loss=α·MSE(I,K)+(1-α)·(1-SSIM(I,K))
Wherein alpha is a weight parameter, the range is between 0 and 1, and the larger the Loss value is, the worse the effect of the super-resolution image reconstructed by the model is.
An electronic device comprising a memory and a processor, the memory storing a computer program, said processor implementing the steps of said one image super-resolution reconstruction method based on FSRCNN and OPE when executing said computer program.
A computer readable storage medium having stored thereon a computer program which when executed by a processor implements the above-described image super-resolution reconstruction method based on FSRCNN and OPE.
The invention has the beneficial effects that:
the image super-resolution reconstruction method based on FSRCNN and OPE can solve the problem of image super-resolution of any scale, does not need to train an independent model aiming at each scale, and saves training time and calculation resources.
According to the image super-resolution reconstruction method based on FSRCNN and OPE, the OPE module is utilized to improve the accuracy of the up-sampling process, reduce the blurring and distortion introduced by the traditional interpolation method, and improve the definition and visual quality of the image.
According to the image super-resolution reconstruction method based on FSRCNN and OPE, the image super-resolution task of any scale can be processed by training the model once.
According to the image super-resolution reconstruction method based on FSRCNN and OPE, the lightweight design of the FSRCNN network ensures the calculation efficiency of the reconstruction process.
The image super-resolution reconstruction method based on FSRCNN and OPE has the effectiveness and potential in practical application. With the continuous development of technology, the method of the invention is expected to play an important role in more fields.
Drawings
FIG. 1 is a flow chart of an image super-resolution reconstruction method based on FSRCNN and OPE according to the invention;
FIG. 2 is a flow chart of the training process of the method of the present invention;
FIG. 3 is a schematic diagram of the structure of the FSRCNN and OPE models according to the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings and detailed description. It should be understood that the embodiments described herein are for purposes of illustration only and are not intended to limit the invention, i.e., the embodiments described are merely some, but not all, of the embodiments of the invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein can be arranged and designed in a wide variety of different configurations, and the present invention can have other embodiments as well.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, are intended to fall within the scope of the present invention.
For further understanding of the invention, the following detailed description is to be taken in conjunction with fig. 1-3, in which the following detailed description is given, of the invention:
the first embodiment is as follows:
A FSRCNN and OPE-based image super-resolution reconstruction method comprises the following steps:
S1, acquiring a high-resolution image or acquiring a high-resolution image based on a DIV2K Dataset open source data set, preprocessing to acquire a high-resolution image HR and a corresponding low-resolution image LR, and constructing a data set for training based on FSRCNN and OPE image super-resolution reconstruction methods;
Further, the specific implementation method of the step S1 includes the following steps:
s1.1, downsampling a high-resolution image by a bicubic interpolation method to obtain a low-resolution image LR, adjusting the reduction ratio of the bicubic interpolation method to obtain a low-resolution image LR with three ratios of 1/2, 1/4 and 1/8, and respectively placing the HR and the LR under different folders;
S1.2, carrying out normalization processing on the HR and LR obtained in the step S1.1, wherein the pixel value range is [0,1], and obtaining a dataset used for training an image super-resolution reconstruction method based on FSRCNN and OPE.
S2, constructing FSRCNN network structures and OPE modules;
further, the specific implementation method of the step S2 includes the following steps:
S2.1, constructing FSRCNN a network structure, wherein the first layer is a 3x3 convolution layer, 32 convolution kernels are used, a ReLU activation function is connected to the first layer, the second layer is a 3x3 maximum pooling layer, the third layer is a 3x3 convolution layer, 64 convolution kernels are used, a ReLU activation function is connected to the second layer, and the fourth layer is a 1x1 convolution layer used for compressing feature mapping;
S2.2, constructing an OPE module, namely selecting sine and cosine functions as orthogonal bases, setting maximum frequency and coding length, selecting Adam as an optimizer for model training, and setting initial learning rate lr to be 0.001.
S3, inputting the low-resolution image in the dataset obtained in the step S1 into a FSRCNN network, extracting the features of the image through a convolution layer, and outputting a feature map F;
Further, the expression of the feature of the extracted image in step S3 is:
Where x m,n is the pixel value in the input image, B i,j is the weight of the convolution kernel, f is the ReLU activation function, y i,j is the pixel value of the output feature image, M×N is the size of the convolution kernel, wherein M and N represent the position coordinates of the pixels in the image, M is the coordinates in the horizontal direction, N is the coordinates in the vertical direction, M is the width of the image, and N is the height of the image;
The output of FSRCNN network is the feature map F, the dimension of F is h×w×c, where H is the height of feature map, W is the width of feature map, and C is the number of channels of feature map.
S4, inputting the feature map obtained in the step S3 into an OPE module, coding the feature map to obtain an orthogonal position code, then carrying out up-sampling processing on the feature map by utilizing the obtained orthogonal position code, and calculating a pixel value of a super-resolution image SR through linear combination of the OPE module to generate a preliminary SR image;
Further, the specific implementation method of the step S4 includes the following steps:
s4.1, defining a group of orthogonal basis functions, wherein the group of orthogonal basis functions comprises an orthogonal basis function in the horizontal direction and an orthogonal basis function in the vertical direction, and the expression is as follows:
Wherein phi p (x) is a cosine base function in the horizontal direction, phi q (y) is a cosine base function in the vertical direction, phi p (x) is a sine base function in the horizontal direction, phi q (y) is a sine base function in the vertical direction, p and q are frequency indexes of orthogonal base functions respectively, and x and y are positions of feature maps in the horizontal direction and the vertical direction respectively;
S4.2, for each pair of orthogonal basis functions, calculating projection coefficients of characteristic mapping on the orthogonal basis functions, wherein the expression is as follows:
Wherein F m,n,c is the pixel value of the feature map at positions m, n and channel c, phi p (m) and phi q (n) are the values of the sine base function at positions m, n, respectively, and phi p (m) and phi q (n) are the values of the cosine base function at positions m, n, respectively;
s4.3, carrying out up-sampling processing on the feature mapping by utilizing the projection coefficient of the obtained feature mapping on the orthogonal basis function, calculating the pixel value of the super-resolution image SR through the linear combination of the OPE module, generating a preliminary SR image, and calculating the expression as follows:
Wherein I i,j is the pixel value of the high-resolution image at position I, j, K and L are the number of orthogonal basis functions in the horizontal and vertical directions, and P p,q is the coefficient of the P-th and q-th basis functions in the encoded feature map;
S5, performing post-processing on the preliminary SR image obtained in the step S4 to obtain an image subjected to super-resolution processing;
further, step S5 performs post-processing on the preliminary SR image, denoising by using a non-local mean technique, performing high-pass filtering by using a Sobel operator, and finally obtaining a super-resolution processed image through histogram equalization;
S6, performing performance evaluation on the super-resolution processed image obtained in the step S5, calculating a loss function, adjusting FSRCNN parameters of a network structure and OPE module through back propagation according to the result of the loss function, performing iterative training by using a data set, and outputting a reconstruction result from an LR image to an SR image;
further, the specific implementation method of the step S6 includes the following steps:
S6.1, performance evaluation, namely performing performance evaluation on the reconstructed super-resolution image by using peak signal-to-noise ratio (PSNR) and Structural Similarity (SSIM) indexes, wherein the PSNR has a calculation expression as follows:
Wherein MAX I is the maximum pixel value of the image, MSE is the mean square error, and the calculation formula of MSE is:
Wherein, I HR is the original high-resolution image, I SR is the image processed by super resolution, M and N are the number of rows and columns of the image respectively;
The calculated expression for SSIM is:
Wherein: And The average of the original high resolution image and the super resolution processed image,AndThe variances of the original high resolution image and the super resolution processed image respectively,Is the covariance of the original high-resolution image and the super-resolution processed image, C 1 and C 2 are constants for stabilizing the denominator of SSIM;
s6.2, setting MSE and SSIM to jointly construct a loss function of model training, wherein the expression is as follows:
Loss=α·MSE(I,K)+(1-α)·(1-SSIM(I,K))
Wherein alpha is a weight parameter, the range is between 0 and 1, and the larger the Loss value is, the worse the effect of the super-resolution image reconstructed by the model is.
Further, as shown in fig. 2, the iterative training process using the data set, the specific implementation method includes the following steps:
S6.2.1, inputting a training set, a verification set, an up-sampling rate and iteration times epoch into a FSRCNN network structure and an OPE module to train, and obtaining a training result;
S6.2.2, calculating a peak signal to noise ratio PSNR average value of the verification set by using a training result, then judging whether the constructed FSRCNN network structure and the OPE module are the best model, if so, updating the best model, and if not, updating the iteration times to carry out the next training until the best model is output.
The face image with 160x160 pixels is input into the model as a test case, and up-sampling multiples of the model are respectively set to be 2 times, 4 times and 6 times, so that HR images with 3 resolutions are obtained. By the implementation steps, a high-resolution face image is obtained, the resolution of the face image is improved from 160x160 pixels to 320x320 pixels, and the like, and the face image with multiple resolutions can be obtained only through one training. The reconstructed HR image has significant improvement in detail, texture, edges, etc., and no significant artifacts or noise. Both the quantitative evaluation indexes PSNR and SSIM reach higher levels.
The second embodiment is as follows:
An electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the image super-resolution reconstruction method based on FSRCNN and OPE according to the embodiment when executing the computer program.
The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit. And the processor is used for executing the computer program stored in the memory to realize the steps of the image super-resolution reconstruction method based on FSRCNN and OPE.
The Processor may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may mainly include a storage program area which may store an operating system, an application program required for at least one function (such as a sound playing function, an image playing function, etc.), etc., and a storage data area which may store data created according to the use of the cellular phone (such as audio data, a phonebook, etc.), etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SMART MEDIA CARD, SMC), secure Digital (SD) card, flash memory card (FLASH CARD), at least one disk storage device, flash memory device, or other volatile solid-state storage device.
And a third specific embodiment:
a computer readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements a method for reconstructing an image super-resolution based on FSRCNN and OPE according to the embodiment.
The computer readable storage medium of the present invention may be any form of storage medium that is readable by a processor of a computer device, including but not limited to, nonvolatile memory, volatile memory, ferroelectric memory, etc., on which a computer program is stored, and when the processor of the computer device reads and executes the computer program stored in the memory, the steps of an image super resolution reconstruction method based on FSRCNN and OPE described above can be implemented.
The computer program comprises computer program code which may be in source code form, object code form, executable file or in some intermediate form, etc. The computer readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth.
It is noted that relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
Although the application has been described above with reference to specific embodiments, various modifications may be made and equivalents may be substituted for elements thereof without departing from the scope of the application. In particular, the features of the disclosed embodiments may be combined with each other in any manner so long as there is no structural conflict, and the exhaustive description of these combinations is not given in this specification solely for the sake of brevity and resource saving. Therefore, it is intended that the application not be limited to the particular embodiments disclosed herein, but that the application will include all embodiments falling within the scope of the appended claims.

Claims (7)

1.一种基于FSRCNN和OPE的图像超分辨率重建方法，其特征在于，包括如下步骤：1. An image super-resolution reconstruction method based on FSRCNN and OPE, characterized by comprising the following steps: S1.采集高分辨率图像或者基于DIV2K Dataset开源数据集得到高分辨率图像，预处理后获得高分辨率图像HR及对应的低分辨率图像LR，构建用于基于FSRCNN和OPE的图像超分辨率重建方法训练的数据集；S1. Collect high-resolution images or obtain high-resolution images based on the DIV2K Dataset open source dataset, obtain high-resolution images HR and corresponding low-resolution images LR after preprocessing, and construct a dataset for training image super-resolution reconstruction methods based on FSRCNN and OPE; S2.构建FSRCNN网络结构和OPE模块；S2. Construct FSRCNN network structure and OPE module; 步骤S2的具体实现方法包括如下步骤：The specific implementation method of step S2 includes the following steps: S2.1.构建FSRCNN网络结构：第一层为一个3x3卷积层，使用32个卷积核，后接ReLU激活函数，第二层为一个3x3最大池化层，第三层为一个3x3卷积层，使用64个卷积核，后接ReLU激活函数，第四层为一个1x1卷积层，用于压缩特征映射；S2.1. Construct the FSRCNN network structure: the first layer is a 3x3 convolution layer, using 32 convolution kernels, followed by a ReLU activation function, the second layer is a 3x3 maximum pooling layer, the third layer is a 3x3 convolution layer, using 64 convolution kernels, followed by a ReLU activation function, and the fourth layer is a 1x1 convolution layer for compressing feature maps; S2.2.构建OPE模块：选用正弦和余弦函数作为正交基，然后设置最大频率和编码长度，选择Adam作为模型训练的优化器，初始学习率lr设置为0.001；S2.2. Construct the OPE module: select sine and cosine functions as orthogonal bases, then set the maximum frequency and encoding length, select Adam as the optimizer for model training, and set the initial learning rate lr to 0.001; S3.将步骤S1中得到数据集中的低分辨率图像输入到FSRCNN网络中，通过卷积层提取图像的特征，输出特征映射F；S3. Input the low-resolution image in the data set obtained in step S1 into the FSRCNN network, extract the features of the image through the convolution layer, and output the feature map F; S4.将步骤S3得到的特征映射输入到OPE模块中，对特征映射进行编码，得到正交位置编码，然后利用得到的正交位置编码对特征映射进行上采样处理，通过OPE模块的线性组合，计算超分辨率图像SR的像素值，生成初步的SR图像；S4. Input the feature map obtained in step S3 into the OPE module, encode the feature map to obtain the orthogonal position code, and then use the obtained orthogonal position code to upsample the feature map, calculate the pixel value of the super-resolution image SR through the linear combination of the OPE module, and generate a preliminary SR image; 步骤S4的具体实现方法包括如下步骤：The specific implementation method of step S4 includes the following steps: S4.1.定义一组正交基函数包括水平方向上的正交基函数和垂直方向上的正交基函数，表达式为：S4.1. Define a set of orthogonal basis functions including orthogonal basis functions in the horizontal direction and orthogonal basis functions in the vertical direction, expressed as: 其中，φp(x)为水平方向上的余弦基函数，φq(y)为竖直方向上的余弦基函数，ψp(x)为水平方向上的正弦基函数，ψq(y)为竖直方向上的正弦基函数，p和q分别为正交基函数的频率索引，x和y分别为特征映射在水平方向上的位置和垂直方向上的位置；Wherein, φ p (x) is the cosine basis function in the horizontal direction, φ q (y) is the cosine basis function in the vertical direction, ψ p (x) is the sine basis function in the horizontal direction, ψ q (y) is the sine basis function in the vertical direction, p and q are the frequency indexes of the orthogonal basis functions, respectively, and x and y are the positions of the feature map in the horizontal direction and the vertical direction, respectively; S4.2.对于每对正交基函数，计算特征映射在正交基函数上的投影系数，表达式为：S4.2. For each pair of orthogonal basis functions, calculate the projection coefficient of the feature map on the orthogonal basis function, expressed as: 其中，Fm,n,c为特征映射在位置m、n处、通道c上的像素值，φp(m)和φq(n)分别为正弦基函数在位置m、n处的值，ψp(m)和ψq(n)分别为余弦基函数在位置m、n处的值；Where Fm ,n,c is the pixel value of the feature map at position m, n and channel c, φp (m) and φq (n) are the values of the sine basis function at position m and n, respectively, and ψp (m) and ψq (n) are the values of the cosine basis function at position m and n, respectively; S4.3.利用得到的特征映射在正交基函数上的投影系数对特征映射进行上采样处理，通过OPE模块的线性组合，计算超分辨率图像SR的像素值，生成初步的SR图像，计算表达式为：S4.3. The feature map is upsampled using the projection coefficients of the feature map obtained on the orthogonal basis function. The pixel values of the super-resolution image SR are calculated through the linear combination of the OPE modules to generate a preliminary SR image. The calculation expression is: 其中，Ii,j为高分辨率图像在位置i,j的像素值，K和L是正交基函数在水平和垂直方向上的数量，Pp,q为编码的特征映射中第p个和第q个基函数的系数；Where I i,j is the pixel value of the high-resolution image at position i,j, K and L are the number of orthogonal basis functions in the horizontal and vertical directions, and P p,q is the coefficient of the pth and qth basis functions in the encoded feature map; S5.将步骤S4得到的初步的SR图像进行后处理，得到超分辨率处理后的图像；S5. Post-processing the preliminary SR image obtained in step S4 to obtain a super-resolution processed image; S6.将步骤S5得到的超分辨率处理后的图像进行性能评估并计算损失函数，根据损失函数的结果，通过反向传播调整FSRCNN网络结构和OPE模块的参数，利用数据集进行迭代训练，输出从LR图像到SR图像的重建结果。S6. Perform performance evaluation on the super-resolution processed image obtained in step S5 and calculate the loss function. According to the result of the loss function, adjust the FSRCNN network structure and the parameters of the OPE module through back propagation, use the data set for iterative training, and output the reconstruction result from the LR image to the SR image. 2.根据权利要求1所述的一种基于FSRCNN和OPE的图像超分辨率重建方法，其特征在于，步骤S1的具体实现方法包括如下步骤：2. According to the method for image super-resolution reconstruction based on FSRCNN and OPE in claim 1, it is characterized in that the specific implementation method of step S1 comprises the following steps: S1.1.将高分辨率图像通过双三次插值方法下采样得到低分辨率图像LR，并且调整双三次插值方法的缩小比例，得到1/2、1/4、1/8三种比例的低分辨率图像LR，将HR和LR分别放置在不同的文件夹下；S1.1. Downsample the high-resolution image to obtain a low-resolution image LR by using the bicubic interpolation method, and adjust the reduction ratio of the bicubic interpolation method to obtain low-resolution images LR with three ratios of 1/2, 1/4, and 1/8, and place HR and LR in different folders respectively; S1.2.将步骤S1.1得到的HR和LR进行归一化处理后，像素值范围在[0,1]，得到用于基于FSRCNN和OPE的图像超分辨率重建方法训练的数据集。S1.2. After normalizing the HR and LR obtained in step S1.1, the pixel value range is [0,1], and a data set for training the image super-resolution reconstruction method based on FSRCNN and OPE is obtained. 3.根据权利要求2所述的一种基于FSRCNN和OPE的图像超分辨率重建方法，其特征在于，步骤S3提取图像的特征的表达式为：3. The image super-resolution reconstruction method based on FSRCNN and OPE according to claim 2 is characterized in that the expression of the feature of the image extracted in step S3 is: 其中，xm,n为输入图像中的像素值，为卷积核的权重，bi,j为偏置项，f为ReLU激活函数，yi,j为输出特征图的像素值，M×N为卷积核的大小，其中m、n代表图像中像素的位置坐标，m为水平方向上的坐标，n为垂直方向上的坐标，M为图像的宽度，N为图像的高度；Where xm ,n is the pixel value in the input image, is the weight of the convolution kernel, bi ,j is the bias term, f is the ReLU activation function, yi ,j is the pixel value of the output feature map, M×N is the size of the convolution kernel, where m and n represent the position coordinates of the pixel in the image, m is the horizontal coordinate, n is the vertical coordinate, M is the width of the image, and N is the height of the image; 得到FSRCNN网络的输出为特征映射F，F的维度为H×W×C，其中，H为特征映射的高度，W为特征映射的宽度，C为特征映射的通道数。The output of the FSRCNN network is a feature map F, and the dimension of F is H×W×C, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map. 4.根据权利要求3所述的一种基于FSRCNN和OPE的图像超分辨率重建方法，其特征在于，步骤S5对初步的SR图像进行后处理，使用非局部均值技术去噪，再使用Sobel算子进行高通滤波，最后通过直方图均衡化，得到超分辨率处理后的图像。4. According to the method for image super-resolution reconstruction based on FSRCNN and OPE in claim 3, it is characterized in that, in step S5, the preliminary SR image is post-processed, denoised using a non-local means technique, high-pass filtered using a Sobel operator, and finally histogram equalized to obtain a super-resolution processed image. 5.根据权利要求4所述的一种基于FSRCNN和OPE的图像超分辨率重建方法，其特征在于，步骤S6的具体实现方法包括如下步骤：5. According to the method of claim 4, the image super-resolution reconstruction method based on FSRCNN and OPE is characterized in that the specific implementation method of step S6 comprises the following steps: S6.1.性能评估：使用峰值信噪比PSNR、结构相似性SSIM指标对重建的超分辨率图像进行性能评估，其中PSNR的计算表达式为：S6.1. Performance evaluation: The performance of the reconstructed super-resolution image is evaluated using the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) indicators. The calculation expression of PSNR is: 其中，MAXI为图像的最大像素值，MSE为均方误差，MSE的计算公式为：Among them, MAX I is the maximum pixel value of the image, MSE is the mean square error, and the calculation formula of MSE is: 其中，IHR为原始高分辨率图像，ISR为超分辨率处理后的图像，M和N分别是图像的行数和列数；Where I HR is the original high-resolution image, I SR is the image after super-resolution processing, M and N are the number of rows and columns of the image respectively; SSIM的计算表达式为：The calculation expression of SSIM is: 其中：和分别为原始高分辨率图像和超分辨率处理后的图像的均值，和分别是原始高分辨率图像和超分辨率处理后的图像的方差，是原始高分辨率图像和超分辨率处理后的图像的协方差，C1和C2是常数，用于稳定SSIM的分母；in: and are the means of the original high-resolution image and the image after super-resolution processing, and are the variances of the original high-resolution image and the image after super-resolution processing, is the covariance of the original high-resolution image and the image after super-resolution processing, C1 and C2 are constants used to stabilize the denominator of SSIM; S6.2.设置MSE和SSIM共同构建模型训练的损失函数，表达式为：S6.2. Set MSE and SSIM to jointly construct the loss function of model training, the expression is: Loss＝α·MSE(I,K)+(1-α)·(1-SSIM(I,K))Loss＝α·MSE(I,K)+(1-α)·(1-SSIM(I,K)) 其中，α为权重参数，范围在[0,1]之间，Loss值越大说明模型重建的超分辨率图像效果越差。Among them, α is the weight parameter, ranging from [0,1]. The larger the Loss value, the worse the super-resolution image reconstructed by the model. 6.一种电子设备，其特征在于，包括存储器和处理器，存储器存储有计算机程序，所述的处理器执行所述计算机程序时实现权利要求1-5任一项所述的一种基于FSRCNN和OPE的图像超分辨率重建方法的步骤。6. An electronic device, characterized in that it includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of an image super-resolution reconstruction method based on FSRCNN and OPE as described in any one of claims 1 to 5 when executing the computer program. 7.一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1-5任一项所述的一种基于FSRCNN和OPE的图像超分辨率重建方法。7. A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the image super-resolution reconstruction method based on FSRCNN and OPE according to any one of claims 1 to 5 is implemented.
CN202410671523.2A 2024-05-28 2024-05-28 Image super-resolution reconstruction method, electronic device and storage medium based on FSRCNN and OPE Active CN118469818B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410671523.2A CN118469818B (en) 2024-05-28 2024-05-28 Image super-resolution reconstruction method, electronic device and storage medium based on FSRCNN and OPE

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410671523.2A CN118469818B (en) 2024-05-28 2024-05-28 Image super-resolution reconstruction method, electronic device and storage medium based on FSRCNN and OPE

Publications (2)

Publication Number Publication Date
CN118469818A CN118469818A (en) 2024-08-09
CN118469818B true CN118469818B (en) 2025-04-18

Family

ID=92153212

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410671523.2A Active CN118469818B (en) 2024-05-28 2024-05-28 Image super-resolution reconstruction method, electronic device and storage medium based on FSRCNN and OPE

Country Status (1)

Country Link
CN (1) CN118469818B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119444578B (en) * 2025-01-08 2025-05-09 哈尔滨工业大学(深圳)(哈尔滨工业大学深圳科技创新研究院) Image super-resolution reconstruction method and system based on mixed expert and stable diffusion

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116245736A (en) * 2023-05-11 2023-06-09 南京信息工程大学 Orthogonal position coding representation method of image block and up-sampling module implementation method
CN116309038A (en) * 2022-11-14 2023-06-23 哈尔滨工业大学 Low-resolution image domain transfer network construction method and device, domain transfer-based unpaired MR image super-resolution method and device

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112862681B (en) * 2021-01-29 2023-04-14 中国科学院深圳先进技术研究院 A super-resolution method, device, terminal equipment and storage medium

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116309038A (en) * 2022-11-14 2023-06-23 哈尔滨工业大学 Low-resolution image domain transfer network construction method and device, domain transfer-based unpaired MR image super-resolution method and device
CN116245736A (en) * 2023-05-11 2023-06-09 南京信息工程大学 Orthogonal position coding representation method of image block and up-sampling module implementation method

Also Published As

Publication number Publication date
CN118469818A (en) 2024-08-09

Similar Documents

Publication Publication Date Title
Ren et al. Single image super-resolution via adaptive high-dimensional non-local total variation and adaptive geometric feature
Liu et al. Recovery analysis for plug-and-play priors using the restricted eigenvalue condition
Zhu et al. Video super-resolution based on a spatio-temporal matching network
CN114565528B (en) A remote sensing image denoising method and system based on multi-scale and attention mechanism
CN106952228A (en) Single image super-resolution reconstruction method based on non-local self-similarity of images
CN101976435A (en) Combination learning super-resolution method based on dual constraint
CN112488920B (en) A method for image regularization super-resolution reconstruction based on Gaussian-like blur kernel
Guo et al. Multiscale semilocal interpolation with antialiasing
CN118469818B (en) Image super-resolution reconstruction method, electronic device and storage medium based on FSRCNN and OPE
Thapa et al. A performance comparison among different super-resolution techniques
Wu et al. Two-level wavelet-based convolutional neural network for image deblurring
Liu et al. Image super-resolution via hierarchical and collaborative sparse representation
Daithankar et al. Analysis of the wavelet domain filtering approach for video super-resolution
CN102339460B (en) Adaptive satellite image restoration method
CN103903239A (en) Video super-resolution reconstruction method and system based on sparse principal component analysis and continued fraction interpolation technology
CN113902618B (en) Image super-resolution algorithm based on multi-modal spatial filtering
CN115564647A (en) Novel super-division module and up-sampling method for image semantic segmentation
Jiang et al. Optimal filter based on scale-invariance generation of natural images
Patel et al. Deep learning in medical image super-resolution: A survey
Zhang et al. Multi-noise and multi-channel derived prior information for grayscale image restoration
Vyas et al. Image restoration
Panda et al. A novel image upscaling method using high order error sharpening
Parisi On image restoration via nonlinear anisotropic PDE with non-standards growth conditions.
CN119338677B (en) A high-precision lake extraction method based on unsupervised remote sensing super-resolution
CN117132468B (en) Curvelet coefficient prediction-based super-resolution reconstruction method for precise measurement image

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant