CN117522748A - Fisheye image correction method and system based on SKNet attention mechanism - Google Patents

Abstract

The invention discloses a fisheye image correction method and system based on a SKNet attention mechanism, belongs to the technical field of image processing, and aims to solve the technical problem of how to correct fisheye images and reduce the problems of steep sharpness and steep disappearance. The method comprises the following steps: constructing a stream estimation network model, which is used for extracting appearance characteristics of an input fisheye image and outputting a multi-scale appearance stream; constructing a distortion correction network model, wherein the distortion correction network model comprises a generator, a correction layer and a discriminator, the generator comprises a U-shaped encoder-decoder structure, and the encoder introduces a SKNet attention mechanism layer; the correction layer is used for correcting the distortion characteristic diagram of the corresponding scale based on the appearance flow; the encoder is used for reconstructing an image based on the input multi-scale corrected distortion characteristic image and outputting a multi-scale corrected image; the discriminator incorporates a spectral normalization layer for determining whether each scale corrected image is a true image.

Description

Fisheye image correction method and system based on SKNet attention mechanism

Technical field

The present invention relates to the field of image processing technology, specifically a fisheye image correction method and system based on the SKNet attention mechanism.

Background technique

In recent years, with the rapid development of robot vision and virtual reality technology, fisheye cameras have been widely used in special-purpose image acquisition fields due to their large viewing angle and small size, such as surveillance systems, security systems and various conferences. However, fisheye images are accompanied by distortion, especially in the peripheral areas. Therefore, distortion correction of fisheye images becomes crucial to conform to human vision and reduce interference in feature extraction.

The fisheye lens is an optical lens with a large field of view and is widely used in fields such as video surveillance, panoramic photography, and high-altitude measurement. However, in the field of measurement, the nonlinear distortion of fisheye lenses can lead to large errors, affecting practical applications. At present, fisheye camera imaging systems use computer image vision technology to correct image distortion. The fisheye image distortion correction algorithm emerged as the times require when facing the challenge of distorted images, and has become a research focus to meet the needs of high-precision measurement and practical applications. need. By applying fisheye correction technology, the image captured by the fisheye lens can be converted into a right-angle projection image that is more in line with the observation habits of the human eye, thereby eliminating or reducing the distortion of the image. The fisheye correction technology helps to improve the visualization effect of the image and provides More accurate image information, and improved performance of subsequent algorithms and applications.

How to correct the fisheye image and reduce the problem of steepness and disappearance is a technical problem that needs to be solved.

Contents of the invention

The technical task of the present invention is to address the above shortcomings and provide a fish-eye image correction method and system based on the SKNet attention mechanism to solve the technical problem of how to correct the fish-eye image and reduce the problems of steepness and disappearance.

In the first aspect, the present invention is a fisheye image correction method based on the SKNet attention mechanism, which includes the following steps:

Image collection: Obtain real images and corresponding fisheye images to construct a sample set;

Flow estimation model construction: Construct a flow estimation network model. The flow estimation network model is a U-shaped encoder-decoder structure, which is used to extract appearance features from the input fisheye image, map the appearance feature map into a flow, and output multi-scale appearance flow;

Distortion correction model construction: Construct a distortion correction network model. The distortion correction network model includes a generator, a correction layer and a discriminator. The generator includes a U-shaped encoder-decoder structure, and the encoder introduces an SKNet attention mechanism layer. , used to extract distortion features from the input fisheye image and output a multi-scale distortion feature map; the correction layer is used to take the multi-scale appearance flow and the multi-scale distortion feature map as input, and extract the distortion features of the corresponding scales based on the appearance flow. The image is corrected and a multi-scale corrected distortion feature map is output; the encoder is used for image reconstruction based on the input multi-scale corrected distortion feature map and outputs a multi-scale corrected image; the discriminator introduces a spectral normalization layer, Used to take multi-scale corrected images as input, determine whether the corrected image at each scale is a real image, and predict and output the category label of the corrected image at each scale and the corresponding category probability. The category labels include real images and fake images;

Model training: Carry out model training on the flow estimation network model and the distortion correction network model based on the sample set, and obtain the post-training flow estimation network model and the post-training distortion correction network model;

Image correction: Taking the fisheye image to be corrected as input, the trained flow estimation network model and the trained distortion correction network model predict and output the category label and category probability of the corrected image at each scale, and the final correction is obtained based on the category probability. After image.

Preferably, for the flow estimation network model, the encoder includes N convolution structures, and the decoder includes N-1 deconvolution structures;

As a downsampling structure, each convolutional structure includes a convolution layer, a normalization layer and a Leaky ReLU activation function connected in sequence, which is used to downsample the input image and extract appearance features, and output the appearance feature map;

Each deconvolution structure includes a residual module and a deconvolution layer as an upsampling structure, which is used to upsample the input appearance feature map and output appearance feature maps of N-1 scales;

The output end of each deconvolution structure is connected to a convolution layer. The convolution layer is used to perform a convolution operation on the input appearance feature map and map the appearance feature map into a stream to obtain an appearance stream.

Preferably, the generator includes an encoder, a decoder, a correction layer, an image processing structure and an image correction structure, the encoder includes N convolution structures and N-1 deconvolution structures, and the decoder includes N-1 deconvolutions. product structure;

For the first N-1 convolution structures, each convolution structure includes a convolution layer and a Leaky ReLU activation function as a down-sampling structure, and each down-sampling structure introduces a sknet attention mechanism layer for input fisheye The image is subjected to downsampling operation and distortion feature extraction, and a distortion feature map of the corresponding scale is output;

For the Nth convolution structure, the convolution structure includes a Leaky ReLU activation function, which is used to perform feature transformation on the input distortion feature map and output the distortion feature map;

The correction layer is connected to the flow estimation network model based on the progressive complementation mechanism and is used to perform the following: taking multi-scale appearance flow and multi-scale distortion feature maps as input, for each scale distortion feature map, based on the appearance flow of the corresponding scale, the distortion feature is The image is spatially transformed to correct the distortion feature map, and the corrected distortion feature map of the corresponding scale is output, in which the appearance flow and the distortion feature map both have a total of N channels and have a one-to-one correspondence;

For each deconvolution structure, the convolution structure includes a convolution layer and a ReLU activation function as an upsampling structure, which is used to perform an upsampling operation on the corrected distortion feature map of the corresponding scale and output the corrected distortion feature of the corresponding scale. picture;

The output end of each deconvolution structure is configured with an image processing structure. The image processing structure includes a convolution layer, a Tanh activation function and a torgb function, which are used to perform convolution operations and activation operations on the input corrected distortion feature map, and pass The torgb function converts the corrected distortion feature map into an RGB image to obtain the corrected distortion feature map in the form of an RGB image;

The image correction structure is a convolution layer, which is used to take the corrected distortion feature map in the form of a multi-scale RGB image as input, perform a downsampling operation on the corrected distortion feature map of each scale, and output a multi-scale corrected image.

Preferably, the discriminator includes M convolutional network structures. Each convolutional network structure includes a convolution layer, a LeakyReLU activation function and a spectral normalization layer, which is used to take the corrected image of each scale as input and make judgments. Whether the corrected image is a real image, the class label and class probability are output. The class label includes real images and fake images.

Preferably, an adversarial loss function is constructed through the Wasserstein distance loss function based on the multi-scale corrected image and the real image of the corresponding scale, and the adversarial loss function is used for model training of the generator and the discriminator;

A multi-scale loss function is constructed based on the multi-scale corrected image and the real image of the corresponding scale. The adversarial loss function and the scale loss function are weighted as the total loss function of the flow estimation network model and the distortion correction network model. Based on the total loss function, the flow estimation network The model and the distortion correction network model are used for model training, and the post-training flow estimation network model and the post-training distortion correction network model are obtained.

In the second aspect, the present invention is a fish-eye image correction system based on the SKNet attention mechanism, which is used to realize fish-eye images through a fish-eye image correction method based on the SKNet attention mechanism as described in any one of the first aspects. Correction, the system includes an image acquisition module, a flow estimation model building module, a distortion correction model building module, a model training module and an image correction module;

The image acquisition module is used to perform the following: obtain real images and corresponding fisheye images to construct a sample set;

The flow estimation model building module is used to perform the following: construct a flow estimation network model, which is a U-shaped encoder-decoder structure, and is used to extract appearance features from the input fisheye image and convert the appearance feature map Map to a stream and output a multi-scale appearance stream;

The distortion correction model building module is used to perform the following: construct a distortion correction network model. The distortion correction network model includes a generator, a correction layer and a discriminator. The generator includes a U-shaped encoder-decoder structure. The encoder introduces The SKNet attention mechanism layer is used to extract distortion features of the input fisheye image and output multi-scale distortion feature maps; the correction layer is used to take multi-scale appearance flow and multi-scale distortion feature maps as input, and based on the appearance flow pair Correct the distortion feature map of the corresponding scale, and output the multi-scale corrected distortion feature map; the encoder is used to perform image reconstruction based on the input multi-scale corrected distortion feature map, and output the multi-scale corrected image; the discriminator introduces spectral The normalization layer is used to take the multi-scale corrected image as input, determine whether the corrected image at each scale is a real image, and predict and output the category label of the corrected image at each scale and the corresponding category probability. The category label includes the real image. images and forged images;

The model training module is used to perform the following: perform model training on the flow estimation network model and the distortion correction network model based on the sample set, and obtain the trained flow estimation network model and the trained distortion correction network model;

The image correction module is used to perform the following: taking the fisheye image to be corrected as input, predicting and outputting the category label and category probability of the corrected image at each scale through the trained flow estimation network model and the trained distortion correction network model, based on the category The class probability is used to obtain the final corrected image.

Preferably, for the flow estimation network model, the encoder includes N convolution structures, and the decoder includes N-1 deconvolution structures;

As a downsampling structure, each convolutional structure includes a convolution layer, a normalization layer and a Leaky ReLU activation function connected in sequence, which is used to downsample the input image and extract appearance features, and output the appearance feature map;

Each deconvolution structure includes a residual module and a deconvolution layer as an upsampling structure, which is used to upsample the input appearance feature map and output appearance feature maps of N-1 scales;

The output end of each deconvolution structure is connected to a convolution layer. The convolution layer is used to perform a convolution operation on the input appearance feature map and map the appearance feature map into a stream to obtain an appearance stream.

Preferably, the generator includes an encoder, a decoder, a correction layer, an image processing structure and an image correction structure, the encoder includes N convolution structures and N-1 deconvolution structures, and the decoder includes N-1 deconvolutions. product structure;

For the first N-1 convolution structures, each convolution structure includes a convolution layer and a Leaky ReLU activation function as a down-sampling structure, and each down-sampling structure introduces a sknet attention mechanism layer for input fisheye The image is subjected to downsampling operation and distortion feature extraction, and a distortion feature map of the corresponding scale is output;

For the Nth convolution structure, the convolution structure includes a Leaky ReLU activation function, which is used to perform feature transformation on the input distortion feature map and output the distortion feature map;

The correction layer is connected to the flow estimation network model based on the progressive complementation mechanism and is used to perform the following: taking multi-scale appearance flow and multi-scale distortion feature maps as input, for each scale distortion feature map, based on the appearance flow of the corresponding scale, the distortion feature is The image is spatially transformed to correct the distortion feature map, and the corrected distortion feature map of the corresponding scale is output, in which the appearance flow and the distortion feature map both have a total of N channels and have a one-to-one correspondence;

For each deconvolution structure, the convolution structure includes a convolution layer and a ReLU activation function as an upsampling structure, which is used to perform an upsampling operation on the corrected distortion feature map of the corresponding scale and output the corrected distortion feature of the corresponding scale. picture;

The output end of each deconvolution structure is configured with an image processing structure. The image processing structure includes a convolution layer, a Tanh activation function and a torgb function, which are used to perform convolution operations and activation operations on the input corrected distortion feature map, and pass The torgb function converts the corrected distortion feature map into an RGB image to obtain the corrected distortion feature map in the form of an RGB image;

The image correction structure is a convolution layer, which is used to take the corrected distortion feature map in the form of a multi-scale RGB image as input, perform a downsampling operation on the corrected distortion feature map of each scale, and output a multi-scale corrected image.

Preferably, the discriminator includes M convolutional network structures. Each convolutional network structure includes a convolution layer, a LeakyReLU activation function and a spectral normalization layer, which is used to take the corrected image of each scale as input and make judgments. Whether the corrected image is a real image, the class label and class probability are output. The class label includes real images and fake images.

Preferably, the model training module is used to construct an adversarial loss function through the Wasserstein distance loss function based on the multi-scale corrected image and the real image of the corresponding scale, and the adversarial loss function is used to perform model training on the generator and the discriminator;

The model training module is used to construct a multi-scale loss function based on the multi-scale corrected image and the real image of the corresponding scale, and weight the adversarial loss function and the scale loss function as the total loss function of the flow estimation network model and the distortion correction network model, And it is used to conduct model training on the flow estimation network model and the distortion correction network model based on the total loss function, and obtain the post-training flow estimation network model and the post-training distortion correction network model.

The fisheye image correction method and system based on the SKNet attention mechanism invented by this [WYong1] has the following advantages:

1. Extract the appearance features of the fisheye image through the flow estimation network model and output the multi-scale appearance flow. Extract the distortion features of the fisheye image through the encoder in the generator, output the multi-scale distortion features, and introduce the appearance based on the progressive complementation mechanism. The stream corrects the distortion features of the corresponding scale, and the decoder in the generator performs image reconstruction based on the corrected distortion features to generate a corrected image, and uses the discriminator to identify the corrected image and determine whether it is a real image, realizing the fisheye image. Correction, and the SKNet attention mechanism layer is introduced in the generator encoder. By introducing the SKNet self-attention mechanism, selective multi-scale convolution kernels are introduced, which improves the performance of feature extraction without increasing network complexity. Ability to introduce a spectral normalization layer in the discriminator to control the size of the weight by constraining the spectral norm of the weight matrix of the network, thus reducing the problems of steep abruptness and steep disappearance [W with 2], which helps to improve Easily train deep neural networks, especially in complex games between generators and discriminators;

2. Introduce the Wasserstein distance loss function as the loss function between the generator and the discriminator, use the Wasserstein distance to measure the difference between the generated image and the real image, completely solve the problem of unstable training of the generator and the discriminator, and no longer Care needs to be taken to balance the degree of training of the generator and discriminator.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings in the following description are only illustrative of the present invention. For some embodiments, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

The present invention will be further described below in conjunction with the accompanying drawings.

Figure 1 is a flow chart of a fisheye image correction method based on the SKNet attention mechanism in Embodiment 1;

Figure 2 is a structural principle block diagram of the SKNet attention mechanism layer in the fisheye image correction method based on the SKNet attention mechanism in Embodiment 1 of Figure 1.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific examples, so that those skilled in the art can better understand the present invention and implement it. However, the illustrated embodiments are not intended to limit the present invention. In the absence of conflict, Below, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

Embodiments of the present invention provide a fish-eye image correction method and system based on the SKNet attention mechanism, which is used to solve the technical problem of how to correct the fish-eye image and reduce the problems of sharpness and disappearance of steepness.

Example 1:

The present invention is a fisheye image correction method based on the SKNet attention mechanism, which includes five steps: image acquisition, flow estimation model construction, distortion correction model construction, model training and image correction.

Step S100 Image collection: Obtain real images and corresponding fisheye images to construct a sample set.

Step S200 Flow estimation model construction: Construct a flow estimation network model. The flow estimation network model is a U-shaped encoder-decoder structure, which is used to extract appearance features from the input fisheye image and map the appearance feature map into a flow. , output a multi-scale appearance stream.

In the flow estimation network model of this embodiment, the encoder includes N convolution structures, and the decoder includes N-1 deconvolution structures. Among them, each convolution structure as a down-sampling structure includes a convolution layer, a normalization layer and a Leaky ReLU activation function connected in sequence, which is used to perform down-sampling operations and appearance feature extraction on the input image, and output the appearance feature map; each A deconvolution structure as an upsampling structure includes a residual module and a deconvolution layer, which is used to upsample the input appearance feature map and output appearance feature maps of N-1 scales; each deconvolution The output end of the structure is connected to a convolution layer. The convolution layer is used to perform a convolution operation on the input appearance feature map and map the appearance feature map into a stream to obtain the appearance stream.

As a specific implementation of the flow estimation network model, the encoder is composed of 6 convolutional layers. Each layer contains 1 convolution, normalization and LReLu activation function activation and other operations, which are used to downsample the image and extract features. The resolution of the input image is reduced, and the 256×256×3 size image is downsampled to a 4×4×512 size.

The decoder consists of 5 deconvolution layers, each layer contains 1 residual block and 1 deconvolution operation, which is used to upsample the image and restore the image to its original resolution.

The flow estimation module utilizes an encoder-decoder structure to extract features and generate a series of shape flows. A convolutional layer is provided at the output end of the decoder. The convolutional layer uses a 3x3 convolution kernel to process the extracted appearance feature map to obtain an appearance stream of two channels. The feature maps of different resolutions are mapped to the stream. The output, in this way, is obtained 5 appearance streams with sizes of 128, 64, 32, 16, and 8.

The flow estimation network model evaluates the degree of distortion of the fisheye image and renders it as an apparent flow. The network model uses an encoder-decoder structure to extract features and generate sequences. Specifically, the output features of the deconvolution structure of each decoder undergo an additional 3x3 convolution operation to obtain an appearance stream of two channels, and finally obtain an appearance stream of 5 graphics.

where I _in is the input fisheye image, G _s presents the appearance flow estimation module, is the appearance stream output by the i-th deconvolution structure.

Step S300 Distortion correction model construction: Construct a distortion correction network model. The distortion correction network model includes a generator, a correction layer and a discriminator. The generator includes a U-shaped encoder-decoder structure, and the encoder introduces SKNet attention. The mechanism layer is used to extract distortion features of the input fisheye image and output a multi-scale distortion feature map; the correction layer is used to take the multi-scale appearance flow and the multi-scale distortion feature map as input, and perform the corresponding scale based on the appearance flow. The distortion feature map is corrected and a multi-scale corrected distortion feature map is output; the encoder is used to perform image reconstruction based on the input multi-scale corrected distortion feature map and output a multi-scale corrected image; the discriminator introduces spectral normalization The layer is used to take multi-scale corrected images as input, determine whether the corrected image at each scale is a real image, and predict and output the category label of the corrected image at each scale and the corresponding category probability. The category labels include real images and fake images. image.

The distortion correction model in this embodiment is similar to a GAN network. The generator includes an encoder, a decoder, a correction layer, an image processing structure and an image correction structure. The encoder includes N convolution structures and N-1 deconvolution structures. The decoder Includes N-1 deconvolution structures.

For the first N-1 convolution structures, each convolution structure includes a convolution layer and a Leaky ReLU activation function as a down-sampling structure, and each down-sampling structure introduces a sknet attention mechanism layer for input fisheye The image is subjected to downsampling operation and distortion feature extraction, and a distortion feature map of the corresponding scale is output; for the Nth convolution structure, the convolution structure includes a Leaky ReLU activation function, which is used to perform feature transformation on the input distortion feature map and output Distortion feature map; the correction layer is connected to the flow estimation network model based on the asymptotic complementation mechanism and is used to perform the following: taking the multi-scale appearance flow and the multi-scale distortion feature map as input, for the distortion feature map of each scale, based on the appearance of the corresponding scale The flow performs spatial transformation on the distortion feature map to correct the distortion feature map, and outputs the corrected distortion feature map of the corresponding scale, in which the appearance flow and the distortion feature map both have a total of N channels and correspond one to one; for each deconvolution structure, the As an upsampling structure, the convolution structure includes a convolution layer and a ReLU activation function, which is used to upsample the corrected distortion feature map of the corresponding scale and output the corrected distortion feature map of the corresponding scale; the output of each deconvolution structure The terminal is equipped with an image processing structure. The image processing structure includes a convolution layer, Tanh activation function and torgb function, which is used to perform convolution and activation operations on the input corrected distortion feature map, and convert the corrected distortion feature map through the torgb function. Convert to an RGB image to obtain a corrected distortion feature map in the form of an RGB image; the image correction structure is a convolutional layer, which is used to take the corrected distortion feature map in the form of a multi-scale RGB image as input, and calculate the corrected distortion features of each scale The image is downsampled and the multi-scale corrected image is output.

The discriminator includes M convolutional network structures. Each convolutional network structure includes a convolution layer, a LeakyReLU activation function and a spectral normalization layer. It is used to take the corrected image at each scale as input and determine whether the corrected image is Real images, output class labels and class probability, class labels include real images and fake images.

In order to solve the problem of distortion divergence, this embodiment inserts a feature correction layer into the generator to pre-correct image features before transmission. At the same time, a SKNet attention mechanism layer is added to the encoder of the generator to improve the experience of the model. With the field and feature selection capabilities, it can be more suitable for processing complex image data. The predicted appearance flow is used to perform spatial transformation to obtain the corrected feature map.

As a specific implementation, the main part of the generator is a repair generation network, which is composed of an encoder and a decoder. The encoder extracts features from the fisheye image to obtain a multi-scale distortion feature map. For each distortion feature map, Correction is performed through the appearance flow to obtain the corrected distortion feature map, and image reconstruction is performed based on the corrected distortion feature map to generate the corrected image.

Among them, the encoder includes six convolution structures, but for the first five convolution structures, each convolution structure includes a convolution layer and a Leaky ReLU activation function as a downsampling layer, and the convolution kernel size is 3*3 , with a step size of 1; the last convolution structure, using the ReLU activation function but without downsampling, is used to perform feature transformation on the extracted feature maps. The output of these convolution structures will be used as the intermediate feature representation of the generator network .

The decoder includes five deconvolution structures. Each deconvolution structure serves as an upsampling layer and consists of a 3*3 convolution kernel with a stride of 1 and a ReLU activation function, which is used to correct the input distortion feature map. Perform upsampling.

The output end of each deconvolution structure is connected to an image processing structure including a convolution layer, Tanh activation function and torgb function. The convolution layer uses a 1x1 convolution with a step size of 1 to perform a convolution operation on the corrected distortion feature map. And carry out Tanh activation, and then map the corrected distortion feature map of the grayscale state into a 3-channel RGB image through the torgb function.

The image correction structure is a convolution layer with a 3×3 kernel. The corrected distortion feature map is input to the convolution layer for convolution operation to obtain a multi-scale corrected image. The sizes are specifically 8, 16, 32, 64, and 128. ,.

The discriminator includes four convolutional network structures. Each convolutional network structure includes a convolutional layer, a LeakyReLU activation function and a spectral normalization layer. The convolutional layer in the first three convolutional network structures has a step size of 2. 5*5 size convolution kernel, the convolution layer in the last convolutional network structure has a 5*5 size convolution kernel with a stride of 1. It is used to receive a corrected image as input and gradually extract features so that the discriminator can identify whether the input corrected image is a real image. The number of output channels of the convolutional layer is 1 because the task of the discriminator is binary classification, which decides whether the input data is real data 1 or fake data generated by the generator 0.

In this embodiment, a spectral normalization layer is introduced in the discriminator to constrain the weight matrix of the discriminator network, thereby improving the stability and training efficiency of the network model. The spectral normalization layer is mainly used in the weight layer of the discriminator (such as convolutional layer, fully connected layer, etc.), and its function is to limit the spectral norm of the weight matrix.

Each column or row of the weight matrix is normalized by a mathematical operation that ensures that the largest singular value (the square root of the largest eigenvalue) of the matrix is equal to a predefined value, usually 1. This process adjusts the weights before each parameter update, so the network's weights are limited to a fixed range. Specifically reflected in the following:

(1) Spectral norm restriction of weights: For each weight matrix (weight layer), the spectral normalization layer constrains its spectral norm by converting it into a weight matrix with unit norm, that is, it ensures that the matrix The maximum singular value (the square root of the maximum eigenvalue) is equal to 1;

(2) Dynamic adjustment: Spectral normalization is a dynamic process. The weights are adjusted before each parameter update; specifically, it calculates the maximum singular value of the weight matrix and adjusts each column of the weight matrix. (or row) is scaled accordingly to make it comply with the set maximum singular value;

(3) Position in the network: The spectral normalization layer is usually placed before the weight layer of the discriminator network. As part of these layers, it can be directly applied to the convolutional layer or fully connected layer of the network. By adjusting these The weight of the layer is used to limit its spectral norm.

In this embodiment, the SKNet attention mechanism layer is introduced into the encoder. The SKNet attention mechanism layer is divided into three parts: split, fuse, and select. The first part is splitting. For any given feature map X∈C×h×w, by default, M (M=2 in the figure) branches (actually convolutions, corresponding to the SKNet structure diagram in Figure 2 X to two feature maps U). The kernel sizes of the two feature maps are 3×3 and 5×5 respectively. The convolution here is composed of effective group/depth convolution, batch normalization and Relu function in sequence. It is nothing more than using depth convolution and grouped convolution on two branches, and one branch is 3×3Conv, and the other is 5×5Conv (actually a 3x3, D=2 hole convolution).

The second part of Fuse is feature fusion. After adding the feature maps U of the two branches of Split, global average pooling is performed to obtain a 1×1×C feature map (s in the picture). s is passed through FC (fully connected ) layer aggregation and compression to obtain a d×1 vector (z in the figure). The value of d is determined by the formula d=max(C/r,L). r is a reduction ratio, and L represents the minimum value of d. The value of L in the experiment is 32.

In the third part, Select, cross-channel soft attention is used to adaptively select different spatial scales of information, which is guided by compressed features z. The vector z is transformed back to length C through M FC layers (schematic diagram M=2). Softmax is calculated for the two vectors in the channel dimension to obtain their respective weight vectors, and the weights are multiplied by the two outputs of Split. The refined feature map is obtained, and the final output V is obtained by summing the two refined feature maps. Specifically, the softmax operator is applied to the channel numbers: the resulting z is passed through two functions ac and bc, and the resulting function value is multiplied by the original U1 and U2. Since the sum of the function values of ac and bc is equal to 1, it is possible to set the weight of the feature map in the branch. Because the convolution kernel sizes of different branches are different, it is possible to let the network choose the appropriate convolution kernel (ac and bc). The A and B matrices in need to be initialized before training, and their sizes are C×d).

Forecasted traffic is used to perform the following spatial transformations,

is the distortion feature map output by the convolutional network structure in the distortion correction network model, and the corresponding corrected distortion feature map is/>

Based on the visualization results of the feature map, this embodiment proposes a gradually complementary mechanism. In the encoder part of the distortion correction network model, continuous convolution and pooling operations will blur the edges of the feature map, causing the degree of distortion to gradually shrink. In the decoder part of the flow estimation network model, the distortion features passed by skip connections force the predicted flows to have larger displacements to represent larger distortions. Intuitively, the degree of deformation and displacement can be expressed as follows:

where D is a function that estimates the degree of distortion of the input features, and M is a function that estimates the degree of displacement of the appearance flow. The greater the image distortion, the greater the displacement required for correction. Furthermore, c and k are constants,

It can be seen from the formula that and/> There is an obvious hierarchical correspondence between them. They are the same size as 128, 64, 32, 16, 8.

Through progressive complementation, the appearance flow can be used to pre-correct the distorted feature map.

In this embodiment, the distortion correction network model uses the appearance flow to correct the corresponding distortion features. The rectified image features are monitored by multi-scale loss to enhance the performance. With the help of the gradual complementation mechanism, the distorted feature map on the decoder has been roughly corrected. The distortion correction network model extracts features on the encoder. Since the encoder features contain distortion, we consider them as distortion features. The features of each layer are sent to the rectification layer, which is pre-rectified using the corresponding appearance stream. Thereafter, with the help of corrective features, the decoder can focus on content reconstruction.

The feature maps used for concatenation at the decoder have been roughly rectified with the help of a progressive complementation mechanism. The corrected feature map can bring a lot of details without conveying distortion structure. As a result, the network can generate visually more realistic rectified images. The working process of the distortion correction network model can be expressed as:

in represents the i-th layer corrected image, and Gc represents the distortion correction network model. In order to ensure the quality of correction, this embodiment sends the cascaded features to a convolutional layer with a 3×3 kernel to obtain multi-scale rectified images, and downsamples the real images at the same scale to supervise them.

Step S400 model training: perform model training on the flow estimation network model and the distortion correction network model based on the sample set, and obtain the trained flow estimation network model and the trained distortion correction network model.

In this embodiment, Wasserstein is introduced as the loss function between the generator and the discriminator, and the Wasserstein distance is used to measure the difference between the generated image and the real image, completely solving the problem of unstable GAN training and no longer needing to carefully balance the generator. and the training level of the discriminator.

For the scaling loss function, the existing loss function construction method can be used to construct the loss function based on the corrected image and the real image.

Step S500 Image correction: Taking the fisheye image to be corrected as input, the trained flow estimation network model and the trained distortion correction network model are used to predict and output the category label and category probability of each scale-corrected image, and based on the category category probability, the Final corrected image.

The method of this embodiment separates correction and content reconstruction, and it includes two network models: traffic estimation and distortion correction. First, a flow estimation network model estimates the distorted image structure and represents the result as a dense appearance flow. Second, the distortion correction network model utilizes a process to correct distorted features and uses the corrected features to reconstruct plausible results. In order to connect these two network models, a progressive complementary mechanism is introduced to achieve multi-level correction. The SKNet mechanism is added to the distortion correction network model to help extract features, solve the problem of multi-scale information fusion, and enable the network to adapt to different Characteristics of scale. At the same time, the spectrum is normalized into the discriminator, and the size of the weight is controlled by constraining the spectral norm of the weight matrix of the network, thus reducing the problems of steepness and disappearance, and helping to train deep neural networks more easily. Networks, especially in the complex game between generator and discriminator, make the generated images more realistic and natural.

Example 2:

The present invention is a fisheye image correction system based on the SKNet attention mechanism, including an image acquisition module, a flow estimation model building module, a distortion correction model building module, a model training module and an image correction module. The system executes the method disclosed in Embodiment 1 .

The image acquisition module is used to perform the following: obtain real images and corresponding fisheye images to construct a sample set.

The flow estimation model building module is used to perform the following: construct a flow estimation network model, which is a U-shaped encoder-decoder structure, and is used to extract appearance features from the input fisheye image and convert the appearance feature map Map to a stream and output a multi-scale appearance stream.

In the flow estimation network model of this embodiment, the encoder includes N convolution structures, and the decoder includes N-1 deconvolution structures. Among them, each convolution structure as a down-sampling structure includes a convolution layer, a normalization layer and a Leaky ReLU activation function connected in sequence, which is used to perform down-sampling operations and appearance feature extraction on the input image, and output the appearance feature map; each A deconvolution structure as an upsampling structure includes a residual module and a deconvolution layer, which is used to upsample the input appearance feature map and output appearance feature maps of N-1 scales; each deconvolution The output end of the structure is connected to a convolution layer. The convolution layer is used to perform a convolution operation on the input appearance feature map and map the appearance feature map into a stream to obtain the appearance stream.

As a specific implementation of the flow estimation network model, the encoder is composed of 6 convolutional layers. Each layer contains 1 convolution, normalization and LReLu activation function activation and other operations, which are used to downsample the image and extract features. The resolution of the input image is reduced, and the 256×256×3 size image is downsampled to a 4×4×512 size.

The decoder consists of 5 deconvolution layers, each layer contains 1 residual block and 1 deconvolution operation, which is used to upsample the image and restore the image to its original resolution.

The flow estimation module utilizes an encoder-decoder structure to extract features and generate a series of shape flows. A convolutional layer is provided at the output end of the decoder. The convolutional layer uses a 3x3 convolution kernel to process the extracted appearance feature map to obtain an appearance stream of two channels. The feature maps of different resolutions are mapped to the stream. The output, in this way, is obtained 5 appearance streams with sizes of 128, 64, 32, 16, and 8.

The flow estimation network model evaluates the degree of distortion of the fisheye image and renders it as an apparent flow. The network model uses an encoder-decoder structure to extract features and generate sequences. Specifically, the output features of the deconvolution structure of each decoder undergo an additional 3x3 convolution operation to obtain an appearance stream of two channels, and finally obtain an appearance stream of 5 graphics.

where I _in is the input fisheye image, G _s presents the appearance flow estimation module, is the appearance stream output by the i-th deconvolution structure.

The distortion correction model building module is used to perform the following: construct a distortion correction network model. The distortion correction network model includes a generator, a correction layer and a discriminator. The generator includes a U-shaped encoder-decoder structure. The encoder introduces The SKNet attention mechanism layer is used to extract distortion features of the input fisheye image and output multi-scale distortion feature maps; the correction layer is used to take multi-scale appearance flow and multi-scale distortion feature maps as input, and based on the appearance flow pair Correct the distortion feature map of the corresponding scale, and output the multi-scale corrected distortion feature map; the encoder is used to perform image reconstruction based on the input multi-scale corrected distortion feature map, and output the multi-scale corrected image; the discriminator introduces spectral The normalization layer is used to take the multi-scale corrected image as input, determine whether the corrected image at each scale is a real image, and predict and output the category label of the corrected image at each scale and the corresponding category probability. The category label includes the real image. images and fake images.

The distortion correction model in this embodiment is similar to a GAN network. The generator includes an encoder, a decoder, a correction layer, an image processing structure and an image correction structure. The encoder includes N convolution structures and N-1 deconvolution structures. The decoder Includes N-1 deconvolution structures.

For the first N-1 convolution structures, each convolution structure includes a convolution layer and a Leaky ReLU activation function as a down-sampling structure, and each down-sampling structure introduces a sknet attention mechanism layer for input fisheye The image is subjected to downsampling operation and distortion feature extraction, and a distortion feature map of the corresponding scale is output; for the Nth convolution structure, the convolution structure includes a Leaky ReLU activation function, which is used to perform feature transformation on the input distortion feature map and output Distortion feature map; the correction layer is connected to the flow estimation network model based on the asymptotic complementation mechanism and is used to perform the following: taking the multi-scale appearance flow and the multi-scale distortion feature map as input, for the distortion feature map of each scale, based on the appearance of the corresponding scale The flow performs spatial transformation on the distortion feature map to correct the distortion feature map, and outputs the corrected distortion feature map of the corresponding scale, in which the appearance flow and the distortion feature map both share N channels and correspond one to one; for each deconvolution structure, the As an upsampling structure, the convolution structure includes a convolution layer and a ReLU activation function, which is used to upsample the corrected distortion feature map of the corresponding scale and output the corrected distortion feature map of the corresponding scale; the output of each deconvolution structure The terminal is equipped with an image processing structure. The image processing structure includes a convolution layer, Tanh activation function and torgb function, which is used to perform convolution and activation operations on the input corrected distortion feature map, and convert the corrected distortion feature map through the togb function. Convert to an RGB image to obtain a corrected distortion feature map in the form of an RGB image; the image correction structure is a convolutional layer, which is used to take the corrected distortion feature map in the form of a multi-scale RGB image as input, and calculate the corrected distortion features of each scale The image is downsampled and the multi-scale corrected image is output.

The discriminator includes M convolutional network structures. Each convolutional network structure includes a convolution layer, a LeakyReLU activation function and a spectral normalization layer. It is used to take the corrected image at each scale as input and determine whether the corrected image is Real images, output class labels and class probability, class labels include real images and fake images.

In order to solve the problem of distortion divergence, this embodiment inserts a feature correction layer into the generator to pre-correct image features before transmission. At the same time, a SKNet attention mechanism layer is added to the encoder of the generator to improve the experience of the model. With the field and feature selection capabilities, it can be more suitable for processing complex image data. The predicted appearance flow is used to perform spatial transformation to obtain the corrected feature map.

As a specific implementation, the main part of the generator is a repair generation network, which is composed of an encoder and a decoder. The encoder extracts features from the fisheye image to obtain a multi-scale distortion feature map. For each distortion feature map, Correction is performed through the appearance flow to obtain the corrected distortion feature map, and image reconstruction is performed based on the corrected distortion feature map to generate the corrected image.

Among them, the encoder includes six convolution structures, but for the first five convolution structures, each convolution structure includes a convolution layer and a Leaky ReLU activation function as a downsampling layer, and the convolution kernel size is 3*3 , with a step size of 1; the last convolution structure, using the ReLU activation function but without downsampling, is used to perform feature transformation on the extracted feature maps. The output of these convolution structures will be used as the intermediate feature representation of the generator network .

The decoder includes five deconvolution structures. Each deconvolution structure serves as an upsampling layer and consists of a 3*3 convolution kernel with a stride of 1 and a ReLU activation function, which is used to correct the input distortion feature map. Perform upsampling.

The output end of each deconvolution structure is connected to an image processing structure including a convolution layer, Tanh activation function and torgb function. The convolution layer uses a 1x1 convolution with a step size of 1 to perform a convolution operation on the corrected distortion feature map. And carry out Tanh activation, and then map the corrected distortion feature map of the grayscale state into a 3-channel RGB image through the torgb function.

The image correction structure is a convolution layer with a 3×3 kernel. The corrected distortion feature map is input to the convolution layer for convolution operation to obtain a multi-scale corrected image. The sizes are specifically 8, 16, 32, 64, and 128. ,.

The discriminator includes four convolutional network structures. Each convolutional network structure includes a convolutional layer, a LeakyReLU activation function and a spectral normalization layer. The convolutional layer in the first three convolutional network structures has a step size of 2. 5*5 size convolution kernel, the convolution layer in the last convolutional network structure has a 5*5 size convolution kernel with a stride of 1. It is used to accept a corrected image as input and gradually extract features so that the discriminator can identify whether the input corrected image is a real image. The number of output channels of the convolutional layer is 1 because the task of the discriminator is binary classification, which decides whether the input data is real data 1 or fake data generated by the generator 0.

In this embodiment, the SKNet attention mechanism layer is introduced into the encoder. The SKNet attention mechanism layer is divided into three parts: split, fuse, and select. The first part is splitting. For any given feature map X∈C×h×w, by default, M (M=2 in the figure) branches (actually convolutions, corresponding to the SKNet structure diagram in Figure 2 X to two feature maps U). The kernel sizes of the two feature maps are 3×3 and 5×5 respectively. The convolution here is composed of effective group/depth convolution, batch normalization and Relu function in sequence. It is nothing more than using depth convolution and grouped convolution on two branches, and one branch is 3×3Conv, and the other is 5×5Conv (actually a 3x3, D=2 hole convolution).

The second part of Fuse is feature fusion. After adding the feature maps U of the two branches of Split, global average pooling is performed to obtain a 1×1×C feature map (s in the picture). s is passed through FC (fully connected ) layer aggregation and compression to obtain a d×1 vector (z in the figure). The value of d is determined by the formula d=max(C/r,L). r is a reduction ratio, and L represents the minimum value of d. The value of L in the experiment is 32.

In the third part, Select, cross-channel soft attention is used to adaptively select different spatial scales of information, which is guided by compressed features z. The vector z is transformed back to length C through M FC layers (schematic diagram M=2). Softmax is calculated for the two vectors in the channel dimension to obtain their respective weight vectors, and the weights are multiplied by the two outputs of Split. The refined feature map is obtained, and the final output V is obtained by summing the two refined feature maps. Specifically, the softmax operator is applied to the channel numbers: the resulting z is passed through two functions ac and bc, and the resulting function value is multiplied by the original U1 and U2. Since the sum of the function values of ac and bc is equal to 1, it is possible to set the weight of the feature map in the branch. Because the convolution kernel sizes of different branches are different, it is possible to let the network choose the appropriate convolution kernel (ac and bc). The A and B matrices in need to be initialized before training, and their sizes are C×d).

Forecasted traffic is used to perform the following spatial transformations,

is the distortion feature map output by the convolutional network structure in the distortion correction network model, and the corresponding corrected distortion feature map is/>

Based on the visualization results of the feature map, this embodiment proposes a gradually complementary mechanism. In the encoder part of the distortion correction network model, continuous convolution and pooling operations will blur the edges of the feature map, causing the degree of distortion to gradually shrink. In the decoder part of the flow estimation network model, the distortion features passed by skip connections force the predicted flows to have larger displacements to represent larger distortions. Intuitively, the degree of deformation and displacement can be expressed as follows:

where D is a function that estimates the degree of distortion of the input features, and M is a function that estimates the degree of displacement of the appearance flow. The greater the image distortion, the greater the displacement required for correction. Furthermore, c and k are constants,

It can be seen from the formula that and/> There is an obvious hierarchical correspondence between them. They are the same size as 128, 64, 32, 16, 8.

Through progressive complementation, the appearance flow can be used to pre-correct the distorted feature map.

In this embodiment, the distortion correction network model uses the appearance flow to correct the corresponding distortion features. The rectified image features are monitored by multi-scale loss to enhance the performance. With the help of the gradual complementation mechanism, the distorted feature map on the decoder has been roughly corrected. The distortion correction network model extracts features on the encoder. Since the encoder features contain distortion, they are treated as distortion features. The features of each layer are sent to the rectification layer, which is pre-rectified using the corresponding appearance stream. Thereafter, with the help of corrective features, the decoder can focus on content reconstruction.

The feature maps used for concatenation at the decoder have been roughly rectified with the help of a progressive complementation mechanism. The corrected feature map can bring a lot of details without conveying distortion structure. As a result, the network can generate visually more realistic rectified images. The working process of the distortion correction network model can be expressed as:

in represents the i-th layer corrected image, and Gc represents the distortion correction network model. In order to ensure the quality of correction, this embodiment sends the cascaded features to a convolutional layer with a 3×3 kernel to obtain multi-scale rectified images, and downsamples the real images at the same scale to supervise them.

The model training module is used to perform the following: perform model training on the flow estimation network model and the distortion correction network model based on the sample set, and obtain the trained flow estimation network model and the trained distortion correction network model.

In this embodiment, Wasserstein is introduced as the loss function between the generator and the discriminator, and the Wasserstein distance is used to measure the difference between the generated image and the real image, completely solving the problem of unstable GAN training and no longer needing to carefully balance the generator. and the training level of the discriminator.

For the scaling loss function, the existing loss function construction method can be used to construct the loss function based on the corrected image and the real image.

The image correction module is used to perform the following: taking the fisheye image to be corrected as input, predicting and outputting the category label and category probability of the corrected image at each scale through the trained flow estimation network model and the trained distortion correction network model, based on the category The class probability is used to obtain the final corrected image.

The present invention has been shown and described in detail through the drawings and preferred embodiments above. However, the present invention is not limited to these disclosed embodiments. Based on the above-mentioned multiple embodiments, those skilled in the art will know that the above-mentioned different embodiments can be combined. The code review means in the method can lead to more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims (10)

1. A fisheye image correction method based on a SKNet attention mechanism is characterized by comprising the following steps:

and (3) image acquisition: acquiring a real image and a corresponding fish-eye image to construct a sample set;

and (3) constructing a flow estimation model: constructing a stream estimation network model, wherein the stream estimation network model is of a U-shaped encoder-decoder structure and is used for extracting appearance characteristics of an input fisheye image, mapping the appearance characteristic image into a stream and outputting a multi-scale appearance stream;

And (3) constructing a distortion correction model: the method comprises the steps of constructing a distortion correction network model, wherein the distortion correction network model comprises a generator, a correction layer and a discriminator, the generator comprises a U-shaped encoder-decoder structure, and an SKNet attention mechanism layer is introduced into an encoder for carrying out distortion feature extraction on an input fisheye image and outputting a multi-scale distortion feature map; the correction layer is used for correcting the distortion characteristic map of the corresponding scale based on the appearance flow by taking the multi-scale appearance flow and the multi-scale distortion characteristic map as input, and outputting the distortion characteristic map after multi-scale correction; the encoder is used for reconstructing an image based on the input multi-scale corrected distortion characteristic image and outputting a multi-scale corrected image; the discriminator is introduced with a spectrum normalization layer and is used for taking the multi-scale corrected image as input, judging whether each scale corrected image is a real image or not, and predicting and outputting a class label and a corresponding class probability of each scale corrected image, wherein the class label comprises the real image and a fake image;

model training: model training is carried out on the stream estimation network model and the distortion correction network model based on the sample set, and a trained stream estimation network model and a trained distortion correction network model are obtained;

Image correction: and taking the fisheye image to be corrected as input, predicting and outputting the class label and class probability of each scale corrected image through the trained stream estimation network model and the trained distortion correction network model, and obtaining the final corrected image based on the class probability.

2. The SKNet attention mechanism based fisheye image correction method of claim 1 wherein for the stream estimation network model, the encoder comprises N convolution structures and the decoder comprises N-1 deconvolution structures;

each convolution structure comprises a convolution layer, a normalization layer and a leakage ReLU activation function which are sequentially connected as a downsampling structure, and is used for performing downsampling operation and appearance feature extraction on an input image and outputting an appearance feature image;

each deconvolution structure comprises a residual error module and a deconvolution layer as an up-sampling structure, and is used for up-sampling an input appearance characteristic diagram and outputting an appearance characteristic diagram with N-1 scales;

the output end of each deconvolution structure is connected with a convolution layer, and the convolution layer is used for carrying out convolution operation on the input appearance characteristic images and mapping the appearance characteristic images into streams to obtain appearance streams.

3. The SKNet attention mechanism based fisheye image correction method of claim 1 wherein the generator comprises an encoder, a decoder, a correction layer, an image processing structure, and an image correction structure, the encoder comprising N convolutions and N-1 deconvolutions, the decoder comprising N-1 deconvolutions;

for the first N-1 convolution structures, each convolution structure comprises a convolution layer and a Leaky ReLU activation function as a downsampling structure, and each downsampling structure is introduced with a sknet attention mechanism layer and is used for downsampling an input fisheye image and extracting distortion characteristics and outputting a distortion characteristic diagram with a corresponding scale;

for the Nth convolution structure, the convolution structure comprises a Leaky ReLU activation function, and is used for carrying out feature transformation on an input distortion feature map and outputting the distortion feature map;

the correction layer is connected with the flow estimation network model based on a progressive complementary mechanism and is used for executing the following steps: taking the multi-scale appearance flow and the multi-scale distortion feature map as inputs, and for each scale distortion feature map, performing spatial transformation on the distortion feature map based on the appearance flow of the corresponding scale to correct the distortion feature map, and outputting corrected distortion feature maps of the corresponding scale, wherein the appearance flow and the distortion feature map are in N paths and are in one-to-one correspondence;

For each deconvolution structure, the convolution structure comprises a convolution layer and a ReLU activation function as an up-sampling structure, and is used for up-sampling the corrected distortion characteristic diagram of the corresponding scale and outputting the corrected distortion characteristic diagram of the corresponding scale;

the output end of each deconvolution structure is provided with an image processing structure, and the image processing structure comprises a convolution layer, a Tanh activation function and a torgb function and is used for carrying out convolution operation and activation operation on an input corrected distortion characteristic image and converting the corrected distortion characteristic image into an RGB image through the torgb function to obtain the corrected distortion characteristic image in the form of an RGB image;

the image correction structure is a convolution layer and is used for taking the corrected distortion characteristic image in the form of a multi-scale RGB image as input, performing downsampling operation on the corrected distortion characteristic image of each scale and outputting the multi-scale corrected image.

4. The SKNet attention mechanism-based fisheye image correction method of claim 1, wherein the discriminator comprises M convolutional network structures, each comprising a convolutional layer, a LeakyReLU activation function, and a spectral normalization layer, for taking each scale corrected image as input, determining whether the corrected image is a real image, and outputting a class label and class probability, the class label comprising a real image and a counterfeit image.

5. The fisheye image correction method based on SKNet attention mechanism according to claim 1, wherein an fight loss function is constructed by a waserstein distance loss function based on the multi-scale corrected image and the real image of the corresponding scale, the fight loss function being used for model training of the generator and the discriminator;

constructing a multi-scale loss function based on the multi-scale corrected image and the real image with the corresponding scale, weighting the counter-scale loss function and the scale loss function as the total loss function of the flow estimation network model and the distortion correction network model, and performing model training on the flow estimation network model and the distortion correction network model based on the total loss function to obtain a trained flow estimation network model and a trained distortion correction network model.

6. A fisheye image correction system based on a SKNet attention mechanism, characterized in that the system is used for realizing fisheye image correction by a fisheye image correction method based on a SKNet attention mechanism according to any one of claims 1-5, and comprises an image acquisition module, a flow estimation model construction module, a distortion correction model construction module, a model training module and an image correction module;

The image acquisition module is used for executing the following steps: acquiring a real image and a corresponding fish-eye image to construct a sample set;

the flow estimation model building module is configured to perform the following: constructing a stream estimation network model, wherein the stream estimation network model is of a U-shaped encoder-decoder structure and is used for extracting appearance characteristics of an input fisheye image, mapping the appearance characteristic image into a stream and outputting a multi-scale appearance stream;

the distortion correction model building module is used for executing the following steps: the method comprises the steps of constructing a distortion correction network model, wherein the distortion correction network model comprises a generator, a correction layer and a discriminator, the generator comprises a U-shaped encoder-decoder structure, and an SKNet attention mechanism layer is introduced into an encoder for carrying out distortion feature extraction on an input fisheye image and outputting a multi-scale distortion feature map; the correction layer is used for correcting the distortion characteristic map of the corresponding scale based on the appearance flow by taking the multi-scale appearance flow and the multi-scale distortion characteristic map as input, and outputting the distortion characteristic map after multi-scale correction; the encoder is used for reconstructing an image based on the input multi-scale corrected distortion characteristic image and outputting a multi-scale corrected image; the discriminator is introduced with a spectrum normalization layer and is used for taking the multi-scale corrected image as input, judging whether each scale corrected image is a real image or not, and predicting and outputting a class label and a corresponding class probability of each scale corrected image, wherein the class label comprises the real image and a fake image;

The model training module is used for executing the following steps: model training is carried out on the stream estimation network model and the distortion correction network model based on the sample set, and a trained stream estimation network model and a trained distortion correction network model are obtained;

the image correction module is used for executing the following steps: and taking the fisheye image to be corrected as input, predicting and outputting the class label and class probability of each scale corrected image through the trained stream estimation network model and the trained distortion correction network model, and obtaining the final corrected image based on the class probability.

7. The SKNet attention mechanism based fisheye image correction system of claim 6 wherein for the stream estimation network model, the encoder comprises N convolution structures and the decoder comprises N-1 deconvolution structures;

each convolution structure comprises a convolution layer, a normalization layer and a leakage ReLU activation function which are sequentially connected as a downsampling structure, and is used for performing downsampling operation and appearance feature extraction on an input image and outputting an appearance feature image;

each deconvolution structure comprises a residual error module and a deconvolution layer as an up-sampling structure, and is used for up-sampling an input appearance characteristic diagram and outputting an appearance characteristic diagram with N-1 scales;

The output end of each deconvolution structure is connected with a convolution layer, and the convolution layer is used for carrying out convolution operation on the input appearance characteristic images and mapping the appearance characteristic images into streams to obtain appearance streams.

8. The SKNet attention mechanism based fisheye image correction system of claim 6 wherein the generator comprises an encoder, a decoder, a correction layer, an image processing structure, and an image correction structure, the encoder comprising N convolutions and N-1 deconvolutions, the decoder comprising N-1 deconvolutions;

for the first N-1 convolution structures, each convolution structure comprises a convolution layer and a Leaky ReLU activation function as a downsampling structure, and each downsampling structure is introduced with a sknet attention mechanism layer and is used for downsampling an input fisheye image and extracting distortion characteristics and outputting a distortion characteristic diagram with a corresponding scale;

for the Nth convolution structure, the convolution structure comprises a Leaky ReLU activation function, and is used for carrying out feature transformation on an input distortion feature map and outputting the distortion feature map;

the correction layer is connected with the flow estimation network model based on a progressive complementary mechanism and is used for executing the following steps: taking the multi-scale appearance flow and the multi-scale distortion feature map as inputs, and for each scale distortion feature map, performing spatial transformation on the distortion feature map based on the appearance flow of the corresponding scale to correct the distortion feature map, and outputting corrected distortion feature maps of the corresponding scale, wherein the appearance flow and the distortion feature map are in N paths and are in one-to-one correspondence;

For each deconvolution structure, the convolution structure comprises a convolution layer and a ReLU activation function as an up-sampling structure, and is used for up-sampling the corrected distortion characteristic diagram of the corresponding scale and outputting the corrected distortion characteristic diagram of the corresponding scale;

the output end of each deconvolution structure is provided with an image processing structure, and the image processing structure comprises a convolution layer, a Tanh activation function and a torgb function and is used for carrying out convolution operation and activation operation on an input corrected distortion characteristic image and converting the corrected distortion characteristic image into an RGB image through the torgb function to obtain the corrected distortion characteristic image in the form of an RGB image;

the image correction structure is a convolution layer and is used for taking the corrected distortion characteristic image in the form of a multi-scale RGB image as input, performing downsampling operation on the corrected distortion characteristic image of each scale and outputting the multi-scale corrected image.

9. The SKNet attention mechanism based fisheye image correction system of claim 6, wherein the discriminator comprises M convolutional network structures, each comprising a convolutional layer, a LeakyReLU activation function, and a spectral normalization layer for taking each scale corrected image as input, determining if the corrected image is a real image, outputting a class label and a class probability, the class label comprising a real image and a counterfeit image.

10. The SKNet attention mechanism based fisheye image correction system of claim 6, wherein the model training module is configured to construct an fight loss function based on the multi-scale corrected image and the corresponding scale real image via a wasperstein distance loss function, the fight loss function being configured to model the generator and the discriminator;

the model training module is used for constructing a multi-scale loss function based on the multi-scale corrected image and the real image with the corresponding scale, weighting the anti-loss function and the scale loss function as the total loss function of the flow estimation network model and the distortion correction network model, and performing model training on the flow estimation network model and the distortion correction network model based on the total loss function to obtain a trained flow estimation network model and a trained distortion correction network model.