CN115412669B - Fog imaging method and device based on image signal-to-noise ratio analysis - Google Patents

Abstract

The present application discloses a fog imaging method and device based on image signal-to-noise ratio analysis, wherein the method includes: obtaining the actual fog concentration or actual fog visibility of the current fog scene; Use the pre-built minimum resolution model of the fog imaging system to determine the current required viewing angle, current time frame number and current camera bit depth of the fog imaging system; use the pre-trained signal-to-noise ratio model to optimize the current required fog imaging system The required viewing angle, current time frame number and other preset camera parameters, and according to the optimized current required viewing angle, current time frame number, other preset camera parameters and current camera bit depth, get the current object scene Foggy imaging results. As a result, the technical problem that in the related art, the dehazing algorithm only considers the ideal noise-free situation, and the target cannot be reconstructed when the fog concentration is high and the sensor noise cannot be ignored, is solved.

Description

Foggy weather imaging method and device based on image signal-to-noise ratio analysis

Technical Field

The present application relates to the technical field of computer vision and digital image processing, and in particular to a foggy weather imaging method and device based on image signal-to-noise ratio analysis.

Background Art

Imaging through fog is one of the current research hotspots in the imaging field and has broad application prospects in security monitoring, autonomous driving, and other aspects.

In related technologies, defogging work only considers defogging algorithms and system construction solutions under ideal noise-free conditions. However, when the fog becomes thicker and the attenuation is severe, the effective ballistic light signal of the target will decay exponentially at e ^-βu . At this time, sensor noise can no longer be ignored. Even when the effective signal is severely attenuated, noise has become the main reason why the target cannot be reconstructed under dense fog. Therefore, there is an urgent need for a solution to improve the signal-to-noise ratio of fog images.

Summary of the invention

The present application provides a foggy weather imaging method and device based on image signal-to-noise ratio analysis to solve the technical problem in the related art that the defogging algorithm only considers the ideal noise-free situation, but cannot reconstruct the target object when the fog concentration is high and the sensor noise cannot be ignored.

The first aspect of the present application provides a fog imaging method based on image signal-to-noise ratio analysis, comprising the following steps: obtaining the actual fog concentration or actual fog visibility of the current fog scene; based on the actual fog concentration or the actual fog visibility and the observed object distance, determining the current required number of viewing angles, the current number of time frames and the bit depth of the current camera of the fog imaging system using a pre-built minimum resolution model of the fog imaging system; and optimizing the current required number of viewing angles, the current number of time frames and other preset camera parameters of the fog imaging system using a pre-trained signal-to-noise ratio model, and obtaining the fog imaging result of the current fog scene based on the optimized current required number of viewing angles, the current number of time frames, the preset other camera parameters and the bit depth of the current camera.

Optionally, in one embodiment of the present application, the use of a pre-trained signal-to-noise ratio model to optimize the current required number of viewing angles, the current number of time frames, and other preset camera parameters of the fog imaging system includes: increasing the full well capacity of the camera, reducing the pixel area, and adjusting the camera sensitivity to the minimum; while ensuring that the camera is not overexposed, increasing the light intensity, exposure time, spectral response function, and target reflectivity, reducing the lens aperture value and the observation object distance, so as to meet the current required number of viewing angles and the current number of time frames while achieving the purpose of optimizing the signal-to-noise ratio.

Optionally, in one embodiment of the present application, the minimum resolution model of the fog imaging system is:

Among them, _ηlin represents the empirical constant of the linear working interval, bit_depth represents the bit depth of the camera, β represents the scattering coefficient of fog, u represents the object distance from the camera to the target object to be observed, _NF represents the number of time frames fused by the system, and _NV represents the number of camera viewing angles of the system.

Optionally, in one embodiment of the present application, before using the pre-built minimum resolution model of the fog imaging system to determine the current required number of viewing angles, the current number of time frames and the bit depth of the current camera of the fog imaging system, it also includes: defining an imaging model under the camera Raw format image; based on the imaging model, establishing a model relationship between the hardware parameters involved in the shooting of the fog imaging system and the image signal-to-noise ratio, and determining the restrictions between the hardware parameters.

Optionally, in one embodiment of the present application, the signal-to-noise ratio square formula in the model relationship is:

Among them, SNR ² represents the square of the signal-to-noise ratio, _Spix represents the pixel area of the sensor, t represents the camera exposure time, SRF ^c represents the spectral response function of different color channels of the camera, A represents the scene light intensity, k represents the constant unknown proportional coefficient, fno represents the aperture value of the lens, J ^c represents the reflectivity of the target object corresponding to different color channels, BRDF(θ) represents the difference caused by the reflection characteristics of the target surface, and T(u) represents the difference caused by the light source type.

Optionally, in an embodiment of the present application, the overexposure limit in the limiting condition is:

Among them, C _full represents the full well capacity of the camera, and ISO represents the camera sensitivity.

The second aspect of the present application provides a fog imaging device based on image signal-to-noise ratio analysis, including: an acquisition module for acquiring the actual fog concentration or actual fog visibility of the current fog scene; a determination module for determining the current required number of viewing angles, the current number of time frames and the bit depth of the current camera of the fog imaging system based on the actual fog concentration or the actual fog visibility and the observed object distance using a pre-built minimum resolution model of the fog imaging system; and an analysis module for optimizing the current required number of viewing angles, the current number of time frames and other preset camera parameters of the fog imaging system using a pre-trained signal-to-noise ratio model, and obtaining the fog imaging result of the current fog scene based on the optimized current required number of viewing angles, the current number of time frames, other preset camera parameters and the bit depth of the current camera.

Optionally, in one embodiment of the present application, the analysis module includes: a first adjustment unit, used to increase the full well capacity of the camera, reduce the pixel area, and adjust the camera sensitivity to the minimum; a second adjustment unit, used to increase the light intensity, exposure time, spectral response function and target reflectivity, reduce the lens aperture value and the observed object distance, while ensuring that the camera is not overexposed, so as to meet the current required number of viewing angles and the current number of time frames, while achieving the purpose of optimizing the signal-to-noise ratio.

Optionally, in one embodiment of the present application, the minimum resolution model of the fog imaging system is:

Wherein, _ηlin represents an empirical constant of the linear working interval, bit_depth represents the bit depth of the camera, β represents the scattering coefficient of fog, u represents the object distance from the camera to the target object to be observed, _NF represents the number of time frames fused by the system, and _NV represents the number of camera viewing angles of the system.

Optionally, in one embodiment of the present application, it also includes: a definition module, used to define an imaging model under the camera Raw format image; a modeling module, used to establish a model relationship between the hardware parameters involved in the shooting of the fog imaging system and the image signal-to-noise ratio based on the imaging model, and determine the restriction conditions between the hardware parameters.

Optionally, in one embodiment of the present application, the signal-to-noise ratio square formula in the model relationship is:

Among them, SNR ² represents the square of the signal-to-noise ratio, _Spix represents the pixel area of the sensor, t represents the camera exposure time, SRF ^c represents the spectral response function of different color channels of the camera, A represents the scene illumination intensity, k represents the constant unknown proportional coefficient, fno represents the aperture value of the lens, J ^c represents the reflectivity of the target object corresponding to different color channels, BRDF(θ) represents the difference caused by the reflective characteristics of the target surface, and T(u) represents the difference caused by the light source type.

Optionally, in an embodiment of the present application, the overexposure limit in the limiting condition is:

Among them, C _full represents the full well capacity of the camera, and ISO represents the camera sensitivity.

The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the foggy weather imaging method based on image signal-to-noise ratio analysis as described in the above embodiment.

A fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned foggy weather imaging method based on image signal-to-noise ratio analysis.

The embodiments of the present application can be guided by suppressing noise and improving the image signal-to-noise ratio. Starting from the hardware level, a model relationship between the hardware parameters involved in the shooting of the fog imaging system and the image signal-to-noise ratio is established. At the same time, the relationship between the minimum resolution capability of the fog imaging system and the highest fog concentration that can be faced is established, and the hardware parameters of the fog imaging system are optimized in this way, and finally a fog imaging system for any fog scene is constructed, thereby increasing the fog concentration that can be imaged by the fog imaging system. In this way, the technical problem that the defogging algorithm in the related technology only considers the ideal noise-free situation, and cannot reconstruct the target object when the fog concentration is high and the sensor noise cannot be ignored is solved.

Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

FIG1 is a flow chart of a foggy weather imaging method based on image signal-to-noise ratio analysis according to an embodiment of the present application;

FIG2 is a schematic diagram showing the principle of a foggy weather imaging method based on image signal-to-noise ratio analysis according to an embodiment of the present application;

FIG3 is a flow chart of an offline model building method of a foggy weather imaging method based on image signal-to-noise ratio analysis according to an embodiment of the present application;

FIG4 is a flow chart of a foggy weather imaging method based on image signal-to-noise ratio analysis according to an embodiment of the present application;

FIG5 is a schematic diagram of the structure of a foggy weather imaging device based on image signal-to-noise ratio analysis according to an embodiment of the present application;

FIG6 is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application.

DETAILED DESCRIPTION

The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

The following describes the foggy weather imaging method and device based on image signal-to-noise ratio analysis of the embodiment of the present application with reference to the accompanying drawings. In view of the technical problem that the defogging algorithm in the related technology mentioned in the background technology center only considers the ideal noise-free situation, and when the fog concentration is high and the sensor noise cannot be ignored, the target object cannot be reconstructed, the present application provides a foggy weather imaging method based on image signal-to-noise ratio analysis. In this method, it can be guided by suppressing noise and improving the image signal-to-noise ratio. Starting from the hardware level, a model relationship between the hardware parameters involved in the shooting of the fog imaging system and the image signal-to-noise ratio is established. At the same time, the relationship between the minimum resolution capability of the fog imaging system and the highest fog concentration that can be faced is established, and the hardware parameters of the fog imaging system are optimized in this way, and finally the fog imaging system for any fog scene is constructed, thereby improving the fog concentration that can be imaged by the fog imaging system. As a result, the technical problem that the defogging algorithm in the related technology only considers the ideal noise-free situation, and when the fog concentration is high and the sensor noise cannot be ignored, the target object cannot be reconstructed is solved.

Specifically, FIG1 is a flow chart of a foggy weather imaging method based on image signal-to-noise ratio analysis provided in an embodiment of the present application.

As shown in FIG1 , the foggy weather imaging method based on image signal-to-noise ratio analysis includes the following steps:

In step S101, the actual fog concentration or actual fog visibility of the current foggy scene is obtained.

In actual implementation, the embodiments of the present application can obtain the actual fog concentration β ₀ or fog visibility Vis ₀ of the current open-air scene and determine the observed object distance u based on visual observation by technicians or through corresponding equipment such as a haze meter, visibility meter, water mist concentration meter, etc.

In step S102, based on the actual fog concentration or actual fog visibility and the observed object distance, the pre-built minimum resolution model of the fog imaging system is used to determine the current required viewing angles, the current time frame number and the bit depth of the current camera of the fog imaging system.

As a possible implementation method, the embodiment of the present application can use the pre-built minimum resolution model of the fog imaging system, based on the actual fog concentration _β0 or fog visibility _Vis0 of the current open-air scene, and the observed object distance u, to obtain the number of viewing angles _NF , the number of time frames _NV and the bit depth bit_depth of the camera required by the fog imaging system under the current fog concentration or actual fog visibility, so that the fog imaging system can adjust the hardware parameters for different fog concentrations, realize the construction of the fog imaging system, and ensure that the fog imaging system can be used for any fog scene, wherein _β0 and _Vis0 can be equivalently replaced, _β0 = 2.996/ _Vis0 .

Among them, a model relationship between the hardware parameters involved in the shooting of the fog imaging system and the image signal-to-noise ratio is established, and the restrictions between the hardware parameters are given. The minimum resolution capability of the fog imaging system is analyzed based on the hardware parameters, and the relationship between the minimum resolution capability and the highest fog concentration that can be faced is established.

It should be noted that the minimum resolution model of the pre-built fog imaging system will be elaborated in detail below.

Optionally, in one embodiment of the present application, before using a pre-built minimum resolution model of the fog imaging system to determine the current required number of viewing angles, the current number of time frames, and the bit depth of the current camera of the fog imaging system, it also includes: defining an imaging model under the camera Raw format image; based on the imaging model, establishing a model relationship between the hardware parameters involved in the shooting of the fog imaging system and the image signal-to-noise ratio, and determining the restrictions between the hardware parameters.

Specifically, the embodiment of the present application may define an imaging model under a camera Raw format image:

和

Where I ^c represents the intensity of a pixel in the Raw format image of the camera, c∈{R,G,B} represents the three color channels of the camera image: red, green, and blue, _Spix represents the pixel area of the sensor, t represents the exposure time of the camera, SRF ^c represents the spectral response function of different color channels of the camera, E ^c represents the irradiance of the sensor plane, g represents the camera gain, which is controlled by the adjustable parameter camera sensitivity ISO (ISO≥1), that is, g＝U _cam /ISO, where the unit of U _cam is the same as that of g and it is a fixed parameter of the camera, which is determined by the full well capacity C _full and the bit depth bit_depth when the camera leaves the factory, that is, 2 ^bit_depth ·U _cam ≈C _full , and n represents the sensor noise, which is mainly composed of three types of zero-mean noise: shot noise _ns , readout noise n _r and quantization noise n _Q , and their variances can be respectively

and

Considering the influence of camera gain g, the noise variance of the Raw format image I ^c can be expressed as:

Since fog imaging has sufficient illumination and the camera receives a large amount of light, the noise is mainly shot noise _ns , and the influence of the other two types of noise can be ignored. At the same time, the camera acquisition also needs to meet the non-overexposure requirement, that is:

为雾气下弹道光辐照度

和环境光辐照度E_s的加和，公式表述可以如下：In addition, the irradiance at the sensor plane

is the ballistic light irradiance under fog

The sum of the ambient light irradiance _Es can be expressed as follows:

Where A represents the scene illumination intensity, k represents a constant unknown proportional coefficient for the sake of simplicity, fno represents the aperture value of the lens, ^Jc represents the reflectivity of the target corresponding to different color channels, and its value range is ^Jc∈ (0,1), u represents the object distance from the camera to the target to be observed, β represents the scattering coefficient of fog, reflecting the fog concentration, and can also be replaced by the fog visibility Vis, that is, β=2.996/Vis, T(u) represents the difference caused by the light source type, BRDF(θ) represents the difference caused by the reflective characteristics of the target surface, and under natural light, all targets are regarded as diffuse reflection targets, Δl represents the radius of the spherical shell of the point light source, which depends on the specific point light source model.

可作为有效信号，即：In fog imaging, only the ballistic light part in I ^c

Can be used as a valid signal, namely:

The rest is the environmental interference term in I ^c , so the square signal-to-noise ratio SNR ² (I ^c ) of the fog image I ^c can be:

Optionally, in one embodiment of the present application, the signal-to-noise ratio square formula in the model relationship is:

Among them, SNR ² represents the square of the signal-to-noise ratio, _Spix represents the pixel area of the sensor, t represents the camera exposure time, SRF ^c represents the spectral response function of different color channels of the camera, A represents the scene light intensity, k represents the constant unknown proportional coefficient, fno represents the aperture value of the lens, J ^c represents the reflectivity of the target object corresponding to different color channels, BRDF(θ) represents the difference caused by the reflection characteristics of the target surface, and T(u) represents the difference caused by the light source type.

Furthermore, considering the signal-to-noise ratio improvement effect of multi-time frame fusion and multi-view fusion at the hardware level of fog imaging system under fog, _NF is used to represent the number of time frames fused by the system, and _NV is used to represent the number of camera views of the system. The final signal-to-noise ratio square formula can be:

Optionally, in one embodiment of the present application, the overexposure limit in the limiting condition is:

Among them, C _full represents the full well capacity of the camera, and ISO represents the camera sensitivity.

Optionally, in one embodiment of the present application, the minimum resolution model of the fog imaging system is:

Among them, _ηlin represents the empirical constant of the linear working range, bit_depth represents the bit depth of the camera, β represents the scattering coefficient of fog, u represents the object distance from the camera to the target object to be observed, _NF represents the number of time frames fused by the system, and _NV represents the number of camera viewing angles of the system.

In the actual implementation process, the embodiment of the present application can define the sensitivity threshold (AST) of the multi-frame multi-view fog imaging system:

至少要大于系统的灵敏度阈值(AST)，即：The definition of the resolution limit in fog imaging can be the minimum effective signal of the image under fog

At least greater than the system sensitivity threshold (AST), that is:

After substituting the system hardware parameters, that is:

In the absence of fog, the target object is imaged within the effective working range of the camera, that is:

Among them, η _lin represents the empirical constant of the linear working range, and η _lin ∈(0.2,0.6) is generally taken. Combining the above two formulas, the minimum resolution model of the fog imaging system can be obtained:

In step S103, the pre-trained signal-to-noise ratio model is used to optimize the current required number of viewing angles, the current number of time frames, and other preset camera parameters of the fog imaging system, and the fog imaging result of the current foggy scene is obtained based on the optimized current required number of viewing angles, the current number of time frames, other preset camera parameters, and the bit depth of the current camera.

As a possible implementation method, the embodiment of the present application may assume that the system needs to image a diffusely reflecting target object illuminated by point lighting under fog, and then formulate a hardware parameter optimization strategy based on the signal-to-noise ratio model, where the hardware parameters are other camera parameters excluding camera bit depth.

It should be noted that other preset camera parameters will be explained below.

Optionally, in one embodiment of the present application, a pre-trained signal-to-noise ratio model is used to optimize the current required number of viewing angles, the current number of time frames, and other preset camera parameters of the fog imaging system, including: increasing the full well capacity of the camera, reducing the pixel area, and adjusting the camera sensitivity to the minimum; while ensuring that the camera is not overexposed, increasing the light intensity, exposure time, spectral response function, and target reflectivity, and reducing the lens aperture value and observation object distance to meet the current required number of viewing angles and the current number of time frames while achieving the purpose of optimizing the signal-to-noise ratio.

Specifically, in order to improve the signal-to-noise ratio of the fog image, the embodiment of the present application can first increase the full well capacity C _full of the camera, reduce the pixel area _Spix , and adjust the camera sensitivity ISO to the minimum; secondly, under the premise that the camera is not actually overexposed, the embodiment of the present application can maximize the light intensity A, exposure time t, spectral response function SRF ^c and target reflectivity J ^c , and minimize the lens aperture value fno and object distance u, wherein, since the reflectivity J ^c , lens aperture value fno and object distance u have a larger power in the signal-to-noise ratio square formula, they need to be adjusted first compared with other parameters; finally, the embodiment of the present application can continue to increase _NF and _NV to improve the signal-to-noise ratio on the basis of satisfying the number of viewing angles and time frames required by the minimum resolution model.

The analysis process for other light source types and target reflectance characteristics is the same as above. By effectively adjusting the hardware parameters, the construction of a fog imaging system for any fog scene can be achieved.

With reference to FIGS. 2 to 4 , the working principle of the foggy weather imaging method based on image signal-to-noise ratio analysis according to an embodiment of the present application is described in detail with reference to an embodiment.

As shown in FIG2 , it is a logical schematic diagram of an embodiment of the present application. In the embodiment of the present application, the system viewing angle _NV , the system frame number _NF and the camera bit depth bit_depth can be determined by the minimum resolution model according to the fog concentration _β0 or the fog visibility _Vis0 and the observation object distance u. The system viewing angle _NV , the system frame number _NF and other camera parameters are optimized under the guidance of the signal-to-noise ratio model. The above two models are integrated to form a fog imaging system with improved signal-to-noise ratio for the purpose of denoising, thereby obtaining the fog imaging result of the current foggy scene.

The parameters of the hardware system may include: the number of viewing angles _NV , the number of frames _NF , the camera bit depth bit_depth, and other camera parameters (aperture, full well capacity, pixel size, ISO, etc.).

During the actual implementation process, the embodiments of the present application may include an offline model building part and an online application part.

As shown in FIG3 , the offline model building process of the embodiment of the present application may include the following steps:

Step S301: For foggy scenes, a model relationship between the hardware parameters involved in fog imaging system shooting and the image signal-to-noise ratio is established, and constraints between the hardware parameters are given.

Specifically, the embodiment of the present application may define an imaging model under a camera Raw format image:

和

Where I ^c represents the intensity of a pixel in the Raw format image of the camera, c∈{R,G,B} represents the three color channels of the camera image: red, green, and blue, _Spix represents the pixel area of the sensor, t represents the exposure time of the camera, SRF ^c represents the spectral response function of different color channels of the camera, E ^c represents the irradiance of the sensor plane, g represents the camera gain, which is controlled by the adjustable parameter camera sensitivity ISO (ISO≥1), that is, g＝U _cam /ISO, where the unit of U _cam is the same as that of g and it is a fixed parameter of the camera, which is determined by the full well capacity C _full and the bit depth bit_depth when the camera leaves the factory, that is, 2 ^bit_depth .U _cam ≈C _full , and n represents the sensor noise, which is mainly composed of three types of zero-mean noise: shot noise _ns , readout noise n _r and quantization noise n _Q , and their variances can be respectively

and

Considering the influence of camera gain g, the noise variance of the Raw format image I ^c can be expressed as:

Since fog imaging has sufficient illumination and the camera receives a large amount of light, the noise is mainly shot noise _ns , and the influence of the other two types of noise can be ignored. At the same time, the camera acquisition also needs to meet the non-overexposure requirement, that is:

为雾气下弹道光辐照度

和环境光辐照度E_s的加和，公式表述可以如下：In addition, the irradiance at the sensor plane

is the ballistic light irradiance under fog

The sum of the ambient light irradiance _Es can be expressed as follows:

Where A represents the scene illumination intensity, k represents a constant unknown proportional coefficient for the sake of simplicity, fno represents the aperture value of the lens, ^Jc represents the reflectivity of the target corresponding to different color channels, and its value range is ^Jc∈ (0,1), u represents the object distance from the camera to the target to be observed, β represents the scattering coefficient of fog, reflecting the fog concentration, and can also be replaced by the fog visibility Vis, that is, β=2.996/Vis, T(u) represents the difference caused by the light source type, BRDF(θ) represents the difference caused by the reflective characteristics of the target surface, and under natural light, all targets are regarded as diffuse reflection targets, Δl represents the radius of the spherical shell of the point light source, which depends on the specific point light source model.

可作为有效信号，即：In fog imaging, only the ballistic light part in I ^c

Can be used as a valid signal, namely:

The rest is the environmental interference term in I ^c , so the square signal-to-noise ratio SNR ² (I ^c ) of the fog image I ^c can be:

Furthermore, considering the signal-to-noise ratio improvement effect of multi-time frame fusion and multi-view fusion at the hardware level of fog imaging system under fog, _NF is used to represent the number of time frames fused by the system, and _NV is used to represent the number of camera views of the system. The final signal-to-noise ratio square formula can be:

The overexposure limits in the constraints are:

Among them, C _full represents the full well capacity of the camera, and ISO represents the camera sensitivity.

Step S302: Analyze the minimum resolution of the system based on hardware parameters, and establish a relationship between the minimum resolution and the maximum fog concentration that can be faced.

In the actual implementation process, the embodiment of the present application can define the sensitivity threshold (AST) of the multi-frame multi-view fog imaging system:

至少要大于系统的灵敏度阈值(AST)，即：The definition of the resolution limit in fog imaging can be the minimum effective signal of the image under fog

At least greater than the system sensitivity threshold (AST), that is:

After substituting the system hardware parameters, that is:

In the absence of fog, the target object is imaged within the effective working range of the camera, that is:

Among them, η _lin represents the empirical constant of the linear working range, and η _lin ∈(0.2,0.6) is generally taken. Combining the above two formulas, the minimum resolution model of the fog imaging system can be obtained:

As shown in FIG4 , the online application process of the embodiment of the present application may include the following steps:

Step S401: Acquire the actual fog concentration or actual fog visibility of the current foggy scene. The embodiment of the present application can acquire the actual fog concentration β ₀ or fog visibility Vis ₀ of the current open-air scene and determine the observed object distance u.

Step S402: Determine the hardware parameters required for the fog imaging system using the proposed minimum resolution model. The embodiment of the present application can use the pre-established minimum resolution model to determine the number of viewing angles _NF , the number of time frames _NV and the bit depth of the camera required for the fog imaging system.

Step S403: Utilize the proposed signal-to-noise ratio model to further optimize the hardware parameters, and realize the construction of a fog imaging system for any fog scene. Furthermore, the embodiment of the present application may assume that the system needs to image a diffuse reflective target object illuminated by a point in the fog, and then formulate a hardware parameter optimization strategy based on the signal-to-noise ratio model.

In order to improve the signal-to-noise ratio of the fog image, the embodiment of the present application can first increase the full well capacity C _full of the camera, reduce the pixel area _Spix , and adjust the camera sensitivity ISO to the minimum; secondly, under the premise that the camera is not actually overexposed, the embodiment of the present application can maximize the light intensity A, exposure time t, spectral response function SRF ^c and target reflectivity J ^c , and minimize the lens aperture value fno and object distance u, wherein, since the reflectivity J ^c , lens aperture value fno and object distance u have a larger power in the signal-to-noise ratio square formula, they need to be adjusted first compared with other parameters; finally, the embodiment of the present application can continue to increase _NF and _NV to improve the signal-to-noise ratio on the basis of satisfying the number of viewing angles and time frames required by the minimum resolution model.

The analysis process for other light source types and target reflectance characteristics is the same as above. By effectively adjusting the hardware parameters, the construction of a fog imaging system for any fog scene can be achieved.

According to the foggy weather imaging method based on image signal-to-noise ratio analysis proposed in the embodiment of the present application, it can be guided by suppressing noise and improving image signal-to-noise ratio. Starting from the hardware level, a model relationship between the hardware parameters involved in the shooting of the fog imaging system and the image signal-to-noise ratio is established. At the same time, the relationship between the minimum resolution capability of the fog imaging system and the highest fog concentration that can be faced is established, and the various hardware parameters of the fog imaging system are optimized in this way, and finally a fog imaging system for any fog scene is constructed, thereby improving the fog concentration that can be imaged by the fog imaging system. In this way, the technical problem that the defogging algorithm in the related art only considers the ideal noise-free situation, and cannot reconstruct the target object when the fog concentration is high and the sensor noise cannot be ignored is solved.

Next, a foggy weather imaging device based on image signal-to-noise ratio analysis proposed in an embodiment of the present application is described with reference to the accompanying drawings.

FIG5 is a block diagram of a foggy weather imaging device based on image signal-to-noise ratio analysis according to an embodiment of the present application.

As shown in FIG. 5 , the foggy weather imaging device 10 based on image signal-to-noise ratio analysis includes: an acquisition module 100 , a determination module 200 and an analysis module 300 .

Specifically, the acquisition module 100 is used to acquire the actual fog concentration or actual fog visibility of the current foggy scene.

The determination module 200 is used to determine the current required viewing angle number, the current time frame number and the current camera bit depth of the fog imaging system based on the actual fog concentration or the actual fog visibility and the observed object distance using the pre-built minimum resolution model of the fog imaging system.

The analysis module 300 is used to optimize the current required number of viewing angles, the current number of time frames and other preset camera parameters of the fog imaging system using a pre-trained signal-to-noise ratio model, and obtain the fog imaging result of the current foggy scene based on the optimized current required number of viewing angles, the current number of time frames, other preset camera parameters and the bit depth of the current camera.

Optionally, in one embodiment of the present application, the analysis module 300 includes: a first adjustment unit and a second adjustment unit.

The first adjustment unit is used to increase the full well capacity of the camera, reduce the pixel area, and adjust the camera sensitivity to the minimum;

The second adjustment unit is used to increase the light intensity, exposure time, spectral response function and target reflectivity, and reduce the lens aperture value and observation object distance while ensuring that the camera is not overexposed, so as to meet the current required number of viewing angles and the current number of time frames while achieving the purpose of optimizing the signal-to-noise ratio.

Optionally, in one embodiment of the present application, the minimum resolution model of the fog imaging system is:

Among them, _ηlin represents the empirical constant of the linear working range, bit_depth represents the bit depth of the camera, β represents the scattering coefficient of fog, u represents the object distance from the camera to the target object to be observed, _NF represents the number of time frames fused by the system, and _NV represents the number of camera viewing angles of the system.

Optionally, in one embodiment of the present application, the foggy weather imaging device 10 based on image signal-to-noise ratio analysis further includes: a definition module and a modeling module.

The definition module is used to define the imaging model of the camera's Raw format image.

The modeling module is used to establish the model relationship between the hardware parameters involved in the shooting of the fog imaging system and the image signal-to-noise ratio based on the imaging model, and to determine the restriction conditions between the hardware parameters.

Optionally, in one embodiment of the present application, the signal-to-noise ratio square formula in the model relationship is:

Among them, SNR ² represents the square of the signal-to-noise ratio, _Spix represents the pixel area of the sensor, t represents the camera exposure time, SRF ^c represents the spectral response function of different color channels of the camera, A represents the scene illumination intensity, k represents the constant unknown proportional coefficient, fno represents the aperture value of the lens, J ^c represents the reflectivity of the target object corresponding to different color channels, BRDF(θ) represents the difference caused by the reflective characteristics of the target surface, and T(u) represents the difference caused by the light source type.

Optionally, in one embodiment of the present application, the overexposure limit in the limiting condition is:

Among them, C _full represents the full well capacity of the camera, and ISO represents the camera sensitivity.

It should be noted that the above explanation of the embodiment of the foggy weather imaging method based on image signal-to-noise ratio analysis is also applicable to the foggy weather imaging device based on image signal-to-noise ratio analysis of this embodiment, which will not be repeated here.

According to the foggy weather imaging device based on image signal-to-noise ratio analysis proposed in the embodiment of the present application, it can be guided by suppressing noise and improving image signal-to-noise ratio. Starting from the hardware level, a model relationship between the hardware parameters involved in the shooting of the fog imaging system and the image signal-to-noise ratio is established. At the same time, the relationship between the minimum resolution capability of the fog imaging system and the highest fog concentration that can be faced is established, and the various hardware parameters of the fog imaging system are optimized in this way, and finally a fog imaging system for any fog scene is constructed, thereby improving the fog concentration that can be imaged by the fog imaging system. In this way, the technical problem that the defogging algorithm in the related art only considers the ideal noise-free situation, and cannot reconstruct the target object when the fog concentration is high and the sensor noise cannot be ignored is solved.

FIG6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

When the processor 602 executes the program, the foggy weather imaging method based on image signal-to-noise ratio analysis provided in the above embodiment is implemented.

Furthermore, the electronic device also includes:

The communication interface 603 is used for communication between the memory 601 and the processor 602 .

The memory 601 is used to store computer programs that can be executed on the processor 602 .

The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in FIG6, but it does not mean that there is only one bus or one type of bus.

Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the foggy weather imaging method based on image signal-to-noise ratio analysis as described above is implemented.

In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

The logic and/or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims (7)

1. a fog imaging method based on image signal-to-noise ratio analysis, is characterized in that, comprises the following steps: Obtain the actual fog concentration or actual fog visibility of the current fog scene; Based on the actual fog concentration or the actual fog visibility and observed object distance, use the pre-built minimum resolution model of the fog imaging system to determine the current required viewing angle, current time frame number and current camera position of the fog imaging system deep, wherein the minimum resolution model of the fog imaging system is:

Among them, η _lin represents the empirical constant of the linear working range, bit_depth represents the bit depth of the camera, β represents the fog concentration, u represents the distance of the observed object, _NF represents the number of time frames for system fusion, and N _V represents the camera angle of view possessed by the system number; and Use the pre-trained signal-to-noise ratio model to optimize the current required viewing angle, current time frame number and other preset camera parameters of the fog imaging system, wherein the full well capacity of the camera is improved, the pixel area is reduced, and the camera is sensitive Adjust the degree to the minimum, increase the light intensity, exposure time, spectral response function and object reflectivity, reduce the lens aperture value and the observed object distance under the condition that the camera is not overexposed to meet the current requirements. The number of viewing angles and the number of current time frames are required to achieve the optimization of the signal-to-noise ratio; And according to the optimized current required viewing angle, the current time frame number, other preset camera parameters and the bit depth of the current camera, the fog imaging result of the current foggy scene is obtained, wherein the preset Other camera parameters of the camera are the full well capacity of the camera, pixel area, camera sensitivity, whether the camera is actually overexposed, light intensity, exposure time, spectral response function, target reflectivity, lens aperture value, and observed object distance. 2. The method according to claim 1, wherein the minimum resolution model of the pre-built fog imaging system is used to determine the current required viewing angle of the fog imaging system, the current time frame number and the current camera. Before bit depth, also include: Define the imaging model under the camera Raw format image; Based on the imaging model, while establishing a model relationship between the hardware parameters involved in the photography of the fog imaging system and the signal-to-noise ratio of the image, constraints between the hardware parameters are determined. 3. method according to claim 2, is characterized in that, the signal-to-noise ratio square formula in the described model relation is:

Among them, SNR ² represents the square of the signal-to-noise ratio, _Spix represents the pixel area of the sensor, t represents the exposure time of the camera, SRF ^c represents the spectral response function of different color channels of the camera, A represents the scene light intensity, k represents the constant unknown scale coefficient, and fno represents The aperture value of the lens, J ^c represents the reflectance of the target object corresponding to different color channels, BRDF(θ) represents the difference caused by the surface reflection characteristics of the target object, and T(u) represents the difference caused by the type of light source. 4. The method according to claim 3, characterized in that the overexposure limit in the limiting conditions is:

Among them, C _full represents the full well capacity of the camera, and ISO represents the camera sensitivity. 5. A fog imaging device based on image signal-to-noise ratio analysis, characterized in that it comprises: The obtaining module is used to obtain the actual fog concentration or actual fog visibility of the current foggy scene; A determination module, configured to determine the current required viewing angle, current time frame number, and current camera angle of the fog imaging system by using a pre-built minimum resolution model of the fog imaging system based on the actual fog concentration or the actual fog visibility. bit depth, wherein the minimum resolution model of the fog imaging system is:

Among them, η _lin represents the empirical constant of the linear working range, bit_depth represents the bit depth of the camera, β represents the fog concentration, u represents the distance of the observed object, _NF represents the number of time frames for system fusion, and N _V represents the camera angle of view possessed by the system number; and The analysis module is used to optimize the current required viewing angle, current time frame number and other preset camera parameters of the fog imaging system by using the pre-trained signal-to-noise ratio model, wherein the full well capacity of the camera is increased and the pixel area is reduced , and adjust the camera sensitivity to the minimum, and increase the light intensity, exposure time, spectral response function and object reflectivity, and reduce the lens aperture value and the observed object distance under the condition that the camera is not over-exposed. While satisfying the currently required number of viewing angles and the current number of time frames, the purpose of optimizing the signal-to-noise ratio is achieved, and according to the optimized current required number of viewing angles, the current number of time frames, other preset camera parameters and the current The bit depth of the camera is used to obtain the foggy imaging result of the current foggy scene, wherein the other preset camera parameters are the full well capacity of the camera, the pixel area, the sensitivity of the camera, whether the camera is actually overexposed, the light intensity , exposure time, spectral response function, target reflectance, lens aperture value and observed object distance. 6. An electronic device, characterized in that it comprises: a memory and a processor, the memory stores a computer program that can run on the processor, and the processor executes the program, so as to realize the requirements of claim 1 The fog imaging method based on image signal-to-noise ratio analysis described in any one of -4. 7. A computer-readable storage medium on which a computer program is stored, characterized in that the program is executed by a processor for realizing the analysis based on the image signal-to-noise ratio as described in any one of claims 1-4 Fog imaging method.

Images