CN112950502B - Image processing method and device, electronic equipment and storage medium - Google Patents

Abstract

The application discloses an image processing method and device, electronic equipment and a storage medium, wherein the image processing method comprises the following steps: acquiring a plurality of visible light images and a plurality of invisible light images; determining image registration parameters based on the plurality of invisible light images; and registering the plurality of visible light images based on the image registration parameters.

Description

Image processing methods and devices, electronic equipment, storage media

Technical field

The present application relates to the field of computer technology, and in particular, to an image processing method and device, electronic equipment, and storage media.

Background technique

Currently, there are two main methods for video noise reduction: one is the spatial domain noise reduction method, and the other is the temporal domain noise reduction method. The spatial domain noise reduction method only needs to process the current video frame, while the temporal domain noise reduction method needs to refer to video frames other than the current video frame (referred to as reference frames) to process the current video frame. For the time domain noise reduction method, the reference frame needs to be registered with the current video frame before noise reduction processing. The current registration method has the problem of high registration error rate.

Contents of the invention

In order to solve the above technical problems, embodiments of the present application provide an image processing method and device, electronic equipment, and storage media.

An embodiment of the present application provides an image processing method, which includes:

Acquire multiple visible light images and multiple invisible light images;

Based on the multiple invisible light images, determine image registration parameters;

Based on the image registration parameters, the multiple visible light images are registered.

In some optional implementations of this application, acquiring multiple visible light images and multiple invisible light images includes:

A plurality of visible light images collected by the first image sensor and a plurality of invisible light images collected by the second image sensor are acquired, wherein the plurality of visible light images and the plurality of invisible light images have a corresponding relationship according to the collection time.

In some optional implementations of the present application, the multiple visible light images include a first image and a second image, and the multiple invisible light images include a third image and a fourth image, wherein the first image and the The third image corresponds to the second image and the fourth image corresponds;

Determining image registration parameters based on the plurality of invisible light images includes:

Extract a first set of feature points from the third image, and extract a second set of feature points from the fourth image;

Determine at least one pair of feature points with a matching relationship from the first feature point set and the second feature point set;

Based on the coordinate information of the at least one pair of feature points, image registration parameters are determined.

In some optional implementations of this application, the method further includes:

The image registration parameters are adjusted based on calibration data; wherein the calibration data is determined based on the relative positional relationship between the first image sensor and the second image sensor.

In some optional implementations of this application, the registration of the multiple visible light images based on the image registration parameters includes:

The second image is transformed based on the image registration parameter to obtain a fifth image registered with the first image.

In some optional implementations of this application, the method further includes:

Image fusion processing is performed on the fifth image and the first image to obtain a sixth image, where the sixth image is an image after removing noise from the first image.

In some optional implementations of this application, performing image fusion processing on the fifth image and the first image includes:

The pixel value of each pixel point in the fifth image is averaged correspondingly with the pixel value of each pixel point in the first image to obtain the pixel value of each pixel point in the sixth image.

In some optional implementations of this application, performing image fusion processing on the fifth image and the first image includes:

Generating a motion mask image for determining motion areas and non-motion areas;

Determine the pixel value of each pixel point in the motion area in the first image as the pixel value of each pixel point in the motion area in the sixth image;

The pixel values of each pixel in the non-motion area in the fifth image and the pixel values of each pixel in the non-motion area in the first image are averaged to obtain the pixel value in the sixth image. The pixel value of each pixel in the non-motion area.

In some optional implementations of this application, generating a motion mask image includes:

Calculate the difference between the pixel value of each pixel in the fifth image and the pixel value of each pixel in the first image to obtain the pixel value of each pixel in the motion mask image;

The area formed by the pixel points in the motion mask image with a pixel value greater than or equal to the threshold is a motion area, and the area formed by the pixel points in the motion mask image with a pixel value less than the threshold is a non-motion area.

In some optional implementations of this application, the invisible light image is an infrared image.

An embodiment of the present application provides an image processing device, which includes:

An acquisition unit is used to acquire multiple visible light images and multiple invisible light images;

A parameter determination unit configured to determine image registration parameters based on the plurality of invisible light images;

An image registration unit is used to register the plurality of visible light images based on the image registration parameters.

In some optional implementations of this application, the acquisition unit is used to acquire multiple visible light images collected by the first image sensor and multiple invisible light images collected by the second image sensor, wherein the multiple visible light images There is a corresponding relationship with the plurality of invisible light images according to the collection time.

In some optional implementations of the present application, the multiple visible light images include a first image and a second image, and the multiple invisible light images include a third image and a fourth image, wherein the first image and the The third image corresponds to the second image and the fourth image corresponds;

The parameter determination unit is configured to extract a first feature point set from the third image and a second feature point set from the fourth image; from the first feature point set and the second feature point set The feature points are concentrated to determine at least one pair of feature points with a matching relationship; based on the coordinate information of the at least one pair of feature points, image registration parameters are determined.

In some optional implementations of the present application, the parameter determination unit is also used to adjust the image registration parameters based on calibration data; wherein the calibration data is based on the first image sensor and the third image sensor. The relative positional relationship between the two image sensors is determined.

In some optional implementations of this application, the image registration unit is configured to transform the second image based on the image registration parameters to obtain a fifth image that is registered with the first image.

In some optional implementations of this application, the device further includes:

An image fusion unit is configured to perform image fusion processing on the fifth image and the first image to obtain a sixth image, where the sixth image is an image after removing noise from the first image.

In some optional implementations of the present application, the image fusion unit is configured to average the pixel values of each pixel in the fifth image and the pixel values of each pixel in the first image, Get the pixel value of each pixel in the sixth image.

In some optional implementations of this application, the image fusion unit is used to generate a motion mask image, and the motion mask image is used to determine the motion area and the non-motion area; the motion area in the first image is The pixel value of each pixel point in the motion area in the sixth image is determined as the pixel value of each pixel point in the motion area in the fifth image; the pixel value of each pixel point in the non-motion area in the fifth image is compared with the pixel value of each pixel point in the motion area in the fifth image. The pixel values of each pixel point in the non-motion area in an image are averaged to obtain the pixel value of each pixel point in the non-motion area in the sixth image.

In some optional implementations of the present application, the image fusion unit is configured to calculate the difference between the pixel value of each pixel in the fifth image and the pixel value of each pixel in the first image. , obtain the pixel value of each pixel point in the motion mask image; wherein, the area formed by the pixel points in the motion mask image with a pixel value greater than or equal to the threshold is a motion area, and the pixel value in the motion mask image The area formed by pixels smaller than the threshold is a non-motion area.

In some optional implementations of this application, the invisible light image is an infrared image.

An embodiment of the present application also provides an electronic device, including a memory and a processor. A computer program is stored in the memory. When the computer program is executed by the processor, it causes the processor to execute the steps described in the above embodiments. image processing methods.

An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. The feature is that when the computer program is executed by a processor, the image processing method described in the above embodiment is implemented.

The technical solution of the embodiment of the present application determines the image registration parameters based on multiple invisible light images. Compared with determining the image registration parameters directly based on multiple visible light images, since the dark part information of the invisible light images is richer, the collected real The characteristics of the object are more accurate. In this way, the image registration parameters will be calculated more accurately. Multiple visible light images are registered through the image registration parameters, which improves the registration accuracy of multiple visible light images and provides a basis for subsequent alignment. Noise reduction processing of visible light images is provided.

Description of the drawings

Figure 1 is a flow chart of time domain noise reduction provided by an embodiment of the present application;

Figure 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

Figure 3 is a schematic diagram of the correspondence between the visible light image sequence and the invisible light image sequence provided by the embodiment of the present application;

Figure 4 is a flow chart of image fusion based on motion mask images provided by an embodiment of the present application;

Figure 5 is a schematic diagram 1 of the principle provided by the embodiment of the present application;

Figure 6 is a schematic flow chart of calculating the registration transformation matrix provided by the embodiment of the present application;

Figure 7 is a second schematic diagram of the principle provided by the embodiment of the present application;

Figure 8 is a schematic diagram of determining registration transformation parameters by region according to an embodiment of the present application;

Figure 9 is a schematic structural diagram of an image processing device provided by an embodiment of the present application;

Figure 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions, and numerical values set forth in these examples do not limit the scope of the present application unless otherwise specifically stated.

At the same time, it should be understood that, for convenience of description, the dimensions of various parts shown in the drawings are not drawn according to actual proportional relationships.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the application or its application or uses.

Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods and devices should be considered a part of the specification.

It should be noted that similar reference numerals and letters refer to similar items in the following figures, so that once an item is defined in one figure, it does not need further discussion in subsequent figures.

In order to facilitate the understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. It should be noted that the following description of the related technologies is for understanding the technical solutions of the embodiments of the present application and does not cause the implementation of the present application. Example of limitations of technical solutions.

Currently, there are two main methods for video noise reduction: one is the spatial domain noise reduction method, and the other is the temporal domain noise reduction method. The spatial domain noise reduction method is a noise reduction method that only targets a single frame of image. Specifically, neighborhood analysis and processing is performed in a single frame of image to achieve noise filtering. As an example, the spatial domain noise reduction method can be achieved through a bilateral filtering algorithm, Or non-local means (Non-Local Means) algorithm to achieve; and the temporal noise reduction method is a noise reduction method that refers to the information of video frames other than the current video frame (referred to as the reference frame) for analysis and processing, for example, the current video frame The previous frame of the frame is used as the reference frame. Since the temporal noise flashes in the time domain and the real information remains unchanged, the addition of the reference frame can better distinguish the real image information and noise, so that Remove noise more accurately.

Figure 1 is a typical flow chart of video temporal noise reduction. As shown in Figure 1, the reference frame is image registered so that the registered reference frame is aligned with the current frame; the registered reference frame is aligned with the current frame. The frames are subjected to image fusion processing, that is, denoising processing is implemented, and the denoising output frame is obtained. It should be pointed out that the denoised output frame can be understood as the frame image after denoising the current frame.

In the flow chart shown in Figure 1, the image registration step is a key step. Since in continuous video frames, the shooting position of each frame may move, and there may be moving objects in each frame, the image information may be offset in two adjacent frames. If an object is in two adjacent frames, If the relative positions in the frames are inconsistent, subsequent image fusion processing will not be able to distinguish the same point on two adjacent frames. The image registration algorithm is the key to solving this problem. The image registration algorithm can analyze the typical characteristics of two adjacent frames, and position the image information of the two adjacent frames based on feature matching.

In some solutions, image registration algorithms use feature point matching. The so-called feature points are points with obvious features in the image, such as corner points. The disadvantage of this solution is that in darker scenes, such as dark streets at night, the picture information in the video will be very dark, making it difficult to extract the real feature points in the picture. Moreover, the noise in the night scene is also very large, which will interfere with the scene. The judgment of the real information of the image will lead to errors in image registration, which will cause serious ghosting and other problems in the video after time domain noise reduction.

To this end, the following technical solution of the embodiment of the present application is proposed. In the technical solution of the embodiment of the present application, the first image sensor and the second image sensor are used to simultaneously collect a visible light image sequence and an invisible light image sequence. Through the invisible light image sequence Determining the image registration parameters suitable for the visible light image sequence can improve the image registration accuracy of the visible light image sequence, which can in turn help improve the signal-to-noise ratio of the visible light image sequence.

The image processing method provided by the embodiment of the present application is applied to an image processing device, and the image processing device can be installed on an electronic device. In some embodiments, the electronic device is, for example: a mobile phone, a tablet computer, a wearable device, an interactive advertising machine, a game console, a desktop computer, an all-in-one machine, a vehicle-mounted terminal, and the like.

The image processing method and image processing device provided by the embodiments of the present application are described below.

Figure 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application. As shown in Figure 2, the image processing method includes the following steps:

Step 201: Obtain multiple visible light images and multiple invisible light images.

In the embodiment of the present application, the electronic device has a first image sensor and a second image sensor, wherein the first image sensor is used to collect visible light images, and the second image sensor is used to collect invisible light images. In some optional implementations, the invisible light image is an infrared image.

In some optional embodiments, the first image sensor is a Complementary Metal-Oxide-Semiconductor (CMOS) sensor (Sensor), and the second image sensor is an Infrared (IR) sensor ( Sensor). Among them, the CMOS Sensor is used to collect (also called capturing or shooting) visible light images. Specifically, the CMOS Sensor collects the visible light band in the scene. Here, the visible light band is red (R), green (G), blue ( B) band, the visible light band is processed by a processor (such as an image signal processor (Image Signal Processing, ISP)) to present a colored image, which is called a visible light image. As an example, the data format of the visible light image can be the YUV format. The IR Sensor is used to collect invisible light images. Specifically, the IRSensor collects infrared light (or infrared ray) with a frequency lower than red light in the scene. Since substances above absolute zero can produce infrared rays, even in extreme In dark environments, the IR Sensor can collect infrared rays generated by objects in the environment. It can be seen that the infrared images collected by the IR Sensor can contain information about objects in darker environments.

In the embodiment of the present application, acquiring multiple visible light images and multiple invisible light images includes: acquiring multiple visible light images collected by the first image sensor, and multiple invisible light images collected by the second image sensor, wherein the plurality of visible light images are acquired by the second image sensor. A visible light image has a corresponding relationship with the plurality of invisible light images according to the collection time.

Here, the multiple visible light images and the multiple invisible light images have a corresponding relationship according to the collection time, which means that the visible light images and the invisible light images collected at the same time or within the same time period have a corresponding relationship. For example, referring to Figure 3, the first image sensor and the second image sensor collect image sequences at the same time to obtain a visible light image sequence (that is, the original image sequence) and an invisible light image sequence (such as an infrared image sequence). At time t1, the first image sensor collects the visible light image 1, and the second image sensor collects the invisible light image 1. The visible light image 1 corresponds to the invisible light image 1. At time t2, the first image sensor collects the visible light image 2, and the second image sensor collects the invisible light image 2. The visible light image 2 and the invisible light image 2 correspond to each other. At time t3, the first image sensor collects the visible light image 3, and the second image sensor collects the invisible light image 3. The visible light image 3 and the invisible light image 3 correspond to each other. By analogy, the multiple visible light images collected by the first image sensor and the multiple invisible light images collected by the second image sensor have a corresponding relationship according to the collection time.

Step 202: Determine image registration parameters based on the multiple invisible light images.

Here, the visible light image (that is, the original image) collected by the first image sensor has a relatively poor signal-to-noise ratio in a darker scene (such as a night scene), and the visibility of dark information in the visible light image is low. If the first image is directly If multiple visible light images output by the sensor are registered (i.e., relative position registration), the registration error rate may be high; whether the invisible light image collected by the second image sensor is in a brighter scene or a darker scene, the The visibility of dark information is high, and even in scenes that are difficult to perceive visually, the second image sensor can still collect the real information of the scene; because the first image sensor and the second image sensor collect images at the same time, the relative positions of multiple visible light images The relationship is similar to the relative positional relationship of multiple invisible light images.

To sum up, the image registration parameters can be determined based on multiple invisible light images, and then the image registration parameters can be used to register multiple visible light images. The following describes the technical solutions of the embodiments of the present application using "multiple pictures" as "two pictures". It should be noted that the solution of "more than two pictures" is also applicable to the technical solutions of the embodiments of the present application.

In some optional implementations, the plurality of visible light images include a first image and a second image, and the plurality of invisible light images include a third image and a fourth image, wherein the first image and the third image Three images correspond, and the second image corresponds to the fourth image. As an example, the first image sensor and the second image sensor respectively collect the first image and the third image at the same time (such as the first time), so the first image and the third image correspond; the first image sensor The second image and the fourth image are respectively collected by the second image sensor at the same time (such as the second time), so the second image and the fourth image correspond to each other.

In the above solution, the collection time of the first image is located after the collection time of the second image, and the collection time of the third image is located after the collection time of the fourth image. As an example, the first image and the third image are respectively the visible light image and the invisible light image currently collected by the first image sensor, and the second image and the fourth image are respectively the visible light image and the invisible light image collected previously by the first image sensor. As an example, the second image is the previous frame of the first image, the first image can be called the first current frame, and the second image is called the first reference frame; the fourth image is the previous frame of the third image. image, the third image can be called the second current frame, and the fourth image can be called the second reference frame. The second current frame is, for example, an infrared current frame, and the second reference frame is, for example, an infrared reference frame.

In the embodiment of the present application, the image registration parameter refers to the image registration parameter of one image relative to another image. For example, the image registration parameter refers to the image registration parameter of image A relative to image B. Through image A and image registration, By multiplying the quasi-parameters, the registered image A can be obtained, and the registered image A and image B are aligned. Since the multiple visible light images collected by the first image sensor and the multiple invisible light images collected by the second image sensor have a corresponding relationship according to the collection time, the image registration parameters corresponding to the multiple invisible light images are related to the multiple visible light images. The corresponding image registration parameters are consistent. Therefore, the image registration parameters can be determined based on multiple invisible light images, and then the image registration parameters are applied to multiple visible light images, thereby achieving registration of multiple visible light images. The following describes how to determine image registration parameters based on multiple invisible light images.

In this embodiment of the present application, a first feature point set is extracted from the third image, and a second feature point set is extracted from the fourth image; from the first feature point set and the second feature point Focus on determining at least one pair of feature points with a matching relationship; and determine image registration parameters based on the coordinate information of the at least one pair of feature points.

For example: the third image is the current infrared frame, the fourth image is the infrared reference frame, and the infrared reference frame can be the frame image preceding the current infrared frame. 1) Extract feature points from the current infrared frame and the infrared reference frame to obtain the first set of feature points in the current infrared frame and the second set of feature points in the infrared reference frame. Here, the number of feature points included in the first feature point set and the second feature point set may be the same or different, and the technical solutions of the embodiments of the present application do not limit this. In specific implementation, feature point extraction algorithms such as the Harris algorithm or the Scale-invariant feature transform (SIFT) algorithm can be used to implement feature point extraction. As an example, the principle of the feature point extraction algorithm is: based on the neighborhood information of each pixel in the image, determine whether the position where the pixel sits is a point with significant features, such as whether it is a point on the edge of the object, and then determine whether the pixel is located at the corner of the object. Whether the point is a feature point. 2) Match the feature points extracted from the current infrared frame and the infrared reference frame, and calculate the image registration parameters based on the matched feature points. Here, the image registration parameters can be embodied as a matrix, called the registration transformation matrix. The registration transformation matrix is a 3×3 matrix, that is, the registration transformation matrix includes 9 parameters. As an example, the registration transformation matrix is an affine transformation matrix, and the last row of the registration transformation matrix can be (0, 0, 1). An example of the registration transformation matrix is given below:

Among them, a, b, c, d, e, f are the parameters that need to be determined in the registration transformation matrix.

Assume that feature point A in the first feature point set matches feature point a in the second feature point set, the coordinates of feature point A are (x1, y1, 1), and the coordinates of feature point a are (x2, y2, 1), Then, the coordinates of these two feature points satisfy the following formula:

By fitting multiple pairs of matching feature points to the above formula, each parameter in the registration transformation matrix can be determined, thereby determining the registration transformation matrix. In specific implementation, algorithms such as the Random Sample Consensus (RANSAC) algorithm can be used to match feature points and calculate the registration transformation matrix. The principle of the RANSAC algorithm is to randomly extract feature points from two frames of images, match the coordinate vectors corresponding to the feature points, and fit the registration transformation matrix based on the coordinate vectors of the matched feature points.

In some optional implementations, the first image sensor and the second image sensor share the same lens assembly (referred to as lens assembly for short). In this case, it can be understood that the first image sensor and the second image sensor are on the electronic device. have the same position, there is no positional deviation between the visible light image collected by the first image sensor and the invisible light image collected by the second image sensor, and the visible light image can be registered directly according to the image registration parameters determined by the invisible light image.

In some optional implementations, the first image sensor and the second image sensor respectively use different lens components. In this case, it can be understood that the positions of the first image sensor and the second image sensor on the electronic device are slightly different. Deviation, there is a positional deviation between the visible light image collected by the first image sensor and the invisible light image collected by the second image sensor. In order to compensate for this deviation, the image registration parameters determined by the invisible light image need to be adjusted. Specifically, the image registration parameters are adjusted based on calibration data; wherein the calibration data is determined based on the relative positional relationship between the first image sensor and the second image sensor. Here, the calibration data may be calibration data when the electronic device leaves the factory. The calibration data is related to the relative positional relationship between the first image sensor and the second image sensor. The image registration parameters are adjusted through the calibration data to achieve invisible light images. Perform position alignment to the visible light image to obtain image registration parameters that can be used for the visible light image.

Step 203: Register the multiple visible light images based on the image registration parameters.

In some optional implementations, the image registration parameters are determined based on a third image and a fourth image among the plurality of invisible light images, wherein the third image corresponds to the first image among the plurality of visible light images, and the fourth image corresponds to Second image correspondence among multiple visible light images. The second image is transformed based on the image registration parameter to obtain a fifth image registered with the first image. Here, the fifth image refers to an image obtained by transforming the second image, and the image is aligned with the first image.

For example: the first image is the current frame output by the CMOS Sensor, and the second image is the reference frame output by the CMOS Sensor. The reference frame can be the previous frame of the current frame. The reference frame is transformed according to the registration transformation matrix to obtain a reference frame aligned with the current frame. Specifically, the coordinates of each pixel point in the reference frame are multiplied by the registration transformation matrix to obtain new coordinates. After each pixel point is rearranged according to the new coordinates, the transformation is completed. At this time, the current frame and the reference frame complete the alignment operation.

After the registration of the first image and the second image is completed through the above technical solutions of the embodiments of the present application, the first image and the registered second image (ie, the fifth image) can be fused, thereby completing temporal denoising. Specifically, image fusion processing is performed on the fifth image and the first image to obtain a sixth image, where the sixth image is an image after removing noise from the first image.

In some optional implementations, the image fusion processing of the fifth image and the first image can be accomplished in the following manner:

Method 1: Average the pixel values of each pixel point in the fifth image and the pixel values of each pixel point in the first image to obtain the pixel value of each pixel point in the sixth image.

For example: Assume that the pixels of the image are N×M, and N and M are positive integers. Then, the pixel value of the pixel point with coordinates (xi, yj, 1) in the fifth image and the coordinates in the first image are The pixel values of the pixel point (xi, yj, 1) are averaged to obtain the pixel value of the pixel point with coordinates (xi, yj, 1) in the sixth image. Among them, i is a positive integer greater than or equal to 1 and less than or equal to N, and j is a positive integer greater than or equal to 1 and less than or equal to M.

Method 2: Generate a motion mask (Mask) image, the motion mask image is used to determine the motion area and the non-motion area; determine the pixel value of each pixel point in the motion area in the first image as the first The pixel value of each pixel point in the motion area in the sixth image; the pixel value of each pixel point in the non-motion area in the fifth image and the pixel value of each pixel point in the non-motion area in the first image The corresponding pixel values of are averaged to obtain the pixel values of each pixel point in the non-motion area in the sixth image.

Here, in order to remove noise more accurately, motion detection can be performed on the image. Specifically, by comparing the fifth image and the first image, the moving area and the non-moving area in the image are determined. In specific implementation, a mask image is generated to reflect which areas in the image are moving areas and which areas are non-moving areas.

In some optional implementations, the mask image can be generated in the following manner: calculating the difference between the pixel value of each pixel point in the fifth image and the pixel value of each pixel point in the first image, Obtain the pixel value of each pixel point in the motion mask image; wherein, the area formed by the pixel points in the motion mask image with a pixel value greater than or equal to the threshold is a motion area, and the pixel value in the motion mask image is less than The area formed by the threshold pixels is a non-motion area.

For example: Suppose the pixels of the image are N , yj, 1) is located in the motion area, then the pixel value of the pixel point with coordinates (xi, yj, 1) in the sixth image is equal to the pixel value of the pixel point with coordinates (xi, yj, 1) in the first image value; if the coordinate (xi, yj, 1) is located in the non-motion area, then the pixel value of the pixel point with the coordinate (xi, yj, 1) in the sixth image is equal to the coordinate (xi, yj, 1) in the fifth image. The average value of the pixel value of the pixel point 1) and the pixel value of the pixel point whose coordinates are (xi, yj, 1) in the first image.

In some optional implementations, the mask image can be generated in the following manner: calculating the difference between the pixel value of each pixel point in the fifth image and the pixel value of each pixel point in the first image, Compare the difference with the threshold. If the difference is greater than or equal to the threshold, the pixel value of the corresponding pixel in the motion mask image is set to 1. If the difference is less than the threshold, the corresponding pixel in the motion mask image is set to 1. The pixel value of is set to 0; wherein, the area formed by the pixel points with the pixel value of 1 in the motion mask image is the motion area, and the area formed by the pixel points with the pixel value of 0 in the motion mask image is Non-sports area.

After the moving area and non-moving area in the image are determined through the above solution, the image fusion method of the fifth image and the first image is: the moving area only uses the pixel value of each pixel point of the first image, and the non-moving area uses the third The pixel values of each pixel point in one image are averaged corresponding to the pixel values of each pixel point in the fifth image.

For example: Referring to Figure 4, the fifth image is obtained after the second image is registered and transformed. The fifth image is aligned with the first image. The motion mask image can be obtained based on the difference between the fifth image and the first image. Perform image fusion processing on the first image and the fifth image with reference to the motion mask image, where the image fusion method of the fifth image and the first image is: the motion area only uses the pixel values of each pixel point of the first image, and does not The moving area uses the pixel value of each pixel point in the first image and the pixel value of each pixel point in the fifth image to calculate the average.

The technical solution of the embodiment of the present application will be described below with reference to FIGS. 5 to 7 . It should be noted that in the embodiments related to Figures 5 to 7, the first image sensor is a CMOS Sensor, and the first image and the second image collected by the first image sensor are respectively called the current frame and the reference frame. The second image sensor is an IR Sensor, and the third image and the fourth image collected by the second image sensor are respectively called the infrared current frame and the infrared reference frame. This is explained as an example.

Referring to Figure 5, the CMOS Sensor outputs the original image sequence (i.e., the original video), and the IR Sensor outputs the infrared image sequence. Match the feature points between the current infrared frame output by the IR Sensor and the infrared reference frame, and calculate the registration transformation matrix based on the coordinate information of the matched feature points. The reference frame output by the CMOS Sensor is transformed through the registration transformation matrix to obtain the registered reference frame. The registered reference frame is aligned with the current frame output by the CMOS Sensor. The current frame output by the CMOSSensor and the registered reference frame are image fused to obtain a denoised output frame, and the denoising process of the current frame is completed. The current frame is constantly updated and changed over time, so that continuous video denoising frames can be output.

For the calculation of the registration transformation matrix in Figure 5, you can refer to Figure 6. Feature points are extracted from the current infrared frame and the infrared reference frame respectively, and then the feature points in the current infrared frame and the feature points in the infrared reference frame are matched. And calculate the registration transformation matrix based on the coordinate information of the matched feature points. Combining the process shown in Figure 6 with the process shown in Figure 5 can obtain the process shown in Figure 7. As shown in Figure 7, the process shown in Figure 7 is roughly divided into two parts: one part is infrared image processing, That is, the registration transformation matrix is determined through the infrared image sequence; the other part is the original image processing, that is, the original image sequence is registered through the registration transformation matrix and image fusion processing is performed to complete the video noise reduction.

According to the above technical solution of the embodiment of the present application, when determining the registration transformation parameters, the registration transformation parameters can be determined without distinguishing the image area, that is, the entire image corresponds to one registration transformation parameter. In this case, each pixel point of the entire image Corresponds to the same registration transformation parameters. Not limited to this, the registration transformation parameters can also be determined by distinguishing regions. For example, referring to Figure 8, the image is divided into two regions. Region 1 corresponds to registration transformation parameter 1, and region 2 corresponds to registration transformation parameter 2. Here, each The method for determining the corresponding registration transformation parameters for each area can refer to the previous solution. Then, each pixel point in area 1 corresponds to registration transformation parameter 1, and each pixel point in area 2 corresponds to registration transformation parameter 2. After determining the registration transformation parameters of different areas, correspondingly, the second image collected by the first sensor also needs to be registered and transformed by area. Specifically, the coordinates of each pixel point in the first area of the second image are Transformation is performed according to the registration transformation parameter 1, and the coordinates of each pixel point in the second area of the second image are transformed according to the registration transformation parameter 2.

As a variation of the above technical solution of the embodiment of the present application, when the above technical solution of the embodiment of the present application is used to determine the registration transformation parameters, only part of the image area can be used, while another part of the image can be determined using a visible light image. Sure. Still referring to Figure 8, the image is divided into two areas. Area 1 corresponds to the registration transformation parameter 1, which is determined based on the matching feature points of area 1 in multiple invisible light images; area 2 corresponds to the registration transformation. Parameter 2. This registration transformation parameter 2 is determined based on the matching feature points of area 2 in multiple visible light images. In one application scenario, area 1 may be a dark area in the image, and area 2 may be a bright area in the image. Here, the division of dark areas and bright areas is based on the visible light image. In specific implementation, the brightness value of each pixel point of the current frame in the visible light image can be analyzed to divide the dark area and the bright area. For the dark area , the invisible light image is used to assist in determining the corresponding registration transformation parameters, and for the bright area, the corresponding registration transformation parameters are directly determined through the visible light image. It should be noted that the method of determining the registration transformation parameters of a certain area can refer to the description of the above-mentioned related solutions. Specifically, the feature points of the two images are extracted from the area, the feature points are matched and based on the matched features The coordinate information of the point calculates the registration transformation parameters corresponding to the area.

As a variation of the above technical solution of the embodiment of the present application, the image fusion processing stage can use more frames for image fusion processing instead of just two frames. In this case, multiple frames of visible light images need to be registered, for example To register L frames of visible light images, L is an integer greater than 2, then it is necessary to analyze the L frames of invisible light images corresponding to the L frames of visible light images, and determine L-1 registration transformation parameters, taking L=3 as For example, the registration transformation parameters between invisible light image 1 and invisible light image 2 can be determined, and the registration transformation parameters between invisible light image 1 and invisible light image 3 can be determined. In this way, the registration transformation parameters between invisible light image 1 and invisible light image 3 can be determined. The registration transformation parameters between the visible light image 2 realize the registration of the visible light image 1 and the visible light image 2, and the registration transformation parameters between the invisible light image 1 and the invisible light image 3 realize the registration of the visible light image 1 and the visible light image 3 Registration, in this way, the registration between visible light image 1, visible light image 2 and visible light image 3 is completed. By performing image fusion processing on the registered visible light image 1, visible light image 2 and visible light image 3, more information can be obtained. Good denoising effect. As an example, invisible light image 1 can be the current infrared frame, invisible light image 2 can be the infrared frame preceding the current infrared frame, invisible light image 3 can be the two infrared frames preceding the current infrared frame, and visible light image 1 can be In the current frame, the visible light image 2 can be the frame before the current frame, and the visible light image 3 can be the two frames before the current frame.

The technical solution of the embodiment of the present application obtains the registration transformation parameters by analyzing the invisible light image output by the second image sensor. Compared with calculating the registration transformation parameters directly from the original image output by the first image sensor, the invisible light image Dark information is richer, and the collected real object features are more accurate. In this way, the calculated registration transformation parameters will be more accurate. After registering the visible light image through the registration transformation parameters, multiple frames can be processed in the future. Avoid ghosting or smearing when noisy. For darker scenes, the video shooting itself has large noise, and the information in the dark parts is unclear. It is necessary to remove the noise and restore the information through multi-frame denoising between adjacent frames. However, the problem of noise and low visibility in the dark parts will hinder multi-frame denoising. The use of invisible light images just makes up for this shortcoming, allowing accurate registration transformation parameters to be calculated even in very dark environments, promoting better temporal noise reduction.

Figure 9 is a schematic structural diagram of an image processing device provided by an embodiment of the present application. As shown in Figure 9, the image processing device includes:

The acquisition unit 901 is used to acquire multiple visible light images and multiple invisible light images;

Parameter determination unit 902, configured to determine image registration parameters based on the multiple invisible light images;

The image registration unit 903 is configured to register the multiple visible light images based on the image registration parameters.

In some optional implementations of this application, the acquisition unit 901 is used to acquire multiple visible light images collected by the first image sensor and multiple invisible light images collected by the second image sensor, wherein the multiple visible light images The image has a corresponding relationship with the plurality of invisible light images according to the collection time.

In some optional implementations of the present application, the multiple visible light images include a first image and a second image, and the multiple invisible light images include a third image and a fourth image, wherein the first image and the The third image corresponds to the second image and the fourth image corresponds;

The parameter determination unit 902 is configured to extract a first feature point set from the third image and a second feature point set from the fourth image; from the first feature point set and the third feature point set Two feature points are concentrated to determine at least one pair of feature points with a matching relationship; based on the coordinate information of the at least one pair of feature points, image registration parameters are determined.

In some optional implementations of this application, the parameter determination unit 902 is also used to adjust the image registration parameters based on calibration data; wherein the calibration data is based on the first image sensor and the The relative position relationship of the second image sensor is determined.

In some optional implementations of this application, the image registration unit 903 is configured to transform the second image based on the image registration parameters to obtain a fifth image that is registered with the first image.

In some optional implementations of this application, the device further includes:

The image fusion unit 904 is configured to perform image fusion processing on the fifth image and the first image to obtain a sixth image, where the sixth image is an image after removing noise from the first image.

In some optional implementations of the present application, the image fusion unit 904 is configured to average the pixel values of each pixel in the fifth image and the pixel values of each pixel in the first image. , obtain the pixel value of each pixel in the sixth image.

In some optional implementations of this application, the image fusion unit 904 is used to generate a motion mask image, the motion mask image is used to determine motion areas and non-motion areas; the motion in the first image is The pixel value of each pixel point in the area is determined as the pixel value of each pixel point in the motion area in the sixth image; the pixel value of each pixel point in the non-motion area in the fifth image is compared with the The pixel values of each pixel point in the non-motion area in the first image are averaged accordingly to obtain the pixel value of each pixel point in the non-motion area in the sixth image.

In some optional implementations of the present application, the image fusion unit 904 is used to calculate the difference between the pixel value of each pixel in the fifth image and the pixel value of each pixel in the first image. value to obtain the pixel value of each pixel in the motion mask image; wherein, the area formed by the pixels in the motion mask image with a pixel value greater than or equal to the threshold is a motion region, and the pixels in the motion mask image The area formed by pixels with a value smaller than the threshold is a non-motion area.

In some optional implementations of this application, the invisible light image is an infrared image.

Those skilled in the art should understand that the functions implemented by each unit in the image processing device shown in FIG. 9 can be understood with reference to the relevant description of the foregoing image processing method. The functions of each unit in the image processing device shown in FIG. 9 can be implemented by programs running on the processor, or by specific logic circuits.

If the above-mentioned neural network training device in the embodiment of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a storage medium (such as a computer-readable storage medium). Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence or those that contribute to the existing technology. The computer software products are stored in a storage medium and include a number of instructions to An electronic device (which may be a personal computer, a server, a network device, etc.) is caused to execute all or part of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read OnlyMemory), magnetic disk or optical disk and other various media that can store program codes. As such, embodiments of the present application are not limited to any specific combination of hardware and software.

Correspondingly, embodiments of the present application also provide a computer program product in which computer-executable instructions are stored. When the computer-executable instructions are executed, the above-mentioned methods of the embodiments of the present application can be implemented.

Figure 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in Figure 10, the electronic device may include one or more (only one is shown in the figure) processors 1002 (the processor 1002 may include but is not limited to a microprocessor). A processing device such as a processor (MCU, Micro Controller Unit) or a programmable logic device (FPGA, Field Programmable GateArray)), a memory 1004 for storing data, and a transmission device 1006 for communication functions. Persons of ordinary skill in the art can understand that the structure shown in FIG. 10 is only illustrative, and it does not limit the structure of the above-mentioned electronic device. For example, the electronic device may also include more or fewer components than shown in FIG. 10 , or have a different configuration than shown in FIG. 10 .

The memory 1004 can be used to store software programs and modules of application software, such as program instructions/modules corresponding to the methods in the embodiments of the present application. The processor 1002 executes various functional applications by running the software programs and modules stored in the memory 1004 and data processing, that is, implementing the above methods. Memory 1004 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 1004 may further include memory located remotely relative to processor 1002, and these remote memories may be connected to the electronic device through a network. Examples of the above-mentioned networks include but are not limited to the Internet, intranets, local area networks, mobile communication networks and combinations thereof.

The transmission device 1006 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the electronic device. In one example, the transmission device 1006 includes a network adapter (NIC, Network Interface Controller), which can be connected to other network devices through a base station to communicate with the Internet. In one example, the transmission device 1006 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

The technical solutions described in the embodiments of this application can be combined arbitrarily as long as there is no conflict.

In the several embodiments provided in this application, it should be understood that the disclosed methods and smart devices can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components may be combined, or can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be electrical, mechanical, or other forms. of.

The units described above as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units; Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be all integrated into a second processing unit, or each unit can be separately used as a unit, or two or more units can be integrated into one unit; The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application. should be covered by the protection scope of this application.

Claims (18)

1. An image processing method, the method comprising:

acquiring a plurality of visible light images and a plurality of invisible light images;

Determining image registration parameters based on the plurality of invisible light images;

registering the plurality of visible light images based on the image registration parameters; wherein, obtain a plurality of visible light images and a plurality of invisible light images, include:

acquiring a plurality of visible light images acquired by a first image sensor and a plurality of invisible light images acquired by a second image sensor, wherein the plurality of visible light images and the plurality of invisible light images have a corresponding relation according to acquisition time;

the plurality of visible light images comprise a first image and a second image, the plurality of invisible light images comprise a third image and a fourth image, wherein the first image corresponds to the third image, and the second image corresponds to the fourth image;

the determining an image registration parameter based on the plurality of invisible light images includes:

extracting a first feature point set from the third image and a second feature point set from the fourth image;

determining at least one pair of feature points with a matching relationship from the first feature point set and the second feature point set;

image registration parameters are determined based on the coordinate information of the at least one pair of feature points.

2. The method according to claim 1, wherein the method further comprises:

adjusting the image registration parameters based on calibration data; wherein the calibration data is determined based on a relative positional relationship of the first image sensor and the second image sensor.

3. The method of claim 2, wherein the registering the plurality of visible light images based on the image registration parameters comprises:

and transforming the second image based on the image registration parameters to obtain a fifth image registered with the first image.

4. A method according to claim 3, characterized in that the method further comprises:

and carrying out image fusion processing on the fifth image and the first image to obtain a sixth image, wherein the sixth image is an image of the first image after noise is removed.

5. The method of claim 4, wherein performing an image fusion process on the fifth image and the first image comprises:

and correspondingly averaging the pixel values of all the pixel points in the fifth image and the pixel values of all the pixel points in the first image to obtain the pixel values of all the pixel points in the sixth image.

6. The method of claim 4, wherein performing an image fusion process on the fifth image and the first image comprises:

generating a motion mask image for determining a motion region and a non-motion region;

determining the pixel value of each pixel point in the motion area in the first image as the pixel value of each pixel point in the motion area in the sixth image;

and correspondingly averaging the pixel values of all the pixel points in the non-motion area in the fifth image and the pixel values of all the pixel points in the non-motion area in the first image to obtain the pixel values of all the pixel points in the non-motion area in the sixth image.

7. The method of claim 6, wherein generating a motion mask image comprises:

correspondingly calculating difference values between the pixel values of all the pixel points in the fifth image and the pixel values of all the pixel points in the first image to obtain the pixel values of all the pixel points in the motion mask image;

the region formed by the pixel points with the pixel values larger than or equal to the threshold value in the motion mask image is a motion region, and the region formed by the pixel points with the pixel values smaller than the threshold value in the motion mask image is a non-motion region.

8. The method according to any one of claims 1 to 7, wherein the invisible light image is an infrared image.

9. An image processing apparatus, characterized in that the apparatus comprises:

an acquisition unit configured to acquire a plurality of visible light images and a plurality of invisible light images;

a parameter determination unit configured to determine an image registration parameter based on the plurality of invisible light images;

the image registration unit is used for registering the plurality of visible light images based on the image registration parameters;

the acquisition unit is used for acquiring a plurality of visible light images acquired by the first image sensor and a plurality of invisible light images acquired by the second image sensor, wherein the plurality of visible light images and the plurality of invisible light images have a corresponding relation according to the acquisition time; the plurality of visible light images comprise a first image and a second image, the plurality of invisible light images comprise a third image and a fourth image, wherein the first image corresponds to the third image, and the second image corresponds to the fourth image;

the parameter determining unit is used for extracting a first characteristic point set from the third image and extracting a second characteristic point set from the fourth image; determining at least one pair of feature points with a matching relationship from the first feature point set and the second feature point set; image registration parameters are determined based on the coordinate information of the at least one pair of feature points.

10. The apparatus according to claim 9, wherein the parameter determination unit is further configured to adjust the image registration parameter based on calibration data; wherein the calibration data is determined based on a relative positional relationship of the first image sensor and the second image sensor.

11. The apparatus according to claim 9, wherein the image registration unit is configured to transform the second image based on the image registration parameters to obtain a fifth image registered with the first image.

12. The apparatus of claim 11, wherein the apparatus further comprises:

and the image fusion unit is used for carrying out image fusion processing on the fifth image and the first image to obtain a sixth image, wherein the sixth image is an image of the first image after noise is removed.

13. The apparatus according to claim 12, wherein the image fusion unit is configured to average the pixel value of each pixel in the fifth image and the pixel value of each pixel in the first image correspondingly, so as to obtain the pixel value of each pixel in the sixth image.

14. The apparatus according to claim 12, wherein the image fusion unit is configured to generate a motion mask image, the motion mask image being used to determine a motion region and a non-motion region; determining the pixel value of each pixel point in the motion area in the first image as the pixel value of each pixel point in the motion area in the sixth image; and correspondingly averaging the pixel values of all the pixel points in the non-motion area in the fifth image and the pixel values of all the pixel points in the non-motion area in the first image to obtain the pixel values of all the pixel points in the non-motion area in the sixth image.

15. The apparatus according to claim 14, wherein the image fusion unit is configured to obtain a pixel value of each pixel in the motion mask image by correspondingly determining a difference between a pixel value of each pixel in the fifth image and a pixel value of each pixel in the first image; the region formed by the pixel points with the pixel values larger than or equal to the threshold value in the motion mask image is a motion region, and the region formed by the pixel points with the pixel values smaller than the threshold value in the motion mask image is a non-motion region.

16. The apparatus according to any one of claims 9 to 15, wherein the invisible light image is an infrared image.

17. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method of any one of claims 1 to 8.

18. A computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the method according to any one of claims 1 to 8.