KR20240049116A - Method for converting near infrared image to rgb image and apparatus for same - Google Patents

Abstract

According to an embodiment of the present invention, a method for converting a near-infrared image into an RGB image by a device including a first network and a second network, includes the steps of: (a) receiving near-infrared image data to be converted into an RGB image, and inputting the same into a first network; (b) extracting, by the first network, feature data of a near-infrared image from the input near-infrared image data; and (c) outputting, by the first network, an RGB image estimated to be converted from the input near-infrared image data into an RGB image using the feature data of the extracted near-infrared image. The first network is a network that receives knowledge to imitate a feature extraction behavior related to extraction of feature data of a near-infrared image and feature data of an RGB image from a second network receiving near-infrared image data which is learning data and RGB image data simultaneously photographed when the corresponding near-infrared image was photographed, and performs knowledge distillation learning.

Description

Method for converting near-infrared image to RGB image and device therefor {METHOD FOR CONVERTING NEAR INFRARED IMAGE TO RGB IMAGE AND APPARATUS FOR SAME}

The present invention relates to a method and device for converting a near-infrared image into an RGB image. More specifically, it relates to a method and device for converting a near-infrared image into an RGB image using knowledge distillation learning, while preserving the texture of the image and faithfully restoring the original color of the object.

Near-infrared images are images in the 700nm to 1000nm wavelength band. Unlike RGB images, near-infrared images are widely used for night vision and surveillance due to their advantages of having less noise and maintaining the texture and details of images even in low-light environments. However, unlike RGB images, they are widely used in color. Because it does not contain information, it is invisible and is not suitable for human perception, and there is a problem in that it cannot be directly applied to computer vision algorithms.

To solve this problem, various studies are being conducted to convert single-band near-infrared images into RGB images while maintaining the advantages of near-infrared images. However, since the near-infrared images themselves do not contain color information, the original color information is Successful restoration is considered a very difficult challenge, and some studies have even stated that the concept of converting from near-infrared images to RGB images is a clearly incorrect problem, so no clear research results have been derived yet.

Republic of Korea Patent Publication No. 10-2017-0103707 (2017.09.13)

The technical problem to be solved by the present invention is to provide a method and device for converting a near-infrared image into an RGB image that can effectively convert a near-infrared image without color information into an RGB image with color information.

Another technical problem that the present invention aims to solve is converting the near-infrared image into a high-quality RGB image that effectively restores the color information of the near-infrared image while maintaining the advantage of the near-infrared image, which is low noise and can maintain the texture and details of the image even in a low-light environment. The aim is to provide a method for converting a near-infrared image into an RGB image and a device for this.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

A method of converting a near-infrared image into an RGB image by a device including a first network and a second network according to an embodiment of the present invention for achieving the above technical problem is (a) near-infrared image data to be converted into an RGB image. receiving input and inputting it to a first network, (b) extracting feature data of the near-infrared image from the received near-infrared image data by the first network, and (c) feature data of the extracted near-infrared image by the first network. Outputting an estimated RGB image to convert the input near-infrared image data into an RGB image using This is a network that performs knowledge distillation learning by receiving knowledge to imitate the feature extraction behavior for extracting feature data of near-infrared images and feature data of RGB images from a second network that also receives image data.

According to one embodiment, the near-infrared image data in step (a) and the near-infrared image data that is the learning data are multi-band near-infrared image data and may be image data acquired through a plurality of different optical filters.

According to one embodiment, the plurality of different optical filters may be three optical band-pass filters with maximum wavelengths of 785 nm, 850 nm, and 940 nm.

According to one embodiment, the first network and the second network may be networks of the same structure including one encoder and two parallel decoders.

According to one embodiment, the first network, the second network extracts feature data of the near-infrared image and feature data of the RGB image, and uses this to produce a black-and-white image and It may be a network that receives knowledge and performs knowledge distillation learning to imitate image estimation behavior regarding RGB image output.

According to one embodiment, after step (c), (d) applying a first loss function (L _rgb ) to the estimated RGB image output in step (c) to output a final RGB image is further performed. Included, the first loss function may be the sum of three loss functions as follows.

First loss function (L _rgb ) = L _content + L _perceptual + L _angular

Reminder here L _content is the L1 norm between the final RGB image and the correct RGB image, L _perceptual is the detail and texture consistency evaluation item of the final RGB image, L _angular is an angle error to improve the color restoration quality of the final RGB image.

According to one embodiment, the knowledge distillation learning is performed by applying a second loss function (L _kd ), and the second loss function may be a loss function as follows.

Second loss function (L _kd ) = ₂

Here, the second loss function is the L2 norm between F ^T , the feature map of the second network, and F ^S , the feature map of the first network, and α is used to make the feature map of the second network more useful for learning the first network. This is an example additional operation to perform.

According to one embodiment, between steps (b) and (c), (b′) the first network converts the input near-infrared image data into a black and white image using the extracted feature data of the near-infrared image. A step of outputting the black and white image estimated to be converted may be further included.

According to one embodiment, between steps (b′) and (c), (b″) a third loss function (L _gray ) is applied to the estimated black and white image output in step (b′) to obtain the final result. It further includes the step of outputting a black-and-white image, and the third loss function may be the sum of two loss functions as follows.

Third loss function (L _gray ) = L _struct + L _SSIM

Here, L _struct is the L1 norm between the final black-and-white image and the correct black-and-white image, and L _SSIM is a loss function to increase the structural similarity between the final black-and-white image and the correct black-and-white image.

A device for converting a near-infrared image into an RGB image according to another embodiment of the present invention for achieving the above technical problem includes one or more processors, a network interface, a memory for loading a computer program executed by the processor, and a large capacity. Storage for storing network data and the computer program, wherein the computer program performs (A) an operation for learning an artificial intelligence classification model, (A) near-infrared image data to be converted to an RGB image by the one or more processors. An operation for receiving input and inputting it to the first network, (B) an operation for the first network to extract feature data of the near-infrared image from the received near-infrared image data, and (C) feature data of the near-infrared image extracted from the first network. An operation is performed to output an estimated RGB image to convert the input near-infrared image data into an RGB image, and the first network uses the near-infrared image data, which is learning data, and the RGB image simultaneously captured at the time of shooting the near-infrared image. This is a network that performs knowledge distillation learning by receiving knowledge to imitate the feature extraction behavior for extracting feature data of near-infrared images and feature data of RGB images from a second network that also receives image data.

A computer program stored in a medium according to another embodiment of the present invention for achieving the above technical problem is combined with a computing device, receiving near-infrared image data to be converted into (AA) RGB image and inputting it to the first network. , (BB) the first network extracting feature data of the near-infrared image from the input near-infrared image data, (CC) the first network using the extracted feature data of the near-infrared image data A step of outputting an RGB image estimated to be converted to an RGB image is performed, wherein the first network is a second network that receives near-infrared image data as learning data and RGB image data taken at the same time as the near-infrared image. This is a networker who performed knowledge distillation learning by receiving knowledge to imitate feature extraction behavior regarding feature data extraction of near-infrared images and feature data of RGB images.

According to the present invention as described above, the port output value of the CPU Execution Port, which is independent of the type of attack, is monitored to detect whether abnormal behavior has occurred, and it is possible to quickly detect whether the CPU abnormality has occurred for all types of attacks. It works.

In addition, the user can freely set the CPU CORE to be monitored and even the CPU Execution Port for the CPU CORE to proceed with the monitoring, which has the effect of accurately detecting the source of abnormal behavior.

In addition, not only the settings of the CPU CORE to be monitored and the corresponding CPU Execution Port, but also the abnormality detection results can be selected and output in the desired way, which has the effect of dramatically improving user convenience.

In addition, the more repeatedly anomaly detection is performed, the more the learning accuracy of the artificial intelligence classification model improves, which has the effect of providing more accurate anomaly detection results.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 is a diagram illustrating the overall configuration of a device for converting a near-infrared image into an RGB image according to a first embodiment of the present invention.
Figure 2 is a flowchart showing representative steps of a method for converting a near-infrared image to an RGB image according to a second embodiment of the present invention.
Figure 3 is a diagram illustrating an example of a multi-band near-infrared image acquired by three optical band-pass filters with the highest wavelengths of 785 nm, 850 nm, and 940 nm for one RGB image.
FIG. 4 is a diagram illustrating the internal structures of a first network and a second network in a method for converting a near-infrared image to an RGB image according to a second embodiment of the present invention.
FIG. 5 is a diagram illustrating the first network and the second network shown in FIG. 4 receiving input data.
Figure 6 is a learning flowchart of the entire first network and second network in the method for converting a near-infrared image to an RGB image according to the second embodiment of the present invention.
Figure 7 is a learning flowchart of the second network in the method for converting a near-infrared image to an RGB image according to the second embodiment of the present invention.
Figure 8 is a learning flowchart of the first network in the method for converting a near-infrared image to an RGB image according to the second embodiment of the present invention.
Figure 9 shows the input data of near-infrared image data and RGB image data into the device, outputting the intermediate output estimated black-and-white image and estimated RGB image, and the final output, the final black-and-white image and final RGB image, and the respective calculation processes. This is a diagram illustrating the loss function applied in .
Figure 10 is a diagram illustrating result 1 of comparison with the prior art regarding the performance of the method for converting a near-infrared image to an RGB image according to the second embodiment of the present invention.
FIG. 11 is a diagram illustrating result 2 of comparison with the prior art regarding the performance of the method for converting a near-infrared image to an RGB image according to the second embodiment of the present invention.

Details regarding the purpose and technical configuration of the present invention and its operational effects will be more clearly understood by the following detailed description based on the drawings attached to the specification of the present invention. Embodiments according to the present invention will be described in detail with reference to the attached drawings.

The embodiments disclosed in this specification should not be construed or used as limiting the scope of the present invention. It is obvious to those skilled in the art that the description, including embodiments, of this specification has various applications. Accordingly, any embodiments described in the detailed description of the present invention are illustrative to better explain the present invention and are not intended to limit the scope of the present invention to the embodiments.

The functional blocks shown in the drawings and described below are only examples of possible implementations. Other functional blocks may be used in other implementations without departing from the spirit and scope of the detailed description. Additionally, although one or more functional blocks of the present invention are shown as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software components that perform the same function.

Additionally, the expression including certain components is an “open” expression and simply refers to the presence of the corresponding components, and should not be understood as excluding additional components.

Furthermore, when a component is referred to as being “connected” or “connected” to another component, it should be understood that although it may be directly connected or connected to the other component, other components may exist in between. do.

Hereinafter, detailed embodiments of the present invention will be looked at with reference to the drawings.

FIG. 1 is a diagram illustrating the overall configuration of an apparatus 100 for converting a near-infrared image into an RGB image according to a first embodiment of the present invention.

However, this is only a preferred embodiment for achieving the purpose of the present invention, and some components may be added or deleted as needed, and of course, the role played by one component may be performed by another component.

The device 100 for converting a near-infrared image into an RGB image according to the first embodiment of the present invention includes a processor 10, a network interface 20, a memory 30, a storage 40, and a data bus 50 connecting them. ), and of course, it may also include other additional components required to achieve the purpose of the present invention.

The processor 10 controls the overall operation of each component. The processor 10 may be one of a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), or an artificial intelligence processor of a type widely known in the technical field to which the present invention pertains. In addition, the processor 10 may perform operations on at least one application or program to perform the method of converting a near-infrared image into an RGB image according to the second embodiment of the present invention. For this purpose, the processor 10 ) includes a first network 11 and a second network 12 that perform knowledge distillation learning, and these can be trained through learning data, which will be described later.

The network interface 20 supports wired and wireless Internet communication of the device 100 for converting a near-infrared image to an RGB image according to the first embodiment of the present invention, and may also support other known communication methods. Accordingly, the network interface 20 may be configured to include a corresponding communication module.

The memory 30 stores various information, commands, and/or information, and one or more computer programs 41 are stored in the storage 40 to perform the method of converting a near-infrared image into an RGB image according to the second embodiment of the present invention. ) can be loaded. In FIG. 1, RAM is shown as one of the memories 30, but of course, various storage media can also be used as the memory 30.

Storage 40 may non-temporarily store one or more computer programs 41 and large-capacity network information 42. This storage 40 includes non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, hard disk (HDD), secondary storage match (SSD), and removable memory. It may be a disk or any type of computer-readable recording medium widely known in the technical field to which the present invention pertains.

The computer program 41 is loaded into the memory 30 and performs (A) an operation of receiving near-infrared image data to be converted into an RGB image and inputting it to the first network by one or more processors 10, (B) the above An operation in which the first network extracts characteristic data of the near-infrared image from the received near-infrared image data, and (C) the first network converts the received near-infrared image data into an RGB image using the extracted feature data of the near-infrared image. You can execute an operation that outputs the estimated RGB image.

The operation performed by the computer program 41 briefly mentioned above can be viewed as a function of the computer program 41, and a more detailed description is provided for the method of converting a near-infrared image into an RGB image according to the second embodiment of the present invention. This will be described later in the explanation.

The data bus 50 serves as a path for moving instructions and/or information between the processor 10, network interface 20, memory 30, and storage 40 described above.

The device 100 for converting a near-infrared image into an RGB image according to the first embodiment of the present invention described above may be in the form of an independent device, for example, an electronic device or a server (including a cloud), where the electronic device is Not only devices such as desktop PCs and server devices that are fixedly installed and used in one place, but also portable devices that are easy to carry, such as smartphones, tablet PCs, laptop PCs, PDAs, and PMPs, are applicable to the processor 10. Any electronic device that has a network function and a CPU installed can be used.

Hereinafter, on the premise that the device 100 for converting a near-infrared image into an RGB image according to the first embodiment of the present invention is in the form of a "server" among the electronic devices that are independent devices, a user who wants to convert a near-infrared image into an RGB image The process of providing a method for converting a near-infrared image into an RGB image according to the second embodiment of the present invention through a dedicated application installed on a user terminal (not shown) will be described with reference to FIGS. 2 to 11.

Figure 2 is a flowchart showing representative steps of a method for converting a near-infrared image to an RGB image according to a second embodiment of the present invention.

However, this is only a preferred embodiment in achieving the purpose of the present invention, and some steps may be added or deleted as needed, and one step may be performed by being included in another step.

Meanwhile, it is assumed that each step is performed through the device 100 for converting a near-infrared image into an RGB image according to the first embodiment of the present invention, and converting the near-infrared image into an RGB image according to the first embodiment of the present invention. Since it was assumed that the device 100 is in the form of a “server,” the dedicated application installed on the user terminal (not shown) is the same as the device 100 for converting a near-infrared image into an RGB image according to the first embodiment of the present invention. This will be viewed as meaning, and for convenience of explanation, all of these will be named “device 100.”

In addition, among the words to be used below, "video" and "image" may be used interchangeably with the same meaning, since a video from a specific viewpoint is an image, and an image is a connection of images from a specific viewpoint to all viewpoints. Although they are in principle distinct concepts, their utilization is treated the same within the method of converting a near-infrared image into an RGB image according to the second embodiment of the present invention.

First, the device 100 receives near-infrared image data to be converted into an RGB image and inputs it to the first network 11 (S210).

Here, the near-infrared image data to be converted to an RGB image is not single-band near-infrared image data as in the prior art, but image data acquired through a plurality of different optical filters for the near-infrared region between 700 nm and 1000 nm, for example, the highest wavelength is 785 nm. , may be multi-band near-infrared image data acquired with three optical band-pass filters of 850 nm and 940 nm.

The device for converting a near-infrared image into an RGB image according to the second embodiment of the present invention uses multi-band near-infrared image data as input data, and the multi-band near-infrared image data contains more information than single-band near-infrared image data. Because there are many, it is useful for finding correlations between each near-infrared band and RGB spectrum.

The device 100 can receive such multi-band near-infrared image data, where the input is a dedicated camera (not shown), which is a separate device from the device 100, for example, capable of shooting near-infrared images and RGB images simultaneously. It is a broad concept that includes both cases of receiving multi-band near-infrared image data captured by a camera such as JAI's AD-130GE directly or through a network, and the device 100 itself is not in the form of a server but constitutes a camera. In the case of electronic devices including, the concept includes cases where multi-band near-infrared image data captured by a camera is received from a camera module in order to be processed.

Figure 3 is an exemplary diagram showing a multi-band near-infrared image (NIR) acquired with three optical band-pass filters with the highest wavelengths of 785 nm, 850 nm, and 940 nm for one RGB image. The RGB image including color information is shown in Figure 3. Regarding this, it can be confirmed that all three near-infrared images are black and white and do not contain color information like the RGB images.

The device 100 inputs near-infrared image data to be converted into an RGB image into the first network. Since the first network 11 is a network included in the processor 10, it is input to the first network 11. This can be seen as the process of converting an RGB image by the processor 10 to start. Hereinafter, the first network 11 and the second network 12, which have been previously described, will be described.

FIG. 4 is a diagram illustrating the internal structure of the first network 11 and the second network 12 in the method of converting a near-infrared image into an RGB image according to a second embodiment of the present invention.

Referring to FIG. 4, the first network 11 and the second network 12 include one encoder (11-1, 12-1) and two parallel decoders (11-2-1/11-2-2, It can be confirmed that it is a network of the same structure including 12-2-1/12-2-2) (of course, the last prediction layer of the parallel decoder may be different), the first network 11 and the second network ( 12) performs Knowledge Distillation learning in which one network transfers the knowledge acquired through learning to another network to learn, so one network is the Teacher Network and the other is the Student Network ( Student Network).

Here, knowledge distillation learning is a learning method that applies the knowledge distillation technique. It is a known learning method that can reduce the learning time by utilizing parameter values when creating a new model through the learned basic model and can also achieve lightweighting of the algorithm. It is suitable for implementing a deep learning model optimized for lightweight user terminals such as mobile or IoT devices. A network that transfers self-learned knowledge to other networks is a teacher network, and a network that receives knowledge learned by other networks and learns with its own knowledge The network is called the student network.

In the method of converting a near-infrared image into an RGB image according to the second embodiment of the present invention, the first network 11 is defined as a student network and the second network 12 is defined as a teacher network, and the internal structures of the two networks are defined as However, as exemplarily shown in FIG. 5, the first network 11, which is a student network, receives only multi-band near-infrared image data as input data, and the second network 12, which is a teacher network, receives multi-band near-infrared image data. In addition, the difference is that RGB image data captured at the same time as the near-infrared image data is received as input data, and both networks can output RGB images.

Figure 6 is a learning flowchart of the entire first network 11 and the second network 12 in the method for converting a near-infrared image to an RGB image according to the second embodiment of the present invention, and Figure 7 is a learning flowchart of the second network 12 ), FIG. 8 is a learning flowchart of the first network 11.

However, this is only a preferred embodiment in achieving the purpose of the present invention, and some steps may be added or deleted as needed, and one step may be performed by being included in another step.

Referring to FIG. 6, first, the second network 12, which is a teacher network, performs learning (S610). Referring to FIG. 7 illustrating this in detail, the second network 12 receives multi-band near-infrared image data and RGB image data taken simultaneously when shooting the near-infrared image data as input data (S610-1) and encodes the encoder ( 12-1) extracts and learns the feature data of the near-infrared image and the feature data of the RGB image (correlation between the multi-band near-infrared image and the RGB spectrum) from this (S610-2), and based on this, two parallel decoders ( 12-2-1/12-2-2) Similar to the black and white image in which the texture of each multi-band near-infrared image is preserved, the texture of the multi-band near-infrared image is preserved and at the same time, an RGB image in which the original color of the object is restored is estimated and output. and learn it (S610-3, S610-3-1/S610-3-2).

Afterwards, the first network 11, which is a student network, performs learning using knowledge distillation learning (S620). Referring to FIG. 8 specifically showing this, the first network performs learning with the same data set as the second network 12, which is a teacher network, but does not receive RGB image data as input data, but multi-band near-infrared image data. receives only the input (S620-1), and learns the near-infrared image by receiving knowledge to imitate the feature extraction behavior for extracting the feature data of the near-infrared image and the feature data of the RGB image from the second network 12 that first performed learning. Extract the features of the data (S620-2) and use the corresponding knowledge from the second network 12 to imitate the image estimation behavior for black-and-white image and RGB image output from the near-infrared image data and RGB image data, which are learning data. By receiving and learning knowledge, it is possible to estimate and output a black-and-white image with the texture of the multi-band near-infrared image preserved and an RGB image with the original color of the object faithfully restored while preserving the texture of the multi-band near-infrared image (S620-3, S620-3-1/S620-3-2).

In this way, since the first network 11, which is a student network, receives and learns the knowledge learned by the second network 12, which is a teacher network, the learning time can be drastically shortened, and the learning algorithm can also be lightweight. This is because the input data that the second network 12 receives includes not only near-infrared image data but also RGB image data. As a result, the second network 12 is more diverse than when it receives only near-infrared image data as input data. It possesses abundant knowledge, and by transmitting the knowledge as is to the first network 11, the first network 11 improves the detail and color quality of the RGB image according to the second network 12 without inputting RGB image data. This is because you will have the knowledge to do so.

Meanwhile, learning of the knowledge transmitted by the second network 12 to the first network 11, more specifically knowledge distillation learning, can be achieved by applying the second loss function (L _kd ), and the second loss function is as follows: same.

Second loss function (L _kd ) = ₂

Here, the second loss function is the L2 norm between F ^T , which is the feature map of the second network 12, and F ^S, which is the feature map of the first network 11, and α is the feature map of the second network 12. 1 This is an example additional operation performed to make network (11) more useful for network learning.

Separately from this, during the learning process of the first network 11, which is a student network, the learning of the second network 12, which is a teacher network, is not additionally performed, and the knowledge is maintained in the same state in which the knowledge was transferred to the first network 11. , Learning about the knowledge received from the second network 12 may be performed before step S620-1, or extract features of near-infrared image data in step S620-2 or S620-3-1 and S620-3-2. It may be performed together in the step of estimating and outputting the black-and-white image and the RGB image, and it is not meaningful as to which step the learning is performed, but rather effectively learning the knowledge received from the second network 12. Technical meaning should be attached to the fact that it does so.

Let us return to the description of FIG. 2 again.

If near-infrared image data is received and input into the first network 11, the device 100 extracts feature data of the near-infrared image from the near-infrared image data received by the first network 11 (S220).

Here, in extracting the feature data of the near-infrared image, the first network 11 uses the knowledge received from the second network 12 described above, more specifically, the feature data of the near-infrared image and the features of the RGB image from the second network 12. The received knowledge can be used to imitate feature extraction behavior regarding data extraction, and detailed explanations are omitted to prevent redundant description.

Meanwhile, after step S220 and before proceeding to step S230, the device 100 converts the input near-infrared image data into a black-and-white image using the feature data of the near-infrared image extracted by the first network 11. Can be printed (S225).

Since this is because the first network 11 estimates and outputs a black-and-white image, S620 estimates and outputs a black-and-white image with the texture of the multi-band near-infrared image preserved, which was previously described with reference to FIG. 8 regarding learning of the first network 11. -It is the same as step 3-1, and detailed description will be omitted to prevent redundant description. In the first network 11 for estimating and outputting the image converted from the near-infrared image to the RGB image, the black and white image is used in addition to the RGB image. By estimating and outputting the image, it can contribute to improving RGB image estimation performance.

Furthermore, the device 100 may output a final black-and-white image by applying the third loss function (L _gray ) to the estimated black-and-white image output in step S225 (S227).

Here, the third loss function may be the sum of two loss functions as follows.

Third loss function (L _gray ) = L _struct + L _SSIM

Here, L _struct is the L1 norm between the final black and white image and the correct black and white image. By using L _struct as the L1 norm, a black and white image with less blurring than the L2 norm can be obtained, which can be expressed as follows.

L _struct = - _One

L _SSIM is a loss function to increase the structural similarity between the final black and white image and the correct black and white image, and can be expressed as follows.

L _SSIM = 1 - SSIM( , I _gray )

The third loss function, which is the sum of the above two loss functions, can be viewed as a loss function used to estimate and output a black and white image.

If the feature data of the near-infrared image has been extracted, the device 100 uses the feature data of the near-infrared image extracted by the first network 11 to output an estimated RGB image to convert the input near-infrared image data into an RGB image (S230) ).

Here, in estimating and outputting the RGB image, the first network 11 uses the knowledge received from the second network 12 described above, more specifically, the knowledge received from the second network 12 (received to imitate feature extraction behavior). Knowledge) can be used to use the knowledge received to imitate image estimation behavior for black-and-white and RGB image output from the near-infrared image data and RGB image data, which are learning data. Detailed explanations will be omitted to prevent redundant description. .

Furthermore, the device 100 may apply the first loss function (L _rgb ) to the estimated RGB image output in step S230 and output the final RGB image (S235).

Here, the first loss function may be the sum of two loss functions as follows.

First loss function (L _rgb ) = L _content + L _perceptual + L _angular

Here, L _content is the L1 norm between the final RGB image and the correct RGB image. By using L _struct as the L1 norm, an RGB image with less blurring than the L2 norm can be obtained, which can be expressed as follows.

L _content = - _One

L _perceptua l is an evaluation item for consistency of detail and texture of the final RGB image, and can be expressed as follows.

L _perceptual = _rgb ) - _layer

L _angular is the angular error to improve the color restoration quality of the final RGB image, and can be expressed as follows.

L _angular = arcos( )

The first loss function, which is the sum of the three loss functions above, can be viewed as a loss function used to estimate and output an RGB image.

Figure 9 shows an estimated black and white image, which is an intermediate product, by inputting near-infrared image data (I _nir ) and RGB image data (I _rgb ), which are input data, into the device 100. _gray ) and estimated RGB image ( _rgb ), the output of the final black-and-white image (I _gray ) and the final RGB image (I _rgb ), which are the final output, and the loss function applied in each calculation process, as shown in Figures 9 and above. According to the description, the total loss function (L _total ) applied to the entire device 100 can be expressed as the sum of the first loss function, the second loss function, and the third loss function as follows.

Total loss function (L _total ) = L _kd + L _gray + L _rgb

So far, a method for converting a near-infrared image into an RGB image according to the second embodiment of the present invention has been described. According to the present invention, by applying knowledge distillation learning, the second network 12, which is a teacher network, receives as input data not only near-infrared image data but also RGB image data taken at the same time as the near-infrared image data, and from this, near-infrared image data is received. Knowledge to imitate feature extraction behavior related to extracting feature data of images and feature data of RGB images, and knowledge to use this to imitate image estimation behavior related to black-and-white image and RGB image output from near-infrared image data and RGB image data. It is transmitted to the first network (11), which is the student network, to learn quickly and efficiently. Even if only a near-infrared image that does not contain color information is received as input data, it can be effectively converted to an RGB image that contains color information. You can. In addition, we aim to improve the quality of image conversion by applying a loss function for knowledge distillation learning, a loss function for final black-and-white image estimation, and a loss function for final RGB image estimation. It is possible to achieve high-quality RGB image conversion that faithfully restores the original color of the object while maintaining detail. In addition, more detailed information can be obtained by using multi-band near-infrared images acquired with three different optical band pass filters rather than single-band near-infrared images, which is useful for finding correlations between each near-infrared band and RGB spectrum. Learning efficiency can be improved.

Figure 10 shows the method for converting a near-infrared image into an RGB image according to the second embodiment of the present invention, from the left, the near-infrared image as input data, the correct answer RGB image, U-Net, S-Net, and Structure corresponding to the prior art. -Aware's estimated image output results, and finally, a diagram showing the estimated RGB image output results according to the present invention. In comparison with the correct answer RGB image, it can be seen that the results of U-Net, which corresponds to the prior art, cause a lot of loss of text readability and texture information, and the results of S-Net and Structure-Arare show visually unnatural effects in local areas. While it can be confirmed that color distortion or loss of texture information occurs, according to the method of converting a near-infrared image into an RGB image according to the second embodiment of the present invention, the readability and texture information of letters are well preserved while color restoration quality is improved. Since this excellent RGB video is being output, it can be confirmed that it has excellent video conversion quality.

Figure 11 shows the method of converting a near-infrared image into an RGB image according to the second embodiment of the present invention, where the upper and lower left sides are the near-infrared image as input data and the correct answer RGB image, and the upper center is the image where knowledge distillation learning was not performed. The estimated video output result in this case, the upper right is the estimated video output result when knowledge distillation learning is applied only between encoders (feature extraction behavior imitation), and the bottom center is the estimated video output result when knowledge distillation learning is applied only between decoders (image estimation behavior imitation). As for the estimated image output result, the bottom right is a diagram showing the estimated RGB image output result according to the present invention. In the case where knowledge distillation learning is not performed, compared to the case where knowledge distillation learning is applied only between encoders or between decoders, it can be seen that the texture is not properly preserved and the quality of color restoration is also poor. This is evidence that helps improve the quality of RGB conversion of images, and the method of converting near-infrared images into RGB images according to the second embodiment of the present invention is a case of applying both knowledge distillation learning between the encoder and decoder, so the correct answer is RGB Compared to video, it can be confirmed that it has the best video conversion quality.

Meanwhile, the device 100 for converting a near-infrared image into an RGB image according to the first embodiment of the present invention and the method for converting a near-infrared image into an RGB image according to the second embodiment of the present invention are the third embodiment of the present invention. It may be implemented as a computer program stored in a computer-readable medium according to the following, in this case, receiving near-infrared image data to be converted into an RGB image by combining with a computing device (AA) and inputting it to the first network, (BB) Extracting characteristic data of the near-infrared image from the received near-infrared image data by the first network, (CC) the first network converts the received near-infrared image data into an RGB image using the extracted feature data of the near-infrared image. The step of outputting the RGB image estimated to be converted can be performed, and although not described in detail for redundant description, the device 100 for converting a near-infrared image into an RGB image according to the first embodiment of the present invention and the device 100 of the present invention It goes without saying that all technical features applied to the method for converting a near-infrared image to an RGB image according to the second embodiment can be equally applied to the computer program stored in the computer-readable medium according to the third embodiment of the present invention. .

Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

10: processor
11: first network
12: Second network
11-1. 12-1: Encoder
11-2-1, 11-2-2, 12-2-1, 12-2-2: Decoder
20: network interface
30: memory
40: storage
41: computer program
50: Information Bus
100: Device that converts near-infrared images into RGB images

Claims (11)

In a method where a device including a first network and a second network converts a near-infrared image into an RGB image,
(a) receiving near-infrared image data to be converted into an RGB image and inputting it to a first network;
(b) the first network extracting feature data of a near-infrared image from the received near-infrared image data; and
(c) outputting an estimated RGB image by the first network to convert the input near-infrared image data into an RGB image using feature data of the extracted near-infrared image;
Includes,
The first network is,
Knowledge is transferred to imitate the feature extraction behavior of extracting the feature data of the near-infrared image and the feature data of the RGB image from the second network that receives the learning data, the near-infrared image data, and the RGB image data taken at the same time as the near-infrared image. A network that performed knowledge distillation learning,
How to convert near-infrared images to RGB images. According to paragraph 1,
The near-infrared image data in step (a) and the near-infrared image data that are the learning data,
It is multi-band near-infrared image data,
Image data acquired with a plurality of different optical filters,
How to convert near-infrared images to RGB images. According to paragraph 2,
The plurality of different optical filters,
Three optical band pass filters with peak wavelengths of 785nm, 850nm and 940nm,
How to convert near-infrared images to RGB images. According to paragraph 1,
The first network and the second network are,
A network of the same structure including one encoder and two parallel decoders,
How to convert near-infrared images to RGB images. According to paragraph 1,
The first network is,
The second network extracts feature data of the near-infrared image and feature data of the RGB image, and uses this to imitate image estimation behavior regarding black-and-white image and RGB image output from the near-infrared image data and RGB image data, which are the learning data. A network that received information and performed knowledge distillation learning,
How to convert near-infrared images to RGB images. According to paragraph 1,
After step (c) above,
(d) applying a first loss function (L _rgb ) to the estimated RGB image output in step (c) to output a final RGB image;
It further includes,
The first loss function is the sum of three loss functions as follows,
How to convert near-infrared images to RGB images.
First loss function (L _rgb ) = L _content + L _perceptual + L _angular
Reminder here L _content is the L1 norm between the final RGB image and the correct RGB image, L _perceptual is the detail and texture consistency evaluation item of the final RGB image, L _angular is an angle error to improve the color restoration quality of the final RGB image. According to paragraph 1,
The knowledge distillation learning is performed by applying the second loss function (L _kd ),
The second loss function is a loss function as follows,
How to convert near-infrared images to RGB images.
Second loss function (L _kd ) = ₂
Here, the second loss function is the L2 norm between F ^T , the feature map of the second network, and F ^S , the feature map of the first network, and α is used to make the feature map of the second network more useful for learning the first network. This is an example additional operation to perform. According to paragraph 1,
Between steps (b) and (c),
(b′) outputting an estimated black-and-white image by the first network to convert the input near-infrared image data into a black-and-white image using the extracted feature data of the near-infrared image;
A method of converting a near-infrared image to an RGB image further comprising: According to clause 8,
Between steps (b′) and (c),
(b″) applying a third loss function (L _gray ) to the estimated black-and-white image output in step (b′) to output a final black-and-white image;
It further includes,
The third loss function is the sum of two loss functions as follows,
How to convert near-infrared images to RGB images.
Third loss function (L _gray ) = L _struct + L _SSIM
Here, L _struct is the L1 norm between the final black-and-white image and the correct black-and-white image, and L _SSIM is a loss function to increase the structural similarity between the final black-and-white image and the correct black-and-white image. One or more processors;
network interface;
a memory that loads a computer program executed by the processor; and
Including storage for storing large-capacity network data and the computer program,
The computer program is operated by the one or more processors,
(A) An operation of receiving near-infrared image data to be converted into an RGB image and inputting it to a first network;
(B) an operation in which the first network extracts feature data of a near-infrared image from the received near-infrared image data; and
(C) an operation in which the first network outputs an estimated RGB image to convert the input near-infrared image data into an RGB image using feature data of the extracted near-infrared image;
Execute ,
The first network is,
Knowledge is transferred to imitate the feature extraction behavior of extracting the feature data of the near-infrared image and the feature data of the RGB image from the second network that receives the learning data, the near-infrared image data, and the RGB image data taken at the same time as the near-infrared image. A network that performed knowledge distillation learning,
A device that converts near-infrared images into RGB images. In combination with a computing device,
(AA) receiving near-infrared image data to be converted into an RGB image and inputting it to the first network;
(BB) extracting feature data of a near-infrared image from the received near-infrared image data by the first network;
(CC) outputting, by the first network, an estimated RGB image to convert the input near-infrared image data into an RGB image using feature data of the extracted near-infrared image;
Execute ,
The first network is,
Knowledge is transferred to imitate the feature extraction behavior of extracting the feature data of the near-infrared image and the feature data of the RGB image from the second network that receives the learning data, the near-infrared image data, and the RGB image data taken at the same time as the near-infrared image. A network that performed knowledge distillation learning,
A computer program stored on a computer-readable medium.