KR102138789B1 - Exemplar-based IMAGE PROCESSING method and IMAGE PROCESSING appratus using selective patch search - Google Patents

Abstract

Disclosed is an image processing method. In an image processing method according to an embodiment of the present invention, a background area excluding an area to be restored from an original image is grouped with areas having similar pixel values, and the area to be restored is the background area Setting a part of the boundary area in contact with the reconstructed block as a reconstructed block; searching for a patch block having a pixel value most similar to the reconstructed block in the background area; and the reconstructed block using a pixel value of the retrieved patch block And restoring the reconstructed block, and when the reconstructed block includes pixel values belonging to a plurality of groups, the reconstructed block is filled using a patch block positioned in a boundary area of the plurality of groups.

Description

Patch-based image processing method and image processing device using selective search {Exemplar-based IMAGE PROCESSING method and IMAGE PROCESSING appratus using selective patch search}

The present invention relates to an image processing technique, and more particularly, to an image processing apparatus and an image processing method that provide a more natural interpolation effect when interpolating a region of an image.

Along with display technology, various image processing technologies are being developed. In particular, an image interpolation technique is often used to transform a 2D image into a 3D image. In this case, the region to be reconstructed may be formed in a form in which similar pixel images are searched for and filled, which is called an inpainting technique.

1 is a diagram illustrating an embodiment of image interpolation using an inpainting technique.

The image of FIG. 1A is composed of a plurality of objects, and a situation in which one of the objects 10 is deleted is shown. 1B shows an image from which the object 10 is deleted and the deleted region 11 is interpolated. It can be seen that the deleted area is filled with pixel values of the background object.

This image restoration algorithm has two important design issues.

The first is the processing speed of the algorithm. Since the image restoration algorithm essentially requires processing in units of pixel values, the overhead is inevitably large. Therefore, a faster and more efficient algorithm design is required.

The second is the quality of the restored image. Since the reconstructed image is to create the pixel area where the pixel value disappears from the surrounding pixel values, the result must be natural so that it matches the surrounding background.

In the case of the first issue, the hardware performance is improving in recent years, and the processing speed depends on the number of pixels to be processed, that is, the resolution, so there is a limit to the efficiency of the algorithm itself.

The second issue is that the result difference may be large depending on the processing method. In particular, when the region to be restored includes the boundary portions of different objects, natural processing becomes a problem. That is, when the region to be reconstructed includes all pixel regions having small similarities, an appropriate boundary region processing technique is required.

The present invention is to solve the above-described problems, and an object of the present invention is to provide an image processing apparatus for naturally restoring a boundary area when the area to be reconstructed includes all pixel areas having small similarities. And an image processing method.

An image processing method according to an embodiment of the present invention for achieving the above object includes the steps of grouping a background region excluding a region to be reconstructed from an original image into regions having similar pixel values, Setting a part of a boundary area in which the area to be restored contacts the background area as a reconstructed block; searching for a patch block having a pixel value most similar to the reconstructed block in the background area; and Restoring the reconstructed block using pixel values, and when the reconstructed block includes pixel values belonging to a plurality of groups, the reconstructed block using a patch block located in a boundary area of the plurality of groups Fill in.

In this case, in the grouping step, the background region may be grouped by grouping regions having a preset pixel value range.

In addition, the step of searching for the patch block may include searching for the patch block from a group having a preset pixel value range to which the pixel value of the reconstructed block belongs.

In addition, the reconstructing the reconstructed block may include filling the reconstructed block with pixel values of the retrieved patch block.

In addition, when the restoration block is restored, the background region may include the restored restoration block.

In addition, the step of searching for the patch block may be performed by comparing a degree of similarity between a pixel excluding a pixel of the region to be reconstructed and a pixel of the background region in the reconstructed block.

Further, in the step of searching for the patch block, the patch block may be searched for a region of the background region to which a pixel excluding a pixel of the region to be restored belongs among the reconstructed blocks.

In an image processing apparatus according to an embodiment of the present invention for achieving the above object, a background region excluding a region to be restored from an original image is grouped with regions having similar pixel values, and the region to be restored is A reconstructed block setting unit configured to set a part of the boundary area in contact with the background area as a reconstructed block; a patch block search unit configured to search for a patch block having a pixel value most similar to the reconstructed block in the background area; and the searched And a control unit for reconstructing the reconstructed block using pixel values of the patch block, wherein the control unit includes a patch located in a boundary area of the plurality of groups when the reconstructed block includes pixel values belonging to a plurality of groups The reconstructed block is filled using blocks.

In this case, the reconstructed block setting unit may group the background region by grouping regions having a preset pixel value range.

In addition, the patch block search unit may search for the patch block from a group having a preset pixel value range to which the pixel value of the reconstructed block belongs.

Also, the control unit may fill the reconstructed block with pixel values of the retrieved patch block.

In addition, when the restoration block is restored, the background region may include the restored restoration block.

In addition, the patch block search unit may compare a degree of similarity between a pixel excluding a pixel of the region to be reconstructed and a pixel of the background region in the reconstructed block.

In addition, the patch block search unit may perform a search for the patch block in an area of the background area to which a pixel other than the pixel of the area to be restored belongs among the reconstructed blocks.

According to various embodiments of the present disclosure as described above, in the case of image restoration, when the regions to be restored include all pixel regions having small similarities, the boundary region may be restored naturally.

1 is a diagram showing an embodiment of image interpolation using an inpainting technique;
2 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention;
3A is a diagram illustrating an embodiment of grouping background regions;
3B to 3C are diagrams illustrating a search area of a patch block according to an embodiment of the present invention;
3D is a diagram showing image restoration processing of a boundary area of a different background area;
FIG. 4A is a diagram illustrating a case in which a reconstructed block includes a boundary area meeting a plurality of background areas;
4B is a diagram illustrating a search of a patch block when a reconstructed block meets a plurality of background groups;
4C is a diagram showing a situation of restoring a reconstructed block using a patch block, and,
5 is a flowchart of an image processing method according to an embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings.

2 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention.

Referring to FIG. 2, an image processing apparatus 100 according to an embodiment of the present invention includes a reconstructed block setting unit 110, a patch block search unit 120, and a control unit 130.

The reconstructed block setting unit 110 is configured to group background regions and set a reconstructed block. First, the reconstructed block setting unit 110 groups a background area excluding an area to be reconstructed from an original image into areas having similar pixel values.

3A is a diagram illustrating an example in which background regions are grouped.

In the present specification, an area excluding the area to be restored 30 as shown in FIG. 3A is defined as the background areas 33, 35, 37, and 39. The reconstructed block setting unit 110 groups the background regions 33, 35, 37, and 39 with regions having similar pixel values. This means that the background regions are grouped by grouping regions having a preset pixel value range. For example, a region 33 having a range of pixel values a to b is referred to as index 1, a region 35 having a range of pixel values b to c is referred to as an index 2, and a region 37 having a range of pixel values c to d Is set to index 3, and the region 39 having a range of pixel values d to e may be set to index 4. However, in the case of a live image such as a movie, since the distribution of pixel values will be very diverse, the group may actually be very broad, and thus, the above content is only an exemplary embodiment.

Further, the restoration block setting unit 110 sets a part of a boundary area in which the area to be restored contacts the background area as a restoration block. In FIG. 3A, it can be seen that some of the boundary regions in which the region 30 to be restored is in contact with the background regions 33 to 39 are set as the restoration block 32. The reconstructed block is enlarged and shown in the drawings after FIG. 3B.

Here, the reconstructed block may be a set of pixels having a preset size. For example, it can achieve a size of 5x5 pixels horizontally and vertically. As for the region to be restored, a restoration block (including both the region to be restored and some pixels of the background region) is set among the boundary regions with the background region, and restoration is performed from the restoration block. Once the reconstructed block is restored, the restored block is included as a background area and excluded from the area to be restored.

When the reconstructed block is set, the patch block search unit 120 searches for a patch block having the highest similarity to the reconstructed block. The patch block is a pixel group for reconstructing the reconstructed block and is set to the same size as the reconstructed block. In the above-described embodiment, if the reconstructed block is a 5x5 pixel size, the patch block is also the same size.

The patch block search unit 120 searches for a patch block in the background area. At this time, the background area to be searched is adjacent to the reconstructed block, and only the background area existing within a preset range is applicable.

For example, if relatively similar colors such as animations are widely arranged, a background area in which patch blocks are searched may be relatively wide. However, in the case of a live screen such as a movie, since various pixel values are distributed, a background area in which a patch block is searched may be relatively narrow.

Also, the patch block may be searched in a group having a preset pixel value range to which the pixel value of the reconstructed block belongs. The reconstructed block includes both pixel values of the area to be reconstructed and pixel values of the background area. In this case, the pixel value of the background area included in the reconstructed block is a criterion for determining the similarity. That is, a patch block is searched in a background area having a pixel value in a range similar to that of a background area included in the reconstructed block.

Meanwhile, the similarity between the reconstructed block and the patch block is determined by the degree of similarity between pixels. This can be expressed by the degree of identity of the pixel value and the pixel position. Since the reconstructed block has some pixels of the area to be reconstructed, substantially only the pixel portion of the background area is compared.

3B to 3C are diagrams illustrating a search area of a patch block according to an embodiment of the present invention.

3B is an enlarged view of an area to be restored in FIG. 3A. It can be seen that the region to be restored is in contact with a plurality of background regions (indexes 1 and 4). In this case, the reconstructed block 32 includes only pixel values of a single background area other than the area to be reconstructed, and the area to search for a patch block is limited to an area (index 4) having a pixel value of the single background area.

FIG. 3C is an enlarged view of an area to be restored in FIG. 3A. Similarly, it can be seen that the region to be restored is in contact with a plurality of background regions (indexes 1 and 4). However, the restoration block 32 includes only pixel values of a single background area (index 1) other than the area to be restored, and the area to search for a patch block is limited to an area (index 1) having a pixel value of the single background area. do.

However, since the region to be restored is in contact with a plurality of background regions, processing of the border region of the plurality of background regions is problematic. 3D presents one solution.

3D is a diagram illustrating image restoration processing of a boundary area of a different background area.

As shown in FIG. 3D, it is possible to process only the reconstructed block of the area to be reconstructed that meets a single background area. That is, since the restoration block 32-1 of FIG. 3D meets the background area at the top, a similar patch block 40-1 is found in the background area at the top, and the restoration block 32-2 meets the background area at the bottom. Therefore, a similar patch block 40-2 is found in the lower background area.

However, this method has a problem in that when an area to be restored meets a plurality of background areas, a restoration block must be set excluding a boundary surface of the plurality of background areas.

As described above, FIG. 4A is a diagram illustrating a case in which a reconstructed block is set excluding boundary surfaces of a plurality of background areas when the reconstructed block includes a border area meeting a plurality of background areas. In this case, when the boundary surface is restored, the patch blocks are imported from different background regions, so that the reconstructed boundary surface formed by the different patch blocks may be unnatural.

As described above, since no processing is performed on the boundary of the different background regions, the image of the boundary portion disappears, so that the boundary portion of the final reconstructed image becomes very unnatural.

As a solution to this, the patch block search unit 120 sets the reconstructed block to include pixel values belonging to a plurality of groups, and in this case, searches for a patch block located in a boundary area of the plurality of groups. Fig. 4b shows this embodiment.

That is, FIG. 4B is a diagram illustrating a search for a patch block when a reconstructed block meets a plurality of background groups.

As shown in FIG. 4B, when the restoration block 32 includes a plurality of background areas, that is, background areas belonging to different groups, the patch block search unit 120 searches for patch blocks located in the boundary areas of the plurality of groups. Search. In addition, interpolation is performed using the patch block having the highest similarity among the searched patch blocks.

The controller 130 performs overall operations of the image processing apparatus 100, and in particular, controls the operations of the above-described restoration block setting unit 110 and patch block search unit 120. Also, the reconstructed block is filled using the retrieved patch block.

4C is a diagram illustrating a situation in which a reconstructed block is restored using a patch block.

As shown in FIG. 4C, the controller 130 fills the reconstructed blocks in contact with a plurality of background areas by using a patch block including a boundary area. Of course, a reconstructed block in contact with a single background area can also be filled using a patch block retrieved from a single background area. Once the restoration block is filled, the filled restoration block is included as a background area and is excluded from the area to be restored. Accordingly, a boundary is newly set between the region to be restored and the background. Since the background has changed, the grouping operation for the background may be performed again. However, since the pixel value of the reconstructed block newly added as a background is already known, it is included in a corresponding group, and the entire grouping process may be omitted.

The above-described process is repeated until all areas to be restored disappear.

The control unit 130 includes a central processing unit (CPU) or a micro processing unit (MPU), a cache, and a memory as a hardware, and includes an operating system, an application, and a framework as a software, and an FPGA, etc. It can be implemented as A control command for each component for the operation of the image processing apparatus 100 is read from the memory according to the system clock, and an electric signal is generated according to the read control command to operate each component of the hardware. In addition, the above-described recovery block setting unit 110, patch block search unit 120, etc. may also be included and implemented as part of the configuration of the control unit 130, and may be implemented as a separate application or implemented as a separate embedded chip. It is possible.

Further, although not depicted in the drawings, the image processing apparatus 100 of the present invention includes a configuration of a general electronic device. In other words, display (panel, drive unit), input device (remote control, mouse, keyboard, motion input unit, touch input unit, gesture input unit), camera, storage unit of sufficient capacity (memory, hard disk), wired/wireless communication interface (video transmission/reception), etc. Includes.

In addition, it can be implemented with various electronic devices, and can be implemented with various display devices such as digital televisions, kiosks, smart phones, tablet PCs, PMPs, set-top boxes, smart watches, and large displays.

Hereinafter, an image processing method according to various embodiments of the present disclosure will be described.

5 is a flowchart of an image processing method according to an embodiment of the present invention.

Referring to FIG. 5, in the image processing method according to an embodiment of the present invention, a background region excluding a region to be reconstructed from an original image is grouped with regions having similar pixel values (S510), the Setting a part of a boundary area in which an area to be restored contacts the background area as a restoration block (S520), searching for a patch block having a pixel value most similar to the restoration block in the background area (S530), and the searched Including the step of restoring the reconstructed block using pixel values of the patch block (S540), and when the reconstructed block includes pixel values belonging to a plurality of groups, a patch block positioned in a boundary area of the plurality of groups The restoration block is filled by using.

In this case, in the grouping step, the background region may be grouped by grouping regions having a preset pixel value range.

In addition, the step of searching for the patch block may include searching for the patch block from a group having a preset pixel value range to which the pixel value of the reconstructed block belongs.

In addition, the reconstructing the reconstructed block may include filling the reconstructed block with pixel values of the retrieved patch block.

In addition, when the restoration block is restored, the background region may include the restored restoration block.

In addition, the step of searching for the patch block may be performed by comparing a degree of similarity between a pixel excluding a pixel of the region to be reconstructed and a pixel of the background region in the reconstructed block.

Further, in the step of searching for the patch block, the patch block may be searched for a region of the background region to which a pixel excluding a pixel of the region to be restored belongs among the reconstructed blocks.

Meanwhile, the above-described image processing method may be configured to be stored in a computer-readable non-transitory recording medium in which a program for performing the method is recorded.

Here, the non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by an electronic device, rather than a medium that stores data for a short moment, such as a register or a cache. For example, it may be a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and the present invention is not departing from the gist of the present invention claimed in the claims. Various modifications are possible by those skilled in the art of course, and these modifications should not be individually understood from the technical idea or perspective of the present invention.

100: image processing device
110: restoration block setting unit 120: patch block search unit
130: control unit

Claims (14)

In the image processing method,
Grouping regions having similar pixel values in a background region excluding a region to be reconstructed from the original image;
Setting a part of a boundary area in which the area to be restored contacts the background area as a restoration block;
Searching for a patch block having a pixel value most similar to the reconstructed block in the background area; And
Restoring the reconstructed block using the pixel values of the retrieved patch block; Including,
When the reconstructed block includes pixel values belonging to a plurality of groups, a patch block located in the boundary area of the plurality of groups is searched, and the reconstructed block is filled using a patch block having the highest similarity among the searched patch blocks. , Image processing method. The method of claim 1,
The grouping step,
And grouping and grouping the background regions with regions having a preset pixel value range. According to claim 2,
The step of searching for the patch block,
And searching for the patch block from a group having a preset pixel value range to which the pixel value of the reconstructed block belongs. The method of claim 1,
Restoring the restoration block,
And filling the reconstructed block with pixel values of the retrieved patch block. The method of claim 1,
When the reconstructed block is reconstructed, the background area includes the reconstructed reconstructed block. The method of claim 1,
The step of searching for the patch block,
And comparing a similarity between a pixel excluding a pixel of the region to be reconstructed and a pixel of the background region in the restoration block. The method of claim 1,
The step of searching for the patch block,
And the patch block is searched for a region of the background region to which a pixel other than a pixel of the region to be restored belongs among the reconstructed blocks. In the image processing apparatus,
A restoration block setting in which a background area excluding the area to be restored from the original image is grouped with areas having similar pixel values, and a part of the boundary area in which the area to be restored contacts the background area is set as a restoration block part;
A patch block search unit for searching for a patch block having a pixel value most similar to the reconstructed block in the background area; And
Including; a control unit for restoring the reconstructed block using the pixel value of the retrieved patch block; and
When the reconstructed block includes pixel values belonging to a plurality of groups, the control unit searches for a patch block located in a boundary area of the plurality of groups, and uses a patch block having the highest similarity among the searched patch blocks. An image processing device that fills the reconstructed block. The method of claim 8,
The restoration block setting unit,
The image processing apparatus according to claim 1, wherein the background regions are grouped and grouped with regions having a range of preset pixel values. The method of claim 9,
The patch block search unit,
And searching for the patch block from a group having a preset pixel value range to which the pixel value of the reconstructed block belongs. The method of claim 8,
The control unit,
And filling the reconstructed block with pixel values of the retrieved patch block. The method of claim 8,
When the reconstructed block is restored, the background region includes the reconstructed block. The method of claim 8,
The patch block search unit,
And comparing a degree of similarity between a pixel excluding a pixel of the region to be reconstructed and a pixel of the background region in the reconstructed block. The method of claim 8,
The patch block search unit,
And searching for the patch block in an area of the background area to which a pixel other than the pixel of the area to be reconstructed belongs among the reconstructed blocks.

Images