CN111784734B - Image processing method and device, storage medium and electronic equipment - Google Patents

Abstract

The present disclosure provides an image processing method, an image processing device, a storage medium and an electronic device, and relates to the field of image processing technology. The image processing method comprises: determining the position of the neighboring pixels of the current pixel to be interpolated contained in the output image in the input image, determining the current image block in the input image by the position of the neighboring pixels, and determining all matching blocks corresponding to the current image block in multiple reference frames corresponding to the current frame; performing image block fusion according to the target matching blocks in all matching blocks and the current image block to obtain multiple fusion blocks, and determining the target direction of the current image block according to the multiple fusion blocks; performing directional interpolation on the current image block according to the target direction to obtain the pixel value of the current pixel to be interpolated. The embodiments of the present disclosure can improve image quality.

Description

Image processing method and device, storage medium and electronic device

Technical Field

The present disclosure relates to the field of image processing technology, and in particular to an image processing method, an image processing device, a computer-readable storage medium, and an electronic device.

Background technique

In order to improve the image resolution, the low-resolution image can be converted to a high-resolution image. In the related art, when performing the conversion, there may be errors in the interpolation process due to the limitations of the input image, resulting in poor quality of the output image.

Summary of the invention

The present disclosure provides an image processing method, an image processing device, a computer-readable storage medium, and an electronic device, thereby overcoming the problem of poor image quality at least to a certain extent.

According to one aspect of the present disclosure, an image processing method is provided, comprising: determining positions of neighboring pixels of a current pixel to be interpolated contained in an output image in an input image, determining a current image block in the input image according to the positions of the neighboring pixels, and determining all matching blocks corresponding to the current image block in multiple reference frames corresponding to the current frame; performing image block fusion according to target matching blocks in all matching blocks and the current image block to obtain multiple fused blocks, and determining a target direction of the current image block according to the multiple fused blocks; and performing directional interpolation on the current image block according to the target direction to obtain a pixel value of the current pixel to be interpolated.

According to one aspect of the present disclosure, an image processing device is provided, comprising: a matching block determination module, used to determine the positions of adjacent pixels of a current pixel to be interpolated contained in an output image in an input image, determine the current image block in the input image through the positions of the adjacent pixels, and determine all matching blocks corresponding to the current image block in multiple reference frames corresponding to the current frame; a direction determination module, used to perform image block fusion according to target matching blocks in all matching blocks and the current image block to obtain multiple fused blocks, and determine a target direction of the current image block according to the multiple fused blocks; and an image interpolation module, used to perform directional interpolation on the current image block through the target direction to obtain a pixel value of the current pixel to be interpolated.

According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image processing method as described in any one of the above is implemented.

According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned image processing methods by executing the executable instructions.

In the technical solutions provided by some embodiments of the present disclosure, on the one hand, since all matching blocks corresponding to the current image block in the current frame can be determined in multiple reference frames through motion estimation, the target matching block and the current matching block are fused to obtain the target direction of the current image block, and the pixel value of the current pixel to be interpolated is obtained by directional interpolation in the current image block based on the target direction, and the pixel value of the current pixel to be interpolated is determined by combining multi-frame motion estimation and directional interpolation. Through the matching strategy, directional interpolation of multi-frame fusion in the same region is performed, which can avoid the problems such as direction misjudgment caused by single-frame directional interpolation in the related technology, avoid limitations, and can comprehensively and accurately determine the target direction based on the texture direction. On the other hand, through motion estimation and directional interpolation of multi-frame fusion in the same region, the pixel value of the current pixel to be interpolated can be accurately calculated, the image quality can be improved, the effect of image super-resolution can be improved, and the operation complexity can be reduced and the operability can be increased.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

FIG1 is a schematic diagram showing an exemplary system architecture to which an image processing method or an image processing device according to an embodiment of the present disclosure can be applied;

FIG2 is a schematic diagram showing the structure of an electronic device suitable for implementing an embodiment of the present disclosure;

FIG3 schematically shows a flow chart of an image processing method according to an exemplary embodiment of the present disclosure;

FIG4 is a schematic diagram showing a process of determining a matching block in an embodiment of the present disclosure;

FIG5 is a schematic diagram showing a method of determining a matching block for a current image block in an embodiment of the present disclosure;

FIG6 is a schematic diagram showing a process of determining a target direction in an embodiment of the present disclosure;

FIG7 is a schematic diagram showing a process flow of performing directional interpolation in an embodiment of the present disclosure;

FIG. 8 schematically shows a block diagram of an image processing apparatus in an exemplary embodiment of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices.

The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation. In addition, all the terms "first" and "second" below are only for the purpose of distinction and should not be used as limitations of the present disclosure.

In order to solve the technical problems existing in the related art, an image processing method is provided in an embodiment of the present disclosure. Fig. 1 is a schematic diagram showing an exemplary system architecture to which an image processing method or an image processing device in an embodiment of the present disclosure can be applied.

As shown in FIG1 , the system architecture 100 may include a first end 101, a network 102, and a second end 103. The first end 101 may be a client, for example, various handheld devices (smart phones), tablet computers, desktop computers, vehicle-mounted devices, wearable devices, etc. that can be used to collect images and display images (play images). The network 102 is used to provide a medium for a communication link between the first end 101 and the second end 103. The network 102 may include various connection types, such as a wired communication link, a wireless communication link, etc. In the embodiment of the present disclosure, the network 102 between the first end 101 and the second end 103 may be a wired communication link, for example, a communication link may be provided through a serial port connection line, or a wireless communication link, and a communication link may be provided through a wireless network. The second end 103 may be a client, such as a portable computer, a desktop computer, a smart phone, and other terminal devices with image processing functions, for image processing. When the first end and the second end are both terminals, they may be the same terminal. The second end may also be a server, such as a local server or a cloud server, etc., which is not limited here.

In the embodiment of the present disclosure, first, the first end 101 can capture an image or use the image to be processed as an input image, and use the image to be displayed as an output image. Next, the second end 103 can use a certain pixel to be interpolated in the output image as the current pixel to be interpolated, and determine the position of its adjacent pixels in the input image. Further, the input image can be divided into blocks based on the positions of adjacent pixels and an image block can be determined as the current image block, and all matching blocks can be determined by motion estimation in multiple reference frames corresponding to the current frame where the current image block is located. Thirdly, the second end can fuse part or all of all matching blocks with each other, and fuse part or all of all matching blocks with the current image block to obtain multiple fused blocks, so as to determine the target direction of the current image block according to the multiple fused blocks. Finally, the current image block is directional interpolated according to the target direction to obtain the pixel value of the current pixel to be interpolated, so as to complete the entire interpolation operation and obtain the final image. The second end can also output the final image to the first end for display or playback.

It should be understood that the number of first terminals, networks and second terminals in Figure 1 is only for illustration purposes. Any number of clients, networks and servers may be provided as required.

It should be noted that the image processing method provided in the embodiment of the present disclosure can be completely executed by the second end or by the first end, and is not specifically limited here. Accordingly, the image processing device can be arranged in the second end 103 .

Fig. 2 is a schematic diagram of an electronic device suitable for implementing the exemplary embodiment of the present disclosure. It should be noted that the electronic device shown in Fig. 2 is only an example and should not bring any limitation to the function and scope of use of the embodiment of the present disclosure.

The electronic device of the present disclosure includes at least a processor and a memory, wherein the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the image processing method of the exemplary embodiment of the present disclosure.

Specifically, as shown in FIG2 , the electronic device 200 may include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, a sensor module 280, a display screen 290, an indicator 292, a motor 293, a button 294, and a Subscriber Identification Module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and/or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. In addition, a memory may be provided in the processor 210 for storing instructions and data.

The USB interface 230 is an interface that complies with the USB standard specification, and specifically can be a MiniUSB interface, a MicroUSB interface, a USB Type C interface, etc. The USB interface 230 can be used to connect a charger to charge the electronic device 200, and can also be used to transfer data between the electronic device 200 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

The charging management module 240 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and/or the charging management module 240 to power the processor 210, the internal memory 221, the display screen 290 and the wireless communication module 260.

The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, the baseband processor, and the like.

The mobile communication module 250 can provide wireless communication solutions including 2G/3G/4G/5G etc. applied on the electronic device 200 .

The wireless communication module 260 can provide wireless communication solutions for application in the electronic device 200, including Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) network), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), infrared technology (IR), etc.

The electronic device 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 can include a program storage area and a data storage area. The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.

The button 294 includes a power button, a volume button, etc. The button 294 can be a mechanical button. It can also be a touch button. The motor 293 can generate a vibration prompt. The motor 293 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 292 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect the SIM card. The electronic device 200 interacts with the network through the SIM card to realize functions such as calls and data communications.

The present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present disclosure, computer-readable storage media may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not constitute limitations on the units themselves in some cases.

FIG3 schematically shows a flowchart of an image processing method of an exemplary embodiment of the present disclosure, which can be applied, for example, in the process of capturing images, transmitting images, or processing images, or can also be applied to super-resolution reconstruction scenarios. The method in the embodiment of the present disclosure is applied to a super-resolution reconstruction scenario as an example for illustration. In a super-resolution reconstruction scenario, texture detail information lost in an image or video during imaging, transmission, etc. can be restored. Referring to FIG3 , with a terminal as the execution subject, the image processing method can include steps S310 to S330, which are described in detail as follows:

In step S310, positions of neighboring pixels of a current pixel to be interpolated contained in the output image in the input image are determined, a current image block is determined in the input image according to the positions of the neighboring pixels, and all matching blocks corresponding to the current image block are determined in multiple reference frames corresponding to the current frame.

In the disclosed embodiments, the input image may be an image captured by a terminal, a frame of an image in a video, or may be obtained from other devices, such as an image downloaded from the Internet or a frame of an image in a video. The output image may be an image with the same content as the input image but a different resolution, that is, the resolution of the input image may be scaled to obtain the output image. The resolution of the output image may be greater than or less than the input image, which may be determined specifically according to the type of usage scenario.

After determining the input image and the output image, the scaling degree of the output image relative to the input image can be determined. The scaling degree here may include the horizontal scaling degree and the vertical scaling degree, and the scaling degree can be specifically represented by a scaling multiple. The output image is obtained by interpolating the input image, so there may be multiple pixels to be interpolated in the output image. The number of pixels to be interpolated in the output image can be determined according to the resolution of the output image. Specifically, the current pixel to be interpolated refers to any one of all the pixels to be interpolated in the output image, and its position in the output image can be represented by (i, j). The adjacent pixel refers to an adjacent pixel that is adjacent to the current pixel to be interpolated and is located in a plurality of neighborhood pixels in the input image. The neighborhood pixels here can be four adjacent pixels represented by D neighborhood pixels, and the adjacent pixel refers to the pixel located at the upper left corner of the current pixel to be interpolated, and its position can be represented by (x, y).

On this basis, the step of determining the position of the adjacent pixels of the current pixel to be interpolated includes: first, according to the ratio of the horizontal coordinate of the current pixel to be interpolated to the horizontal scaling degree, determining the horizontal coordinate of the adjacent pixels. The horizontal coordinate of the current pixel to be interpolated refers to its coordinate in the output image. The horizontal scaling factor may be a horizontal magnification factor, specifically represented by ratio _w . The horizontal coordinate of the adjacent pixels may be a ratio of the horizontal coordinate of the current pixel to be interpolated to the horizontal magnification factor. Further, the vertical coordinate of the adjacent pixels may be determined according to the ratio of the vertical coordinate of the current pixel to be interpolated to the vertical scaling degree. The vertical coordinate of the current pixel to be interpolated refers to its coordinate in the output image. The vertical scaling factor may be a vertical magnification factor, specifically represented by ratio _h . The vertical coordinate may be a ratio of the vertical coordinate of the current pixel to be interpolated to the vertical magnification factor. The coordinates of the current pixel to be interpolated and its adjacent pixels can be accurately determined by the scaling degree of the resolution. The horizontal coordinate and the vertical coordinate may be determined simultaneously or in sequence, which is not specifically limited here. After obtaining the horizontal coordinate and the vertical coordinate, the position of the adjacent pixels in the input image can be obtained based on the two, which can be specifically expressed by formula (1):

After obtaining the adjacent pixels located in the input image, the current image block is determined in the input image by the positions of the adjacent pixels, and all matching blocks corresponding to the current image block are determined in multiple reference frames corresponding to the current frame. In the disclosed embodiment, after determining the positions of the adjacent pixels in the input image, for the current pixel to be interpolated, the input image can be divided into blocks according to the positions of the adjacent pixels and the preset image block size in the input image to determine the image block. Specifically, the position of the adjacent pixels can be used as the center, and the image block corresponding to the current pixel to be interpolated can be determined in the input image according to the preset image block size. The preset image block size can be 4×4 or other suitable sizes. The size of the image block can be set according to actual needs and is not specifically limited here. The number of image blocks can be multiple, and the size of the image block is negatively correlated with the number of image blocks.

After determining the image block, any one of the image blocks being processed can be used as the current image block, and then all matching blocks corresponding to the current image block can be determined in multiple reference frames. The multiple reference frames can be reference frames corresponding to the current frame where the current image block is located, such as multiple frames adjacent to the current frame. Any one of the multiple reference frames can be used as the current reference frame. First, the processing process in a current reference frame is used as an example for explanation. Specifically, a multi-frame motion estimation method can be adopted to determine the motion vector of the current image block of the current frame relative to the current reference frame in the current reference frame, and determine the matching block similar to the current image block in the current reference frame based on the motion vector. The basic idea of motion estimation is to divide each frame of the image sequence into many non-overlapping macroblocks, and assume that the displacement of all pixels in the macroblock is the same, and then find the block most similar to the current image block within a given specific search range of each macroblock to the reference frame according to a certain matching criterion, that is, the matching block, and the relative displacement between the matching block and the current image block is the motion vector. When compressing the video, only the motion vector and residual data need to be saved to completely restore the current image block. In the disclosed embodiment, multi-frame motion estimation is used to search for matching blocks similar to the current image block from other multiple frames to increase the comprehensiveness and accuracy of the search, avoid directional errors caused by interpolation based on only a single frame, and improve the accuracy of the interpolation direction through multi-frame motion estimation.

FIG4 schematically shows a flow chart of determining a matching block in a current reference frame. Referring to FIG4 , the flow chart mainly includes the following steps:

In step S410, the searchable position of the current reference frame is determined according to a search window with the position of the current image block as the center.

In this step, the search window can be used to specify the area range for executing the search process. In the current reference frame, the search window of each image block may be inconsistent, and the size of the search window can be set according to the search window rule. The search window rule may include the gradient size or the number of traversed image blocks. For example, the gradient of the current image block can be used as a basis so that the gradient is positively correlated with the search window, that is, the larger the gradient, the larger the search window; otherwise, the smaller it is. The window size of the search window can also gradually increase or decrease as the number of traversed image blocks increases.

The searchable position refers to the searchable range of the current reference frame, that is, the area range where matching blocks may exist, which can be determined based on the size of the search window of each image block. The searchable position of each reference frame may be the same or different.

In step S420, the searchable positions are traversed with a fixed step size to obtain all reference blocks having the same size as the current image block in the current reference frame.

In this step, the searchable positions may be traversed and matched with a fixed step size (eg, several pixels) to obtain all reference blocks having the same size as the current image block from the current reference frame.

In step S430, all reference blocks are matched with the current image block to obtain a matching degree, and the matching block is determined from all reference blocks of the current reference frame according to the matching degree.

In this step, all reference blocks can be matched with the current image block to obtain the degree of match between the two. Specifically, feature extraction can be performed on the reference block to obtain reference features, and feature extraction can be performed on the current image block to obtain current features, and then the degree of match between the two can be calculated based on the extracted reference features and current features. The degree of match can be expressed in one or more ways including but not limited to SAD (Sum of Absolute Differences, absolute error and algorithm), Euclidean distance, texture gradient, etc. For the absolute error and algorithm, the smaller the mean absolute difference, the more similar it is.

After the matching degree is calculated, the reference block with the highest matching degree can be used as at least one matching block of the current image block in the current reference frame. In addition, the number of matching blocks in each reference frame can be the same or different, which is specifically determined according to the value of the matching degree. Selecting at least one matching block based on the matching degree can improve the accuracy of the matching block.

When the matching blocks are obtained, the credibility of each matching block can be recorded. The credibility here is used to indicate the matching degree of the matching blocks for interpolation operations. The credibility can be expressed by SAD (Sum of Absolute Differences), which is used to determine the weight in the subsequent interpolation process, thereby affecting the interpolation effect.

It should be noted that the method of determining matching blocks for all image blocks in all reference frames is the same as the step in FIG. 4 , and thus will not be described in detail here.

FIG5 schematically shows a schematic diagram of using motion estimation of multiple frames to determine a matching block similar to the current image block. Specifically, the previous frame F _CUR-1 adjacent to the current frame F _CUR can be used as a reference frame and the next frame F _CUR+1 adjacent to the current frame F _CUR can be used as a reference frame, or the first two frames F _CUR-2 adjacent to the current frame F CUR can be used as reference frames and the next two frames F _CUR+2 adjacent to the current frame F CUR can be used as reference frames, until the first M frames F _CUR-M and the next N frames F _CUR+N are used as reference frames, so as to determine the matching block most similar to the current image block according to the motion vector between the two. Determining the matching block most similar to the current image block in the input image by using motion estimation of multiple frames can improve the accuracy of the matching block.

Continuing to refer to FIG. 3 , in step S320 , image blocks are fused according to the target matching blocks in all matching blocks and the current image block to obtain multiple fused blocks, and the target direction of the current image block is determined according to the multiple fused blocks.

In the embodiment of the present disclosure, when performing image block fusion, the target matching blocks corresponding to all matching blocks of the current image block can be selected for fusion. The target matching blocks can be all matching blocks or partial matching blocks, which can be specifically determined according to the number of matching blocks. For example, when the number of matching blocks of the current image block is greater than the number threshold, the partial matching blocks can be selected as the target matching blocks; when the number of matching blocks of the current image block is not greater than the number threshold, all matching blocks can be selected as the target matching blocks. Among them, the partial matching blocks can be K matching blocks with fixed frame positions of all matching blocks. The fixed frames can be the frames that are most adjacent to the current frame.

On this basis, K matching blocks at fixed frame positions can be matched with one current image block to perform fusion between the matching blocks and the current image block and between the matching blocks, and t fusion blocks are obtained by fusion. The number of matching blocks required for each fusion block may be different. Specifically, the attribute parameters of each fusion block can be determined according to the ratio of the sum of the product of the weight of each matching block and the attribute parameter of each matching block to the sum of the weights of all matching blocks. The attribute parameters here can be the pixel values of each fusion block at the pixel level. Each matching block can be 4×4. Based on this, K 4×4 matching blocks can be fused with one current image block respectively, and the pixel values at the same position in the matching block are weighted averaged according to the weight corresponding to the matching block to obtain a weighted average value, and the weighted average value is used as the pixel value at the same position in the fusion block to determine the attribute parameters of each fusion block. The specific method of fusing the matching block and the current image block to obtain the fusion block can be shown in formula (2):

BLK＝(w ₀ *BLK ₀ +w ₁ *BLK ₁ + ... +w _k *BLK _k )/(w ₀ +w ₁ + ... +w _k ) Formula (2)

Among them, w ₀ , w ₁ , and w _k are weights of corresponding matching blocks respectively, and the weight of each matching block can be one or more of time distance, matching credibility, and manual setting, etc. For different matching blocks, the corresponding matching block weights can be different, that is, the type of weight can be determined according to the type of matching block.

In the disclosed embodiment, the weight of the matching block is taken as the time distance for explanation. Specifically, the weight of each matching block can be determined according to the time distance; then the two target matching blocks are fused according to the weight of each matching block, and the target matching block is fused with the current image block according to the weight of each matching block to obtain multiple fused blocks. For example, if there are 12 matching blocks, the target matching blocks of the two frames with the closest time distance can be formed into a fused block, thereby obtaining 6 fused blocks. By forming the matching blocks with the closest time distance into fused blocks, the accuracy of the fused blocks can be improved.

Furthermore, after obtaining multiple fusion blocks, the target direction of the current image block can be determined according to the multiple fusion blocks. The target direction refers to the texture direction of the current image block. Direction fusion can be performed according to the multiple fusion blocks to obtain the target direction of the current image block.

FIG6 schematically shows a flow chart of determining the target direction. Referring to FIG6 , the flow chart mainly includes the following steps:

In step S610, the gradient of each fused block in multiple directions is calculated, and the texture direction of each fused block is determined according to the direction with the smallest gradient;

In step S620, the texture directions are fused according to the weight of each fusion block to obtain the target direction.

In the disclosed embodiment, the multiple directions may be any directions, for example, any directions including but not limited to horizontal direction, vertical direction, 45 degree direction and 135 degree direction. Specifically, the gradient of each fusion block in each of the multiple directions may be calculated. Furthermore, the gradients of the multiple directions of each fusion block may be compared to determine the minimum gradient. Further, the direction with the minimum gradient may be determined as the texture direction corresponding to the fusion block. The texture directions of each fusion block may be the same or different, and are specifically determined according to the value of the gradient. The range of the texture direction of each fusion block may be any value between 0 and 359 degrees. For example, the texture direction of fusion block 1 is the horizontal direction, and the texture direction of fusion block 2 is the vertical direction.

After obtaining the texture direction of each fusion block, the texture directions of all fusion blocks can be fused based on the weight corresponding to each fusion block. The weight of each fusion block is used to indicate its importance or proportion. The weight of each fusion block can be the same or different, and can be determined according to the number of original blocks used to form the fusion block. Among them, the original blocks are used to form the fusion block, and the original blocks include matching blocks and current image blocks. The weight of the fusion block is positively correlated with the number of original blocks that constitute the fusion block, that is, the more the number of original blocks, the greater the value of the weight of the fusion block.

After obtaining the weight of the fusion block, the texture direction of each fusion block can be weighted averaged according to the weight of each fusion block to obtain the target direction. Specifically, the target direction can be determined according to the ratio of the sum of the product of the weight of each fusion block and the texture direction of each fusion block to the sum of the weights of all fusion blocks. The specific calculation method can be shown in formula (3):

dir＝(dir ₀ *u ₀ +dir ₁ *u ₁ + ... +dir _t *u _t )/(u ₀ +u ₁ + ... +u _t ) Formula (3)

Among them, dir _t is the texture direction of the t-th fusion block, and _ut is the weight of the t-th fusion block.

In the disclosed embodiment, by fusing the texture direction of each fusion block through the weight of the fusion block, an accurate target direction can be obtained, thereby avoiding errors that may be caused when the direction is determined by only one frame of image, and improving accuracy.

In step S330, directional interpolation is performed on the current image block according to the target direction to obtain a pixel value of the current pixel to be interpolated.

In the disclosed embodiment, directional interpolation refers to a method of interpolating in a fixed direction. After determining the target direction, directional interpolation can be performed based on the reference pixels corresponding to the target direction to calculate the pixel value of the current pixel to be interpolated. The reference pixel refers to all pixels that already exist in the target direction. Directional interpolation can be mean filtering or other methods. Based on this, the pixel values of the pixels existing in the target direction can be mean filtered, and the pixel value of the current pixel to be interpolated is calculated based on the result of the mean filtering. Mean filtering refers to giving a template to the target pixel on the image, which template includes the neighboring pixels around it (8 pixels around the target pixel as the center, forming a filtering template, that is, removing the target pixel itself), and then replacing the original pixel value with the average value of all pixels in the template. Based on this, the pixel value of the current pixel to be interpolated can be determined based on the mean value of the pixel values of the reference pixels in the target direction.

For all pixels to be interpolated included in the output image, the method in step S310 to step S330 can be used to calculate the pixel value of each pixel to be interpolated until all pixels to be interpolated in the output image obtain corresponding pixel values. After the pixel values of the pixels to be interpolated are determined, the output image can be obtained according to the pixel values of these pixels to be interpolated, so as to realize image super-resolution reconstruction processing through this method, thereby improving the super-resolution effect and image quality.

FIG7 schematically shows a flow chart of performing directional interpolation. Referring to FIG7 , the flow chart mainly includes the following steps:

In step S710, the scaling degree of the output image relative to the input image is calculated.

In step S720, it is determined whether the interpolation of the current output image is completed; if so, the process ends; if not, the process proceeds to step S730.

In step S730, the most adjacent position of the pixel to be interpolated in the input image is determined, which may be the adjacent pixel in the upper left corner.

In step S740 , the size of the image block of the input image is determined.

In step S750, a matching block matching all image blocks is searched in the reference frame.

In step S760, the matching blocks are fused to obtain a fused block.

In step S770 , the texture direction of the fused block is determined.

In step S780, the texture directions of the fusion blocks are fused.

In step S790, directional interpolation is performed, and the process returns to step S720.

The technical solution in FIG7 utilizes the changing motion characteristics of the foreground and background between adjacent frames, and adopts the motion estimation method to find corresponding multiple matching blocks for the current image block in multiple reference frames, and finally integrates all matching blocks to obtain a fused block, and then performs directional judgment on the fused block to determine the texture direction of each fused block; after obtaining the texture direction of the pixel position to be interpolated of the current image block, a directional interpolation filter is used for interpolation to obtain the current pixel to be interpolated. It avoids the problem in the related art that the neural network algorithm is heavily dependent on the data set during training, has poor actual generalization ability, and is too complex, reduces complexity, and increases the scope of application. It also avoids the misjudgment of texture due to the input image being a single frame, avoids limitations, and leads to subjective problems of the image caused by errors in directional interpolation, thereby improving image quality.

It should be added that the super-resolution scheme in the embodiment of the present disclosure can also be used in the denoising process. For example, in the video path, multiple intermediate resolutions can be set and multiple interpolations can be used to obtain the final output resolution image, that is, each input is the output of the previous one, so as to achieve a better denoising effect. The super-resolution scheme can also be used in the image restoration process. For example, for the details of some single-frame loss in the video, other frames can be used to fill in, correct, and other operations.

It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and/or one step may be decomposed into multiple steps, etc.

FIG8 schematically shows a block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure. Referring to FIG8 , an image processing apparatus 800 according to an exemplary embodiment of the present disclosure may include the following modules:

A matching block determination module 801 is used to determine positions of neighboring pixels of a current pixel to be interpolated contained in an output image in an input image, determine a current image block in the input image according to the positions of the neighboring pixels, and determine all matching blocks corresponding to the current image block in a plurality of reference frames corresponding to the current frame;

A direction determination module 802 is used to perform image block fusion according to the target matching block in all matching blocks and the current image block to obtain multiple fused blocks, and determine the target direction of the current image block according to the multiple fused blocks;

The image interpolation module 803 is used to perform directional interpolation on the current image block according to the target direction to obtain the pixel value of the current pixel to be interpolated.

In an exemplary embodiment of the present disclosure, the matching block determination module includes: a first coordinate determination module, used to determine the horizontal coordinate of the adjacent pixel according to the ratio of the horizontal coordinate of the current pixel to be interpolated to the horizontal scaling degree; and a second coordinate determination module, used to determine the vertical coordinate of the adjacent pixel according to the ratio of the vertical coordinate of the current pixel to be interpolated to the vertical scaling degree.

In an exemplary embodiment of the present disclosure, the matching block determination module includes: a motion estimation matching module, which is used to determine, in the current reference frame, a motion vector of a current image block of the current frame relative to the current reference frame, and determine, in the current reference frame based on the motion vector, a matching block similar to the current image block.

In an exemplary embodiment of the present disclosure, the motion estimation matching module includes: a position determination module, which is used to determine the searchable position of the current reference frame according to the search window with the position of the current image block as the center; a reference block determination module, which is used to traverse the searchable position with a fixed step size to obtain all reference blocks with the same size as the current image block in the current reference frame; and a matching block selection module, which is used to match all reference blocks with the current image block to obtain a matching degree, and determine the matching block from all reference blocks of the current reference frame according to the matching degree.

In an exemplary embodiment of the present disclosure, the matching block selection module is configured to: select at least one reference block with the highest matching degree among all reference blocks as the matching block of the current image block in the current reference frame.

In an exemplary embodiment of the present disclosure, the direction determination module includes: a weight determination module, which is used to determine the weight of each matching block according to the time distance; a matching block fusion module, which is used to fuse the target matching blocks in all the matching blocks according to the weight of each matching block, and fuse the target matching block with the current image block according to the weight of each matching block to obtain multiple fused blocks.

In an exemplary embodiment of the present disclosure, the direction determination module includes: a texture direction determination module, which is used to calculate the gradient of each fusion block in multiple directions, and determine the texture direction of each fusion block according to the direction with the smallest gradient; a direction fusion module, which is used to fuse the texture directions according to the weight of each fusion block to obtain the target direction.

In an exemplary embodiment of the present disclosure, the direction fusion module includes: a target direction determination module, configured to perform weighted averaging processing on the texture direction of each fusion block according to the weight of each fusion block to obtain the target direction.

In an exemplary embodiment of the present disclosure, the image interpolation module is configured to: perform the directional interpolation on the current image block according to the pixel value of the reference pixel corresponding to the target direction to determine the pixel value of the current pixel to be interpolated.

Since the functional modules of the image processing device in the embodiment of the present disclosure are the same as those in the embodiment of the above-mentioned image processing method, they will not be described in detail here.

Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (12)

1. An image processing method, comprising:

Determining the position of a neighboring pixel of a current pixel to be interpolated in an input image, which is contained in an output image, determining a current image block in the input image through the position of the neighboring pixel, and determining all matching blocks corresponding to the current image block in a plurality of reference frames corresponding to the current frame based on a motion estimation mode;

Image block fusion is carried out according to target matching blocks in all the matching blocks and the current image block to obtain a plurality of fusion blocks, and the target direction of the current image block is determined according to the fusion blocks;

and performing directional interpolation on the current image block through the target direction to obtain the pixel value of the current pixel to be interpolated.

2. The image processing method according to claim 1, wherein determining the position of the adjacent pixel of the current pixel to be interpolated included in the output image in the input image includes:

Determining the horizontal coordinates of the adjacent pixels according to the ratio of the horizontal coordinates of the current pixel to be interpolated to the horizontal scaling degree;

And determining the vertical coordinates of the adjacent pixels according to the ratio of the vertical coordinates of the current pixel to be interpolated to the vertical scaling degree.

3. The image processing method according to claim 1, wherein said determining all matching blocks corresponding to the current image block among a plurality of reference frames corresponding to a current frame includes:

A motion vector of a current image block of the current frame relative to the current reference frame is determined in the current reference frame, and a matching block similar to the current image block is determined in the current reference frame based on the motion vector.

4. The image processing method according to claim 3, wherein said determining a motion vector of a current image block of the current frame with respect to the current reference frame in the current reference frame, and determining a matching block similar to the current image block in the current reference frame based on the motion vector, comprises:

the position of the current image block is taken as the center, and the searchable position of the current reference frame is determined according to a search window;

Traversing the searchable position by a fixed step length, and obtaining all reference blocks with the same size as the current image block in the current reference frame;

and matching all the reference blocks with the current image block to obtain a matching degree, and determining the matching block from all the reference blocks of the current reference frame according to the matching degree.

5. The image processing method according to claim 4, wherein said determining the matching block from all reference blocks of the current reference frame according to the matching degree comprises:

And taking at least one reference block with the highest matching degree in all the reference blocks as the matching block of the current image block in the current reference frame.

6. The image processing method according to claim 1, wherein the performing image block fusion according to the target matching block and the current image block in all the matching blocks to obtain a plurality of fusion blocks includes:

determining the weight of each matching block according to the time distance;

fusing target matching blocks in all the matching blocks according to the weight of each matching block, and fusing the target matching block with the current image block according to the weight of each matching block to obtain a plurality of fusion blocks.

7. The image processing method according to claim 1, wherein the determining the target direction of the current image block from the plurality of the fusion blocks includes:

Calculating gradients of each fusion block in multiple directions, and determining the texture direction of each fusion block according to the direction with the smallest gradient;

and fusing the texture directions according to the weight of each fusion block to obtain the target direction.

8. The image processing method according to claim 7, wherein the fusing the texture directions according to the weight of each of the fused blocks to obtain the target direction includes:

And carrying out weighted average processing on the texture direction of each fusion block according to the weight of each fusion block to obtain the target direction.

9. The image processing method according to claim 1, wherein the performing directional interpolation on the current image block by the target direction to obtain a pixel value of the current pixel to be interpolated includes:

And carrying out directional interpolation on the current image block according to the pixel value of the reference pixel corresponding to the target direction so as to determine the pixel value of the current pixel to be interpolated.

10. An image processing apparatus, comprising:

a matching block determining module, configured to determine a position of a neighboring pixel of a current pixel to be interpolated included in an output image in an input image, determine a current image block in the input image by using the position of the neighboring pixel, and determine all matching blocks corresponding to the current image block in a plurality of reference frames corresponding to the current frame based on a motion estimation manner;

the direction determining module is used for carrying out image block fusion according to target matching blocks in all the matching blocks and the current image block to obtain a plurality of fusion blocks, and determining the target direction of the current image block according to the fusion blocks;

and the image interpolation module is used for carrying out directional interpolation on the current image block through the target direction to obtain the pixel value of the current pixel to be interpolated.

11. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the image processing method according to any one of claims 1-9.

12. An electronic device, comprising:

A processor; and

A memory for storing executable instructions of the processor;

wherein the processor is configured to perform the image processing method of any of claims 1-9 via execution of the executable instructions.