KR101007381B1 - Image Coding Device Considering Region of Interest - Google Patents

Abstract

An image encoding apparatus considering a region of interest is disclosed. The apparatus for encoding an image comprises: a detector for detecting at least one region of interest, including one macroblock or a plurality of adjacent macroblocks, and a remaining region from an image frame; A determination for determining a quantization step size of each of the macroblocks included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one region of interest is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. part; And a quantizer for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size. Therefore, the compression ratio can be improved without deteriorating the visual quality.

Description

Apparatus for video encoding considering region of interest}

The present invention relates to a video encoding apparatus and method.

According to multimedia communication, various kinds of data are transmitted and received, among which video data requires much higher data rates than any kind of data. In response to these demands, various image compression algorithms have been proposed. As a result, H.264 / AVC, H.264 / SVC, etc., which are recent image compression algorithms, have compared to the previous image compression standards. In terms of performance.

Despite the development of such image compression technology, as an image becomes increasingly high quality, there is a continuous demand for an image encoding algorithm capable of improving the compression ratio while maintaining visual quality.

In response to these demands, efforts have been made to improve the compression rate or the image quality by using the human visual characteristics. However, since the existing technologies involve additional redundancy, further compression rate improvement is possible.

SUMMARY OF THE INVENTION The present invention has been made in an effort to provide an image encoding apparatus, an image encoding method, and a computer-readable recording medium capable of improving a compression ratio without degrading visual quality.

In order to achieve the above technical problem, a first aspect of the present invention provides a method for detecting at least one Region of Interest (ROI) including one macroblock or a plurality of adjacent macroblocks from an image frame and a remaining region. A detection unit; A determination for determining a quantization step size of each of the macroblocks included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one region of interest is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. part; And a quantization unit configured to quantize transform coefficients of each of the macroblocks according to the determined quantization step size.

Preferably, the at least one ROI is at least one object having a motion size greater than or equal to a predetermined threshold among at least one object extracted from the image frame.

Preferably, the detection unit extracts the at least one object from the image frame by using a depth map obtained from a stereo image. Preferably, the detector calculates an average magnitude value of corresponding motion vectors for each of the at least one extracted object, and the at least one ROI is at least one object having an average magnitude value of more than the predetermined threshold. .

Preferably, the detection unit, from the image frame, extracts macro blocks belonging to a boundary block and having a motion size value equal to or greater than a predetermined threshold, and grouping the extracted macro blocks to group similar macro blocks in terms of motion size and direction. Generate at least one group including the at least one group, and generate a convex hull including group members for each generated group, wherein the at least one region of interest is an inner region of the generated at least one convex hull. . Preferably, the detector determines macroblocks for which motion compensation is performed in subblock units as macroblocks belonging to the boundary block. Preferably, the region of interest includes a center region of the image frame, including one macro block or a predetermined number of adjacent macro blocks.

Preferably, the determination unit may include: a first importance value calculator for calculating, for each macroblock, a first import value that is inversely proportional to a minimum distance from the at least one region of interest; And a rate controller for determining the quantization step size of the macroblock based on the calculated first importance value.

Preferably, the determination unit may include: a first importance value calculator for calculating, for each macroblock, a first import value that is inversely proportional to a minimum distance from the at least one region of interest; A second significant value calculator for calculating a weighted sum of transform coefficients for each macro block and determining the calculated weighted sum as a second significant value of the corresponding macro block; And a rate controller for determining a quantization step size of the macroblock based on the calculated first and second significance values, wherein each of the weights used for calculating the weighted sum is inversely proportional to the spatial frequency of the corresponding transform coefficient. Has a value.

Preferably, the determination unit may include: a first importance value calculator for calculating, for each macroblock, a first import value that is inversely proportional to a minimum distance from the at least one region of interest; A third importance value calculator configured to calculate a third importance value for the at least one region of interest and the remaining region; And a rate controller configured to determine a quantization step size of the macroblock based on the calculated first and third important values, wherein the third important value is a motion vector belonging to the corresponding region within a range up to a predetermined motion size. Is proportional to the average size of the motion vectors, and is inversely proportional to the average size of the motion vectors outside the range, and the predetermined motion size is proportional to the width of the corresponding region of interest with respect to the average direction of the motion vectors.

Preferably, the determination unit may include a first importance value calculator for calculating a first importance value for each macro block in inverse proportion to a minimum distance from the at least one region of interest; A second significant value calculator for calculating a weighted sum of transform coefficients for each macro block and determining the calculated weighted sum as a second significant value of the corresponding macro block; A third importance value calculator configured to calculate a third importance value for the at least one region of interest and the remaining region; And a rate control unit for determining a quantization step size of the macroblock based on the calculated first important value, the second important value, and the third important value, wherein each of the weights used for calculating the weighted sum is a value of the corresponding transform coefficient. Having a value inversely proportional to the spatial frequency, the third importance value is proportional to the average magnitude of motion vectors belonging to the region within a range up to a predetermined magnitude of motion, and is inversely proportional to the average magnitude of the motion vectors outside the range, The predetermined motion magnitude is proportional to the width of the region of interest in the average direction of the motion vectors.

Preferably, the determining unit comprises: a third importance value calculator for calculating a third importance value for the at least one region of interest and the remaining region; And a rate controller configured to determine a quantization step size of the macroblock based on the calculated third importance value, wherein the third importance value is determined by an average size of motion vectors belonging to the region within a range up to a predetermined motion size. Outside the range, inversely proportional to the average size of the motion vectors, and the predetermined motion size is proportional to the width of the region of interest in the average direction of the motion vectors.

Preferably, the determination unit may include a second significant value calculator for calculating a weighted sum of transform coefficients for each macro block and determining the calculated weighted sum as a second significant value of the corresponding macro block; A third importance value calculator configured to calculate a third importance value for the at least one region of interest and the remaining region; And a rate controller for determining a quantization step size of the macroblock based on the calculated second significant value and the third significant value, wherein each of the weights used for calculating the weighted sum is inversely proportional to the spatial frequency of the corresponding transform coefficient. The third importance value is proportional to an average size of motion vectors belonging to a corresponding region within a range up to a predetermined motion size, and inversely proportional to the average size of the motion vectors outside the range, The size is proportional to the width of the region of interest in the average direction of the motion vectors.

Preferably, the first important value calculation unit,

Where ν is the viewing distance, k is the index of the macroblock, d ^k is the minimum distance between macroblock k and the at least one region of interest, N is the image width, e ₂ , α, and CT ₀ are preset values ), The first important value of the macroblock is calculated. Preferably, the first important value calculator calculates a maximum value obtained by the equation for the image frame, normalizes the result of the equation for the macroblock to the maximum value, and normalizes the normalized result. Determined as the first significant value of the macro block.

Preferably, the second significant value calculation unit,

Where k and N represent the macroblock index and the macroblock size, respectively, x and y represent the transform coefficient index and a (k) _{x and y} represent the transform corresponding to the macro block k x and y index. Coefficient, f _r (x, y) is the spatial frequency of the transform coefficient corresponding to the x, y index, and A, B, D, E, F represent a predetermined positive real number) Calculate the important value. Preferably, the second significant value calculating unit calculates a maximum value obtained by the equation for the image frame, normalizes the result of the equation for the macroblock to the maximum value, and normalizes the normalized result. The second significant value of the macro block is determined.

Preferably, the third important value calculation unit,

(Where k represents the index of the macroblock, υ ^k , and m ^k represent the average angular velocity and width of the region of interest or the remaining region to which macroblock k belongs, and B ₁ , B ₂ , B ₃ , B ₄ , B ₅ , B ₆ , B ₇ , B ₈ , and B ₉ represent a preset amount of real number) to calculate the third significant value of the macro block. Preferably, the third significant value calculating unit calculates a maximum value obtained by the equation for the image frame, normalizes the result of the equation for the macroblock to the maximum value, and normalizes the normalized result. The third significant value of the macro block is determined.

Preferably, the rate controller calculates a weighted sum of the first significant value, the second significant value, and the third important value for each macro block, and determines the quantization step size of the macro block based on the calculated weighted sum. .

Preferably, the rate controller calculates a value proportional to the product of the first important value, the second important value, and the third important value for each macro block, and determines the quantization step size of the macro block based on the calculated value. do.

According to a second aspect of the present invention, a weighted sum of transform coefficients is calculated for each macroblock included in an image frame, and the quantization step size of the macroblock is determined based on the calculated weighted sum. Determination unit for determining; And a quantization unit for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size, wherein each of the weights used for calculating the weighted sum has a value inversely proportional to the spatial frequency of the corresponding transform coefficient. to provide.

Embodiments of the present invention presented above may have an effect including the following advantages. However, all the embodiments of the present invention are not meant to include them all, and thus the scope of the present invention should not be understood as being limited thereto.

According to an embodiment of the present invention, the compression ratio can be improved without deteriorating visual quality, and thus can be effectively used even in a channel shortage or poor channel environment in a network.

In addition, according to an embodiment of the present invention, it can be compatible with the existing image compression standard, there is an advantage in terms of market share.

In addition, according to an embodiment of the present invention, even if the image encoding using the human visual characteristics does not have additional redundancy.

In addition, according to an embodiment of the present invention, in consideration of various factors affecting the human vision together, it is possible to remove a signal that is not visually important and to further improve compression ratio by allocating more bits to important information. have.

Since descriptions of embodiments of the present invention are merely illustrated for structural to functional descriptions of the present invention, the scope of the present invention should not be construed as limited by the embodiments described in the present invention. That is, the embodiments of the present invention may be variously modified and may have various forms, and thus, it should be understood to include equivalents that may realize the technical idea of the present invention.

On the other hand, the meaning of the terms described in the present invention will be understood as follows.

Terms such as "first" and "second" are intended to distinguish one component from another component, and the scope of the present invention should not be limited by these terms. For example, the first component may be named a second component, and similarly, the second component may also be named a first component.

The term “and / or” should be understood to include all combinations that can be presented from one or more related items. For example, "first item, second item, and / or third item" means "at least one or more of the first item, second item, and third item". A combination of all items that can be presented from two or more of the first, second and third items as well as the third item.

When a component is referred to as being "connected" to another component, it should be understood that there may be other components in between, although it may be directly connected to the other component. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that there are no other elements in between. On the other hand, other expressions describing the relationship between the components, such as "between" and "immediately between" or "neighboring to" and "directly neighboring to", should be interpreted as well.

Singular expressions described herein are to be understood to include plural expressions unless the context clearly indicates otherwise, and the terms "comprise" or "having" include elements, features, numbers, steps, operations, and elements described. It is to be understood that the present invention is intended to designate that there is a part or a combination thereof, and does not exclude in advance the possibility of the presence or addition of one or more other features or numbers, steps, actions, components, parts or combinations thereof. .

Each step described in the present invention may occur out of the stated order unless the context clearly dictates the specific order. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and shall be interpreted as having ideal or overly formal meanings unless expressly defined in this application. Can't be.

According to embodiments of the present invention, by using a visual model of psychophysics, additional compression efficiency may be obtained while maintaining image quality similar to that of an existing image codec. In this case, examples of the visual model include a model considering the foveation-based perceptual importance, a model considering the frequency-based perceptual importance, and a motion-based cognitive importance. A model that takes into account motion-based perceptual importance is mentioned.

Assuming that human vision is concentrated on a particular object included as part of an image frame, the spatial resolution of the visual for each image region or picture region is determined by The cognitive importance derived from the property of decreasing as far away is the perception-based cognitive importance. Here, detecting spatial importance based on the focus of a person is called foveation.

The optic nerve in the retina is not evenly distributed but concentrated in a section called fovea. When a person sees an image or the like, at least one point is focused, which corresponds to a povia. Among the points forming the image, a point corresponding to the povia is called a foveation point. Therefore, each of the image regions constituting the image has a high cognitive importance as the nearer to the fob point, and a lower cognitive importance as the farther.

1 is a view for explaining a fob point.

As shown in FIG. 1, an observer who sees an image with a face will generally focus on the face, so the focal points are distributed around the face. Thus, assuming that six points around the face are poking points, as shown in the left part of FIG. 1, a local bandwidth can be obtained as shown in the right part of FIG. 1. Here, the local bandwidth represents a frequency band that is human perceptible to the signal possessed by the image region according to the distance between the position of the image region and the focal point. For example, if the local bandwidth of the image region k is f _localBW (k), the main signal component that can be perceived by human vision is the signal component belonging to the frequency range [0, f _localBW (k)] among the signals of the image region. to be. Where k is the image area index.

This local bandwidth may be calculated by Equation 1, and may be used to calculate a first critical value, which will be described later.

Here, v represents a viewing distance, which is inversely proportional to the width of the displayed image, and means a relative distance proportional to the distance from the image to the human eye. In addition, d ^k is the distance from the positioning point to the signal position (e.g., the coordinates of the pixel, the position of the image area or the corresponding macroblock) k, N is the image width in pixels, the remainder is a constant and e ₂ = 2.3 , α = 0.106, CT ₀ = 1/64.

The cognitive importance derived from the characteristics that the human eye is relatively sensitive to low frequency components and insensitive to high frequency components is frequency-based perceptual importance. This frequency-based cognitive importance can be calculated using transform coefficients that contain information about frequency components for the picture region. Here, the transform coefficients are coefficients obtained by orthogonal transforming an image region, and examples of orthogonal transformations include various methods such as discrete cosine transform (hereinafter, referred to as DCT). In the present specification, for convenience, embodiments of the present invention will be described assuming a transform coefficient as a DCT coefficient.

Examples of parameters involved in frequency-based cognitive importance include spatial frequency, time frequency, average luminance, and the like. Equation 2 represents a contrast sensitivity function with respect to the spatial frequency f _r , which will be described later, but through this, a frequency-based cognitive importance, that is, a second importance value, may be calculated.

Here, as an example of a method of calculating the spatial frequency f _r , a method using Equation 3 may be used.

Here, x, y means the index (frequency index) of the DCT coefficients. In addition, p means display visual resolution, and an example of a method of calculating p may be a method using Equation 4.

Here, d means the resolution (pixels / cm) to be displayed, and ν means the viewing distance (cm) described above.

When a certain object is moved in an image having a stationary background, the cognitive importance derived from the characteristic that a person's interest is likely to be focused on the moving object is a motion-based cognitive importance. This motion-based cognitive importance can be used to calculate the third importance value described below.

However, on the contrary, as described later in FIG. 2, there is a study result in that the contrast sensitivity decreases for an object moving at a too high speed. This is because the speed of an object that a human can identify is limited.

2 is a graph showing contrast sensitivity with respect to speed and width in order to explain motion-based cognitive importance.

Referring to FIG. 2, in principle, the speed and the contrast sensitivity of the object are proportional to each other. However, when the speed is too large, the contrast sensitivity may decrease rapidly.

The contrast sensitivity associated with this speed is also affected by the size of the object. In this case, the size of the object is measured to be parallel to the moving direction. As shown in Fig. 2, the wider the width, the higher the perceptible speed limit.

3 illustrates an image encoding apparatus according to an embodiment of the present invention.

The technical idea of the present invention can be applied not only to the H.264 standard but also to various image codec standards such as H.263 and MPEG-2. However, for convenience of description, FIG. 3 illustrates that the technical idea of the present invention is applied to the H.264 standard. This is the case. That is, the rate controller 318, the transformer 320, the quantizer 324, the dequantizer 324, the de-blocking filter 330, the memory 332, and the intra prediction unit 340. ), The motion predictor 350 and the motion compensator 352 are blocks that can be included in a general image codec standard such as the H.264 standard. Therefore, detailed description of these blocks will be omitted.

The apparatus according to the first embodiment of the present invention includes a detector 300, a determiner 310, and a quantization unit 324.

The detector 300 detects at least one region of interest (including one macro block or a plurality of adjacent macro blocks) and the remaining region from the image frame.

According to an exemplary embodiment, the determination unit 310 includes a first importance value calculation unit 310 and / or a third importance value calculation unit 316 together with a rate control unit 318 to quantize the at least one ROI. The quantization step size of each of the macroblocks included in the image frame is determined such that an average of quantization step sizes to be used is smaller than an average of quantization step sizes to be used for quantization of the remaining area. The determination unit 310 according to another embodiment may further include a second important value calculator 314.

The quantization unit 324 quantizes the transform coefficients of each of the macroblocks according to the determined quantization step size.

Unlike the first embodiment, the apparatus according to the second embodiment of the present invention may not include the detector 300, and calculates a weighted sum of transform coefficients for each macroblock included in an image frame. A determination unit 310 for determining a quantization step size of the macro block based on the calculated weighted sum; And a quantization unit 324 for quantizing the transform coefficients of each of the macroblocks according to the determined quantization step size.

The determination unit 310 according to the present embodiment includes a second important value calculation unit 314 and a rate control unit 318. The second importance value calculator 314 calculates a weighted sum of the transform coefficients for each macro block included in the image frame. Here, each of the weights used for calculating the weighted sum has a value inversely proportional to the spatial frequency of the corresponding transform coefficient.

The general rate controller 318 determines the quantization scheme and performs the rate adjustment while looking at the number of bits and channel conditions generated by the encoding. The rate controller 318 according to the present embodiment may perform the above-described operation in the same manner. Further, the quantization step size of the macroblock is determined based on the calculated weighted sum.

According to the above-described embodiments, the rate controller 318 obtains at least one of the first important value, the second important value, and the third important value to determine the quantization step size. Hereinafter, for convenience, the determiner 310 may determine the first quantization step size. The technical spirit of the present invention will be described based on an embodiment including all of the important value calculator 312, the second important value calculator 314, and the third important value calculator 316.

There are various methods of detecting the ROI of the detector 300. If the application of the video conferencing, there may be a method for detecting a face that is a region of interest using a face detection algorithm already known.

Since the gaze may be focused on the moving object as described above, an object having a movement size greater than or equal to a predetermined threshold may be a region of interest among objects extracted from the image frame. According to this characteristic, the detector 300 generates a depth map by receiving a stereo image (that is, a main video input and an auxiliary video input of FIG. 3), and then generates a depth map. The at least one object is extracted from the image frame using. Next, the detector 300 receives a motion vector for each macroblock provided from the motion predictor 350, calculates an average magnitude value of the corresponding motion vectors for each of the extracted at least one object, and then selects the predetermined motion vector. At least one object having an average size value above the threshold is determined for each region of interest. That is, the depth map is generated through the main video image and the auxiliary video image having sufficient binocular vision, and then the background and the foreground (that is, the object) are distinguished from the generated depth map. The big object is the area of interest.

A region of interest detection method according to another embodiment of the present invention will be described later with reference to FIG. 4.

The first importance value calculator 312 calculates the first importance value for each macroblock, which is generally inversely proportional to the minimum distance from the at least one region of interest detected by the detector 300. For example, the first importance value calculator 312 receives an area map (that is, information about macroblocks belonging to each region of interest and information about macroblocks belonging to the remaining region) from the detector 300, and according to each macroblock. The first important value is calculated.

According to an embodiment, the first significant value of the k-th macroblock is the local bandwidth calculated by Equation 1. According to another embodiment, the first significant value of the k-th macroblock is w ^k _spatial calculated by Equation 5, which approximates Equation 1. Each parameter in Equation 5 is the same as that described in Equation 1 such that d ^k represents a minimum distance between the k-th macroblock and the ROI.

Meanwhile, as an example of a method of calculating the first weight of a macroblock belonging to the region of interest, Equation 1 or 5 is used as it is (that is, d ^k = 0 is set), instead of using Equation 1 or 5 , A method of allocating a first weight relatively higher than a first weight of macroblocks belonging to the remaining region.

Further, the first matter of the k-th macroblock in accordance with another exemplary embodiment is a result of w ^'k _spatial normalized by dividing the value of the acid is chuldoe by Equation 5 max {w _spatial}, is defined as Equation (6).

Max {w _spatial } in Equation 6 refers to the maximum value of the values calculated by Equation 5 for the macroblocks included in the current image frame, and generally, to the macroblock corresponding to the foviation point. The calculated value of Equation (5).

The second significant value calculator 314 receives transform coefficients such as DCT coefficients from the transformer 320, calculates a weighted sum of the transform coefficients for each macro block, and calculates the weighted sum of the calculated macro sums of the corresponding macroblocks. We decide with 2 important value. Here, each of the weights used for calculating the weighted sum has a value inversely proportional to the spatial frequency of the corresponding transform coefficient.

According to an embodiment, the second significant value of the k-th macroblock is calculated by Equation 7.

That is, the weight applied to the corresponding transform coefficient is the contrast sensitivity function value calculated by Equation 2. That is, in Equation 7, k and N represent the index of the macro block and the size of the macro block, respectively, x and y represent the index of the transform coefficient, and a (k) _{x, y} represent the x, y index of the macro block k. The corresponding transform coefficient, f _r (x, y), represents the spatial frequency of the transform coefficient corresponding to the x, y index. A, B, D, E, and F are predetermined positive real numbers, which are values defined in Equation 2 or values belonging to a range similar to these values.

The second matter of the k-th macroblock in accordance with another exemplary embodiment is a result of w ^'k _freq normalized by dividing the value as max {w} _freq calculated by Equation (7), is defined by equation (8).

Max {w _freq } in Equation 8 means the maximum value among the values calculated by Equation 7 for the macroblocks included in the current image frame.

The third importance value calculator 316 calculates a third importance value, ie, a motion-based perceptual importance level, for the at least one ROI and the remaining area detected by the detector 300.

For example, the third importance value calculator 316 is provided with an area map (ie, information about macroblocks belonging to each ROI and information about macroblocks belonging to the remaining area) from the detector 300, and predicts motion. The third critical value is determined for each macroblock by receiving the motion vector values from the unit 350 and calculating the width of the region (interested region or the remaining region) along the average direction of the movement and the average size of the movement for each region. When the third importance value is calculated according to Equation 11 to be described later, the third importance value is proportional to an average size of motion vectors belonging to the corresponding region within a range up to a predetermined motion size, and the average of the motion vectors is outside the range. Inversely proportional to size. Here, the predetermined motion size is proportional to the width of the corresponding ROI in the average direction of the motion vectors.

Here, since the motion-based perceptual importance described in FIG. 2 depends on the motion of the object and the width of the object, it may be modeled as in Equation 9.

In Equation 9 υ _k , and m _k represent the angular velocity and the width of the object associated with the k-th macro block. More specifically, m _k is an object width with respect to the advancing direction of the object, and may be calculated based on the direction of the motion vector and the area occupied by the detected object.

ν _k can be obtained using Equation 10. In Equation 10, M _max is a measure of the size of the motion vector, P is the screen resolution (eg, pixels per inch), L is the distance between the human and the image, and R _F is the frame rate.

Therefore, using Equation 9 described above, the third significant value of the k-th macroblock according to an embodiment is calculated by Equation 11.

In Equation 11, k denotes an index of the macroblock, υ ^k , and m ^k denote average angular velocities and widths of the region of interest or the remaining region to which the k-th macroblock belongs, and B ₁ , B ₂ , B ₃ , B ₄ , B ₅ , B ₆ , B ₇ , B ₈ , and B ₉ are preset positive real numbers representing corresponding values in Equation 9 or values in a range similar to the corresponding values.

The third matter in the k-th macroblock in accordance with another exemplary embodiment is a value calculated by Equation (11) as a result of w ^'k normalized by dividing the _motion max {w} _motion, is defined by Equation (12).

Max {w _motion } in Equation 12 refers to a maximum value among the values calculated by Equation 11 for the macroblocks included in the current image frame.

Meanwhile, the third importance calculation method using Equation 11 or Equation 12 is a method of calculating Equation 11 or Equation 12 in macroblock units, or Equation 11 or Equation in units of an area (ie, a region of interest or a remaining region). A method of calculating Equation 12 and allocating a calculated value of a corresponding region to macroblocks belonging to the region is given.

The rate controller 318 is configured to calculate the first important value, the second important value, and the third important value calculated by the first important value calculator 312, the second important value calculator 314, and the third important value calculator 316, respectively. On the basis, the quantization step size of the macro block is determined.

The quantization step size determination method according to an embodiment calculates the final significant value w ^k of the k-th macroblock based on the first to third significant values of the k-th macroblock, and then applies the calculated final importance value to the rate distortion model. To determine the optimal quantization step size.

Here, the method for calculating the final important value may be proportional to a product of the method for calculating the weighted sum of the first important value, the second important value, and the third important value, the first important value, the second important value, and the third important value. The final value may be determined as a value of, but is not necessarily limited thereto. An example of the final significance according to the former method is "final significance = first weight * first importance + second weight * second importance + third weight * third importance". , "Final important value = constant * first important value * second important value * third important value". Here, the first weight, the second weight, the third weight, and the constant are preset values.

On the other hand, if the rate control unit 318 receives only two of the first to third critical values (eg, the first and third critical values), the rate controller 318 may be a value proportional to the weighted sum or the product of each other as in the above-described method. Determine final significance. In addition, when the rate controller 318 receives only one of the first to third important values (eg, the first important value), the received important value itself becomes the final important value.

An example of a method of determining an optimal quantization step size by applying the final critical value thus calculated to a rate distortion model is the solution of the optimization problem defined by Equation 13, that is, the values of Δ ₁ , Δ ₂ , ..., Δ _K. It's like finding.

은 주어진 총 비트율을 나타낸다.In Equation 13, Δ _k is the quantization step size of the k-th macroblock, K represents the number of macroblocks included in the current image frame,

Represents the total bit rate given.

In addition, in Equation 13, D (Δ _k ) and R _k (Δ _k ) are functions representing the amount of distortion and data required when quantized by Δ _k , and according to the classical model, Equation 14 and Equation 15, respectively. Is defined as

Β is a constant in Equations 14 and 15, and a general value is 12. Ε ² in Equation 15 is a constant determined according to the statistical distribution of the object to be quantized, and has a value of 1.4 when the object to be quantized is Gaussian Source and 1.2 when the object is Laplacian Source. In Equation 15, σ _k ² represents a variance value of the object to be quantized. That is, the variance value obtained by the transform coefficient values belonging to the k-th macroblock.

Applying Lagrangian relaxation to the optimization problem given by Equation 13, the optimal solution, i.e., the optimal quantization step size Δ _k ^* of the k-th macroblock, is given by Equation 16.

That is, according to an embodiment of the present invention, the rate control unit 318 calculates the final significant value w ^k by the method described above, and then applies the calculated final significant value to Equation 16 to quantize the step size of the k-th macroblock. Determine. Referring to Equation 16, it can be seen that the optimal quantization step size is generally inversely proportional to the final significance.

4 shows an image encoding apparatus according to an embodiment of the present invention.

Unlike the detection method of FIG. 3, the detection method according to the exemplary embodiment of the present invention used in FIG. 4 detects an ROI through a mono image. Therefore, since the operation of the remaining blocks is the same as the description in FIG. 3, only the operation of the detection unit 400 will be described.

The detector 400 extracts macroblocks belonging to a boundary block and having a motion size value greater than or equal to a predetermined threshold from the image frame, grouping the extracted macroblocks, and including macroblocks that are similar to each other in terms of motion size and direction. After generating at least one group, and generating a convex hull including group members for each of the generated groups, an inner region of the generated at least one convex hull is determined as the region of interest. That is, according to this method, the macroblocks belonging to the boundary block and having a motion magnitude value above a predetermined threshold are estimated as the fobification points, and then each convex hull is decomposed to the macroposition points having similar directions and magnitudes of motion. To create each region of interest.

This method has the advantage that the object can be detected using only one camera. That is, unlike the binocular camera-based method of FIG. 3, the image can be used even in a situation where a depth map cannot be created by storing an image with one camera.

Here, the macro block belonging to the boundary block should be determined. The detection unit 300 according to an embodiment of the present invention determines the macro blocks in which motion compensation is performed in units of sub blocks as macro blocks belonging to the boundary block. This is because motion compensation in units of sub-blocks is applied to macro blocks having complex textures, that is, macro blocks having large dispersion values, and macro blocks having complex textures are likely to correspond to boundary blocks.

For example, the detector 300 receives the configuration information of the macro block (for example, whether or not the macro block is to be subjected to motion compensation on a sub-block basis) and the motion vector values from the motion predictor 350, and belongs to the boundary block, and belongs to a predetermined threshold. Macro blocks having the above motion size values are extracted, and the extracted macro blocks are grouped to generate at least one group including macro blocks that are similar to each other in terms of motion size and direction.

Since the line of sight is generally concentrated in the center of the screen, in consideration of this characteristic, the detection unit 400 according to another embodiment of the present invention provides the center area of the image frame-one macro block or a predetermined number of adjacent numbers. Include macroblocks-also determine the region of interest.

In FIG. 5A, macroblocks extracted from the macroblocks included in the image frame, which are estimated to belong to the boundary block and have a motion size value greater than or equal to a predetermined threshold, and the macroblocks corresponding to the central region are displayed in monochrome.

In FIG. 5B, the convex hull generated according to the result of FIG. 5A is represented by a thick solid line, and the center region and the inside of each convex hull become the region of interest.

6 is a flowchart illustrating an image encoding method according to an embodiment of the present invention.

Since the present embodiment may be performed in the apparatus of FIG. 3 or 4, the contents described with reference to FIGS. 3 and 4 also apply to the present embodiment, and thus a detailed description thereof will be omitted.

For convenience, the embodiment of FIG. 6 will be described with reference to FIG. 3.

In operation S600, the detection unit 300 detects at least one Region of Interest (ROI)-including one macroblock or a plurality of adjacent macroblocks-from the image frame and the remaining region.

In operation S610, the determiner 310 may include a macroblock included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one ROI is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. Determine each quantization step size.

In step S620, the quantization unit 324 quantizes the transform coefficients of each of the macroblocks according to the quantization step size determined in step S610.

7 is a flowchart illustrating an image encoding method according to another embodiment of the present invention.

Since the present embodiment may be performed in the apparatus of FIG. 3 or 4, the contents described with reference to FIGS. 3 and 4 also apply to the present embodiment, and thus a detailed description thereof will be omitted.

For convenience, the embodiment of FIG. 7 will be described with reference to FIG. 3.

In operation S700, the determiner 310 calculates a weighted sum of transform coefficients for each macroblock included in the image frame, and determines the quantization step size of the macroblock based on the calculated weighted sum. Here, each of the weights used for calculating the weighted sum has a value inversely proportional to the spatial frequency of the corresponding transform coefficient.

In operation S710, the quantization unit 324 quantizes transform coefficients of each of the macroblocks according to the determined quantization step size.

Since most embodiments of the present invention are interposed between an orthogonal transform process and a quantization process, it is possible to apply a scalable image compression codec such as H.264-SVC without special modification. If you grow up, you can fully understand.

The invention can also be embodied as computer readable code on a computer readable recording medium. Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, optical data storage devices, and the like, which are also implemented in the form of carrier waves (for example, transmission over the Internet). Include. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.

Such a method and apparatus of the present invention have been described with reference to the embodiments shown in the drawings for clarity, but these are merely exemplary, and various modifications and equivalent other embodiments are possible to those skilled in the art. Will understand. Therefore, the true technical protection scope of the present invention will be defined by the appended claims.

Embodiments of the present invention presented above may have an effect including the following advantages. However, all the embodiments of the present invention are not meant to include them all, and thus the scope of the present invention should not be understood as being limited thereto.

According to an embodiment of the present invention, the compression ratio can be improved without deteriorating visual quality, and thus can be effectively used even in a channel shortage or poor channel environment in a network.

In addition, according to an embodiment of the present invention, it can be compatible with the existing image compression standard, there is an advantage in terms of market share.

In addition, according to an embodiment of the present invention, even if the image encoding using the human visual characteristics does not have additional redundancy.

In addition, according to an embodiment of the present invention, in consideration of various factors affecting the human vision together, it is possible to remove a signal that is not visually important and to further improve compression ratio by allocating more bits to important information. have.

1 is a view for explaining a fob point.

2 illustrates contrast sensitivity in relation to speed and width to illustrate motion-based cognitive importance.

3 and 4 are block diagrams illustrating an image encoding apparatus according to an embodiment of the present invention.

5A and 5B are diagrams for describing a region of interest detection method according to an embodiment of the present invention.

6 and 7 are flowcharts illustrating an image encoding method according to an embodiment of the present invention.

Claims (22)

delete delete delete delete A detector for detecting, from an image frame, at least one Region of Interest (ROI), including one macroblock or a plurality of adjacent macroblocks, and a remaining region; A determination for determining a quantization step size of each of the macroblocks included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one region of interest is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. part; And A quantizer for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size, The detection unit may extract macroblocks belonging to a boundary block and having a motion size value greater than or equal to a predetermined threshold from the image frame, and group the extracted macroblocks to include macroblocks that are similar to each other in terms of motion size and direction. Create at least one group, generate a convex hull containing group members for each of the created groups, And the at least one region of interest is an inner region of the generated at least one convex hull. The method of claim 5, wherein the detection unit, And determining macroblocks for which motion compensation is performed in subblock units as macroblocks belonging to the boundary block. delete A detector for detecting, from an image frame, at least one Region of Interest (ROI), including one macroblock or a plurality of adjacent macroblocks, and a remaining region; A determination for determining a quantization step size of each of the macroblocks included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one region of interest is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. part; And A quantizer for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size, The determination unit, A first importance value calculator configured to calculate, for each macroblock, a first importance value inversely proportional to a minimum distance from the at least one region of interest; And And a rate controller for determining a quantization step size of the macroblock based on the calculated first importance value.  A detector for detecting, from an image frame, at least one Region of Interest (ROI), including one macroblock or a plurality of adjacent macroblocks, and a remaining region; A determination for determining a quantization step size of each of the macroblocks included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one region of interest is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. part; And A quantizer for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size, The determination unit, A first importance value calculator configured to calculate, for each macroblock, a first importance value inversely proportional to a minimum distance from the at least one region of interest; A second significant value calculator for calculating a weighted sum of transform coefficients for each macro block and determining the calculated weighted sum as a second significant value of the corresponding macro block; And A rate controller configured to determine a quantization step size of the macroblock based on the calculated first important value and the second important value, Each of the weights used for calculating the weighted sum has a value inversely proportional to the spatial frequency of the corresponding transform coefficient. A detector for detecting, from an image frame, at least one Region of Interest (ROI), including one macroblock or a plurality of adjacent macroblocks, and a remaining region; A determination for determining a quantization step size of each of the macroblocks included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one region of interest is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. part; And A quantizer for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size, The determination unit, A first importance value calculator configured to calculate, for each macroblock, a first importance value inversely proportional to a minimum distance from the at least one region of interest; A third importance value calculator configured to calculate a third importance value for the at least one region of interest and the remaining region; And A rate controller configured to determine a quantization step size of the corresponding macroblock based on the calculated first and third important values, The third importance value is proportional to an average size of motion vectors belonging to a corresponding region within a range up to a predetermined motion size, and inversely proportional to the average size of the motion vectors outside the range, And the predetermined motion size is proportional to a width of a corresponding ROI in an average direction of the motion vectors. A detector for detecting, from an image frame, at least one Region of Interest (ROI), including one macroblock or a plurality of adjacent macroblocks, and a remaining region; A determination for determining a quantization step size of each of the macroblocks included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one region of interest is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. part; And A quantizer for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size, The determination unit, A first importance value calculator for calculating a first importance value for each macro block in inverse proportion to a minimum distance from the at least one region of interest; A second significant value calculator for calculating a weighted sum of transform coefficients for each macro block and determining the calculated weighted sum as a second significant value of the corresponding macro block; A third importance value calculator configured to calculate a third importance value for the at least one region of interest and the remaining region; And A rate control unit for determining a quantization step size of the macroblock based on the calculated first important value, second important value, and third important value, Each of the weights used for calculating the weighted sum has a value inversely proportional to the spatial frequency of the corresponding transform coefficient, The third importance value is proportional to an average size of motion vectors belonging to a corresponding region within a range up to a predetermined motion size, and inversely proportional to the average size of the motion vectors outside the range, And the predetermined motion size is proportional to a width of a corresponding ROI in an average direction of the motion vectors. A detector for detecting, from an image frame, at least one Region of Interest (ROI), including one macroblock or a plurality of adjacent macroblocks, and a remaining region; A determination for determining a quantization step size of each of the macroblocks included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one region of interest is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. part; And A quantizer for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size, The determination unit, A third importance value calculator configured to calculate a third importance value for the at least one region of interest and the remaining region; And A rate controller configured to determine a quantization step size of the macroblock based on the calculated third important value, The third importance value is proportional to an average size of motion vectors belonging to a corresponding region within a range up to a predetermined motion size, and inversely proportional to the average size of the motion vectors outside the range, And the predetermined motion size is proportional to a width of a corresponding ROI in an average direction of the motion vectors. A detector for detecting, from an image frame, at least one Region of Interest (ROI), including one macroblock or a plurality of adjacent macroblocks, and a remaining region; A determination for determining a quantization step size of each of the macroblocks included in the image frame such that an average of quantization step sizes to be used for quantization of the at least one region of interest is smaller than an average of quantization step sizes to be used for quantization of the remaining areas. part; And A quantizer for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size, The determination unit, A second significant value calculator for calculating a weighted sum of transform coefficients for each macro block and determining the calculated weighted sum as a second significant value of the corresponding macro block; A third importance value calculator configured to calculate a third importance value for the at least one region of interest and the remaining region; And A rate control unit for determining a quantization step size of the macroblock based on the calculated second important value and the third important value, Each of the weights used for calculating the weighted sum has a value inversely proportional to the spatial frequency of the corresponding transform coefficient, The third importance value is proportional to an average size of motion vectors belonging to a corresponding region within a range up to a predetermined motion size, and inversely proportional to the average size of the motion vectors outside the range, And the predetermined motion size is proportional to a width of a corresponding ROI in an average direction of the motion vectors. The said 1st important value calculation part of Claim 8 thru | or 11, Equation

Where ν is the viewing distance, k is the index of the macroblock, d ^k is the minimum distance between macroblock k and the at least one region of interest, N is the image width, e ₂ , α, and CT ₀ are preset values And calculating a first significant value of the macroblock according to. The method of claim 14, wherein the first important value calculation unit, Calculating a maximum value obtained by the equation for the image frame, normalizing the result of the equation for the macro block to the maximum value, and determining the normalized result as the first significant value of the corresponding macro block And a video encoding apparatus. The method according to any one of claims 9, 11, and 13, wherein the second important value calculating unit Equation

Where k and N represent the macroblock index and the macroblock size, respectively, x and y represent the transform coefficient index and a (k) _{x and y} represent the transform corresponding to the macro block k x and y index. Coefficient, f _r (x, y) is the spatial frequency of the transform coefficient corresponding to the x, y index, and A, B, D, E, F represent a predetermined positive real number) An image encoding device, characterized by calculating an important value. The method of claim 16, wherein the second important value calculation unit, Calculating a maximum value obtained by the equation for the image frame, normalizing the result of the equation for the macro block to the maximum value, and determining the normalized result as a second significant value of the corresponding macro block And a video encoding apparatus. The said 3rd important value calculation part of Claim 10 thru | or 13, Equation

(Where k represents the index of the macroblock, υ ^k , and m ^k represent the average angular velocity and width of the region of interest or the remaining region to which macroblock k belongs, and B ₁ , B ₂ , B ₃ , B ₄ , B _{And 5} , B ₆ , B ₇ , B ₈ , and B ₉ represent a predetermined amount of real number. 3. The method of claim 18, wherein the third important value calculation unit, Calculating a maximum value obtained by the equation for the image frame, normalizing the result of the equation for the macro block to the maximum value, and determining the normalized result as a third important value of the corresponding macro block And a video encoding apparatus. The method of claim 11, wherein the rate control unit, And a weighted sum of the first significant value, the second significant value, and the third significant value for each macro block, and determines the quantization step size of the macro block based on the calculated weighted sum. The method of claim 11, wherein the rate control unit, And calculating a value proportional to the product of the first important value, the second important value, and the third important value for each macro block, and determining the quantization step size of the macro block based on the calculated value. Device. A determination unit for calculating a weighted sum of transform coefficients for each macroblock included in an image frame and determining a quantization step size of the macroblock based on the calculated weighted sum; And A quantizer for quantizing transform coefficients of each of the macroblocks according to the determined quantization step size, Each of the weights used for calculating the weighted sum has a value inversely proportional to the spatial frequency of the corresponding transform coefficient.

Images