KR101866101B1 - Method and Apparatus for Sensor-Triggered Weighted Prediction for High Efficiency Video Coding - Google Patents

Abstract

A method and apparatus for predicting weights based on an illuminance sensor in HEVC encoding are presented. In the HEVC encoding proposed in the present invention, the weightedness predicting method based on the illuminance sensor includes a step of comparing a rate of change of sensor data received from a sensor with a threshold value, a step of calculating a weight and an offset value when the rate of change of the sensor data is higher than a threshold value Calculating a weighting and an offset value, then setting the AE flag to 1, encoding the current frame into a WP configuration using the calculated weight and offset values, determining whether the rate of change of the sensor data is less than a threshold value A step of estimating a weight according to the determined AE, and a step of comparing an estimated weight with a default value to set an AE flag.

Description

[0001] The present invention relates to a method and apparatus for predicting a weight based on an illuminance sensor in HEVC encoding,

The present invention relates to a sensor-trigger weight prediction method and apparatus for high-efficiency video coding.

Recently, wearable devices are emerging as important applications. A wearable device equipped with a camera can record the daily life of a person or vividly broadcast scenes and places. Due to wearability, these devices can be used both internally and externally. In particular, there are various light sources outside. Light intensity changes are sometimes unpredictable and sudden. The camera captures these lighting changes as well as the view. The Auto Exposure (AE) technique is one of the tools to cope with lighting changes. When the camera obtains pixel information at 8-bit depth, the color intensity is from 0 to 255. If the input image is too dark or too bright, the saturation effect will render the details inaudible. To overcome the limited dynamic range, the AE method controls the degree of exposure and allows the input image to reach the target brightness under a given illumination.

1 is a view showing exposure adjustment after a sudden illumination change according to the prior art.

Referring to FIG. 1, when the indoor lighting is turned off, the captured image becomes very dark. After that, the captured image gradually begins to brighten.

Most video encoders seeking high compression ratios use an inter-prediction scheme to exploit the temporal redundancy of video where the compression ratio is entirely dependent on the difference between the current and reference frames. Once the video of the wearable device is encoded, a change in illumination has a significant impact on compression efficiency. Even if the scene is static, the difference between consecutive frames becomes very large due to the change of illumination as shown in Fig. To address compression inefficiencies due to illumination changes, some video coding standards such as H.264 and High Efficiency Video Coding (HEVC) employ Weighted Prediction (WP) Is compensated by adding a weight to the predicted block of the frame.

SUMMARY OF THE INVENTION The present invention provides a method and apparatus for predicting sensor-trigger weights for high-efficiency video coding that utilizes the advantages of WP to reduce computational complexity and memory access overhead.

According to an aspect of the present invention, in an HEVC encoding method of the present invention, an illuminance sensor-based weight prediction method includes comparing a rate of change of sensor data received from a sensor with a threshold value, Calculating a weight and an offset value, calculating a weight and an offset value, then setting the AE flag to 1, encoding the current frame in a WP configuration using the calculated weight and offset values, Determining the AE, estimating the weight according to the determined AE, and comparing the estimated weight with the default value to set the AE flag.

The step of estimating the weight according to the determined AE when the rate of change of the sensor data is lower than the threshold value and the step of comparing the estimated weight with the default value to set the AE flag may be such that the rate of change of the sensor data is larger than the threshold Estimating a weight when the determined AE is 1; comparing an estimated weight with a default value; setting an AE flag to 0 if the estimated weight is equal to a default value; Determining whether the WP is applied based on the sensor data received from the sensor, and determining whether to apply the WP based on the sensor data received from the sensor if the estimated weight is not the same as the default weight.

Wherein if the determined AE is equal to 1, weighting in the first three frames is calculated from the pixel intensity, and if the pixel intensity is linear according to the AE effect by the initial calculated value for the first three frames, The weights in the remaining frames are estimated.

The proposed method further includes a step of setting a weight and an offset value as default values and deactivating WP when the determined AE is 0.

In another aspect, in the HEVC encoding proposed in the present invention, the lightness sensor-based weight prediction apparatus compares the rate of change of the sensor data received from the sensor with a threshold value, and when the rate of change of the sensor data is lower than the threshold value A calculating unit for calculating a weight and an offset value when the rate of change of the sensor data is higher than a threshold value; A weight estimation unit for estimating a weight when the AE determined by the comparison unit is 1, a weight estimation unit for estimating a weight based on the weight estimated from the weight estimation unit, The AE flag is set according to the comparison result, and the sensor data received from the sensor And determines whether or not the WP is applied.

The weight estimator may calculate the weights in the first three frames from the pixel intensities and linearly change the pixel intensities according to the AE effect based on the initial calculated values for the first three frames, do.

If the rate of change of the sensor data is higher than the threshold value, the control unit calculates the weight and the offset value through the calculation unit, and then sets the AE flag to 1.

If the rate of change of the sensor data is lower than the threshold value, the AE is 1, the AE flag is set to 0 when the estimated weight is equal to the default value, and whether or not the WP is applied based on the sensor data received from the sensor Determines whether or not WP is applied based on the sensor data received from the sensor when the rate of change of the sensor data is lower than the threshold value and the AE is 1 and the default weight is not the same as the estimated weight, If the rate is lower than the threshold and AE is 0, the weight and offset values are set to default values and WP is disabled.

According to embodiments of the present invention, an additional overhead for WP is achieved by using a sensor-trigger weighting prediction method and apparatus for high-efficiency video coding that utilizes the advantages of WP to reduce computational complexity and memory access overhead Can occur in a timely manner.

1 is a view showing exposure adjustment after a sudden illumination change according to the prior art.
2 is a flowchart illustrating a sensor-based weight prediction method according to an embodiment of the present invention.
3 is a block diagram of a sensor-based weight prediction apparatus according to an embodiment of the present invention.

The camera of the wearable device captures not only the view but also the change in illumination. Even if the scene is static, the difference between successive input frames due to lighting changes can be very large. The HEVC standard adopts weighted prediction (WP) for illumination compensation. However, conventional WPs require high computational complexity and memory access overhead. In the present invention, a sensor-triggered WP is proposed to activate the WP in a timely manner. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a flowchart illustrating a sensor-based weight prediction method according to an embodiment of the present invention.

When the WP is in an on state in the HEVC encoder, pre-processing is performed to determine whether the intensity compensation is applied. First, the weights and offset values are calculated based on the pixel intensities of the current and reference frames. The weight is the ratio of the two pixel deviations of the current frame and the reference frame. The pixel deviation is the sum of the differences between each pixel intensity and the average pixel intensity. The offset is calculated by multiplying the average pixel intensity of the reference frame by a weight and then subtracting this value from the average pixel intensity of the current frame. Next, SAD _i and SAD _{i, wp} are obtained and compared. SAD _i represents the sum of the absolute differences between the current and ith (i _th ) reference frames in pixel units, but SAD _{i, w} is calculated in a manner similar to multiplying weights and adding offsets to the pixels of the ith reference frame . If SAD _{i, wp} is less than SAD _i , the weighted and predicted reference frame is used for inter-prediction. To perform this weighted prediction, weights and offset values must be calculated for all reference frames. In addition, SAD calculation and comparison are required. The computational complexity increases in proportion to the number of reference frames and the video resolution. Since most of the reference frames are stored in external memory, the required memory bandwidth also increases. For 1920x1080 sized video, 2MB of memory reads are required in one frame of reference. When four reference frames are used for encoding, the required memory bandwidth is only 8 MB for WP. Despite this significant computation and memory access overhead, the benefits of WP will be negligible if the illumination does not change.

In the proposed HEVC encoding, weighting based prediction method based on an illuminance sensor compares the rate of change of sensor data received from a sensor with a threshold value, calculating weight and offset values when the rate of change of sensor data is higher than a threshold value , Calculating a weight and an offset value, then setting the AE flag to 1, encoding the current frame in the WP configuration using the calculated weight and offset values, and if the rate of change of the sensor data is lower than the threshold value, Estimating a weight according to the determined AE, and comparing the estimated weight with a default value to set an AE flag.

First, the rate of change of the sensor data received from the sensor is compared with a threshold value (210). In the present invention, a sensor-triggered WP is proposed to reduce computational complexity and memory access overhead using the advantages of WP. Especially, it focuses on AE after sudden illumination change. In Fig. 2, the encoder receives illumination information from the sensor as well as frames from the camera. Before encoding the current frame, the illumination change is checked. A rate of change greater than a given threshold represents a sudden change in illumination. As a result, this leads to AE.

 If the rate of change of the sensor data is higher than the threshold value, a weight value and an offset value are calculated (221). The weights and offset values are calculated in the same manner as described above.

Thereafter, the AE flag is set to 1 (222). Then, the current frame is encoded into the WP configuration using the calculated weight and offset values (240).

On the other hand, a case in which the rate of change of the sensor data received from the sensor in step 210 is compared with the threshold value and the rate of change of the sensor data is lower than the threshold value will be described below.

The step of estimating a weight based on the determined AE when the rate of change of the sensor data is lower than the threshold value and the step of comparing the estimated weight with the default value to set the AE flag may include: A step 231 of determining AE, a step 232 of estimating a weight when the determined AE is 1, a step 233 of comparing an estimated weight with a default value 233, A step 234 of setting the AE flag to 0 and determining whether WP is applied based on the sensor data received from the sensor, and if the estimated weight is not the same as the default value, based on the sensor data received from the sensor And determining whether to apply the WP (step 240).

In other words, if the rate of change of illumination is less than a given threshold, the AE flag is checked (231). The zero AE flag means there is no current AE effect since there was no sudden light change in the past. In this case, WP is not required. Thus, the weight and offset values are set to default (0) (235). This default setting actually disables WP (240).

If the AE determination result in step 231 is AE = 1, the next input frame gradually becomes darker or brighter. Here, a weight is estimated (232).

For example, in the first three frames, the weights are calculated from the pixel intensities in the same manner as described above. From the initial calculation, the remaining weights are estimated since the pixel intensity changes linearly due to the AE effect.

If the estimated weight is compared with the default weight (233) and the estimated weight reaches the default value, the AE effect is terminated and the AE flag is set to 0 (234). Thereafter, the proposed method determines whether WP is applied (240) based on the information from the sensor.

3 is a block diagram of a sensor-based weight prediction apparatus according to an embodiment of the present invention.

In the proposed HEVC encoding, an illuminance sensor-based weight prediction apparatus includes a sensor 310, a comparison unit 320, a calculation unit 330, an encoding unit 350, a weight estimation unit 360, and a control unit 340 .

The comparator 320 compares the rate of change of the sensor data received from the sensor 310 with a threshold value. When the rate of change of the sensor data is lower than the threshold, the comparator 320 determines the AE. Compare the weights with the default values.

The calculation unit 330 calculates a weight value and an offset value when the rate of change of the sensor data is higher than the threshold value.

The encoding unit 350 encodes the current frame in the WP configuration using the weight and offset values calculated in the calculation unit.

If the AE determined by the comparison unit is 1, the weight estimation unit 360 estimates a weight.

The control unit 340 compares the weights and default values estimated from the weight estimation unit through the comparison unit, sets the AE flag according to the comparison result, and determines whether to apply the WP based on the sensor data received from the sensor. The operation of the weight prediction device based on the illuminance sensor in the HEVC encoding proposed below will be described in more detail.

First, the rate of change of the sensor data received from the sensor 310 is compared with a threshold value through the comparing unit 320. In the present invention, a sensor-triggered WP is proposed to reduce computational complexity and memory access overhead using the advantages of WP. Especially, it focuses on AE after sudden illumination change. In Fig. 2, the encoder accepts illumination information from the sensor 310 as well as frames from the camera. Before encoding the current frame, the illumination change is checked. A rate of change greater than a given threshold represents a change in illumination through the abrupt comparator 320. As a result, this leads to AE.

 When the rate of change of the sensor data is higher than the threshold value, the weight and offset values are calculated through the calculation unit 330. The weights and offset values are calculated in the same manner as described above.

Thereafter, the AE flag is set to 1 through the control unit 340. Then, the current frame is encoded into the WP configuration using the weight value and the offset value calculated in the encoding unit 350.

On the other hand, a case where the rate of change of the sensor data received from the sensor 310 is compared with the threshold value through the comparing unit 320 and the rate of change of the sensor data is lower than the threshold value will be described below.

If the rate of change of the sensor data is lower than the threshold value, the comparator 320 determines the AE. If the determined AE is 1, the weight estimation unit 360 estimates a weight. The comparison unit 320 compares the estimated weight and the default value.

If the estimated weight is the same as the default value, the AE flag is set to 0 through the control unit 340. Based on the sensor data received from the sensor 310, it is determined whether or not the WP is applied.

On the other hand, if the estimated weight is not the same as the default value, the control unit 340 determines whether to apply the WP based on the sensor data received from the sensor 310.

In other words, when the illumination change rate is smaller than a given threshold value, the AE flag is checked through the comparison unit 320. [ The zero AE flag means there is no current AE effect since there was no sudden light change in the past. In this case, WP is not required. Therefore, the weight value and the offset value are set to default (default) 0 through the control unit 340. This default setting actually disables WP in the encoding unit 350. [

When the AE determination result through the comparison unit 320 is AE = 1, the next input frame gradually becomes darker or brighter. Here, the weight value is estimated through the weight value estimator 360.

For example, in the first three frames, the weights are calculated from the pixel intensities in the same manner as described above. From the initial calculation, the remaining weights are estimated since the pixel intensity changes linearly due to the AE effect.

When the estimated weight reaches the default value, the AE effect is terminated and the AE flag is set to 0 through the control unit 340. The AE flag is set to " 0 " Then, based on the information from the sensor 310, it is determined whether WP is applied.

In the performance test of the method and apparatus for predicting weights based on the illuminance sensor in the proposed HEVC encoding, for example, the AE effect was generated by turning on and off the light.

<Table 1>

<Table 2>

<Table 3>

Referring to Table 1, in test 1, five sudden light changes occur during 300 frames, while in test two, two sudden light changes occur during 900 frames. The test video was encoded using the HEVC reference software HM13.0 with low delay P configuration. The number of reference frames is set to 1 and the quantization parameter is set to 32. [ In Tables 1, 2 and 3, WP Off and WP On, which represent the weighted prediction options of HM13.0, are off and on, respectively. Table 1 shows the compression efficiency of the proposed algorithm compared to WP Off and WP On. The bit rate of the proposed algorithm is between WP On and Off. Estimation error and WP activation only for AE cause bit rate increase compared to WP On. Table 2 and Table 3 show WP overhead. If WP is always On, the necessary computation and read bandwidth overhead is important. However, in the proposed technique, the additional overhead for WP occurs in a timely manner based on the sensor information.

Thus, in accordance with embodiments of the present invention, the additional overhead for the WP is reduced by using a sensor-trigger weight prediction method and apparatus for high-efficiency video coding that utilizes the advantages of WP to reduce computational complexity and memory access overhead, It can happen on a timely basis based on information.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device As shown in FIG. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (8)

In a sensor-based weight prediction method,
Comparing a rate of change representing a degree of change in brightness per hour of sensor data received from the sensor with a threshold value;
Calculating a weight and an offset value if the rate of change of the sensor data is higher than a threshold value;
Calculating a weight and an offset value, then setting the auto exposure flag to one, and using the calculated weight and offset values to encode the current frame into a weighted prediction configuration; And
Estimating a weight based on the determined automatic exposure when the rate of change of the sensor data is lower than a threshold value, setting an automatic exposure flag by comparing the estimated weight with a default value
Wherein the weighting factor is a weighted value. The method according to claim 1,
Wherein the step of estimating a weight based on the determined automatic exposure when the rate of change of the sensor data is lower than the threshold value and determining the automatic exposure by comparing the estimated weight with the default value,
Determining the automatic exposure flag if the rate of change of the sensor data is lower than a threshold value;
Estimating a weight when the determined automatic exposure flag is 1;
Comparing the estimated weight with a default value;
Setting the automatic exposure flag to 0 if the estimated weight is the same as the default value, and determining whether to apply the weight prediction based on the sensor data received from the sensor; And
Determining whether the weighted prediction is applied based on the sensor data received from the sensor if the estimated weight is not the same as the default weighted value
Wherein the weighting factor is a weighted value. 3. The method of claim 2,
Wherein when the determined automatic exposure flag is 1, estimating a weight includes:
The weights in the initial three frames are calculated from the pixel intensities and the pixel intensity is linearly changed according to the automatic exposure effect by the initial calculated values for the initial three frames so that the weights in the remaining frames are estimated
Weighting Prediction Method Based on Light Sensor. 3. The method of claim 2,
If the determined automatic exposure flag is 0, setting the weight and offset values as default values and deactivating weight prediction
Based on the brightness sensor. A sensor-based weight prediction apparatus,
The rate of change of the rate of change of the brightness per unit time of the sensor data received from the sensor is compared with a threshold value and when the rate of change of the sensor data is lower than the threshold value, the automatic exposure is determined, A comparing unit comparing the estimated weight with a default value;
A calculation unit for calculating a weight value and an offset value when the rate of change of the sensor data is higher than a threshold value;
An encoding unit in which a current frame is encoded in a weight prediction configuration using the weight and offset values calculated in the calculation unit;
A weight estimator for estimating a weight when the automatic exposure flag determined by the comparator is 1; And
A controller for comparing the weight value estimated from the weight estimating unit with the default value through the comparing unit to set an automatic exposure flag according to a result of the comparison and determining whether to apply the weight prediction based on the sensor data received from the sensor,
And a weight sensor based on the brightness sensor. 6. The method of claim 5,
The weight-
The weights in the initial three frames are calculated from the pixel intensities and the pixel intensity is linearly changed according to the automatic exposure effect by the initial calculated values for the initial three frames to estimate the weights in the remaining frames
Weighting Prediction Device Based on Light Sensor. 6. The method of claim 5,
Wherein,
When the rate of change of the sensor data is higher than the threshold value, the weight and offset values are calculated through the calculation unit, and then the automatic exposure flag is set to 1
Weighting Prediction Device Based on Light Sensor. 6. The method of claim 5,
Wherein,
If the rate of change of the sensor data is lower than the threshold value, the automatic exposure flag is 1, and the default value is the same as the estimated weight, the automatic exposure flag is set to 0 and the weighted prediction is applied based on the sensor data received from the sensor And,
Determining whether the weight prediction is applied based on the sensor data received from the sensor when the rate of change of the sensor data is lower than the threshold, the auto exposure flag is 1, and the default weight is not the same as the estimated weight,
When the rate of change of the sensor data is lower than a threshold value and the auto exposure flag is 0, the weight and offset values are set to default values, and the weight prediction is disabled
Weighting Prediction Device Based on Light Sensor.

Images