KR101945233B1 - Method and Apparatus for Stabilizing Video - Google Patents

Abstract

An image stabilization method and apparatus are disclosed.
There is provided an image stabilization method and an apparatus for stabilizing an image to be reproduced at a speed that is obtained by calculating a path for each frame to recover an error when calculating a camera path based on a variation amount between frames, .

Description

[0001] The present invention relates to an image stabilization method and apparatus,

This embodiment relates to an image stabilization method and apparatus.

 The contents described below merely provide background information related to the present embodiment and do not constitute the prior art.

Generally, "Time Lapse" is an imaging method in which still image data is continuously shot with an interval and recorded as one moving image data. For example, "time-lapse" is a photographing method in which a process of growing a plant or a change in weather is photographed at a predetermined time interval (interval) and then stored as one moving image data.

In recent years, as general imaging equipment such as 'sports' / 'action' / 'life' camcorders such as a handheld camera or GoPro HERO, a 'wearable camera' It is a tendency to frequently record, store and share everyday life, extreme situations, and non-shootable sports. However, images taken with a general shooting device (such as a 'sports' / 'action' / 'life' camcorder or a 'smart phone') have a problem that the image is too long or complicated and monotonous for the user to watch later.

Conventional techniques for solving the above-mentioned problems include a technique of applying a time-lapse image to an image photographed by general photography equipment ('sports' / 'action' / 'living' camcorder or 'smart phone' A technique of sampling only a frame corresponding to a set speed (for example, " 10 times speed ") has been used. In other words, the vast majority of videos that have not been edited and edited by professionals are long-lived and monotonous, providing a time-lapse feature that reduces video length for certain vendors.

However, when a frame corresponding to a predetermined speed is simply sampled and played back (image playback speed), the image may appear shaky or shaky. In other words, the time-lapse skips the frame according to the user-specified speed. In the case of the image taken while the photographer moves without the professional equipment for stabilizing the photographing equipment, it is inconvenient to view the camera because the camera has an unstable camera path there is a problem. Especially, in the case of playing back an image captured in a moving state rather than a fixed camera at a speed of 2x, the image corresponding to the final result applied with time-lapse may be largely shaken.

In this embodiment, when a camera path is calculated on the basis of a change amount between frames, an image stabilization method is performed in which an image to be reproduced at a speed is stabilized by calculating and optimizing paths for each frame in order to solve the cumulative error, The purpose of the device is to provide.

According to an aspect of the present invention, there is provided a method of stabilizing an image by a video double speed playback apparatus, the method comprising: inputting a selected one of frames of an input image; A frame-by-frame path generation step of generating a frame-by-frame camera path for each of the selected frames by decomposing the frame based on a motion model of a correlation matrix for the selected frame; An optimization process of optimizing paths by applying line fitting or applying a smoothing filter to each camera path for each frame; A stabilization process for generating a stabilization path after the optimizing paths are restored to a matrix; And a rendering step of applying a stabilization path to the selected frame and outputting a stabilized result image.

According to another aspect of the present invention, there is provided an image processing method comprising: inputting, in combination with hardware, inputting a selected one of frames of an input image; A frame-by-frame path generation step of generating a frame-by-frame camera path for each of the selected frames by decomposing the frame based on a motion model of a correlation matrix for the selected frame; An optimization process of optimizing paths by applying line fitting or applying a smoothing filter to each camera path for each frame; A stabilization process for generating a stabilization path after the optimizing paths are restored to a matrix; And a rendering process of outputting a stabilized result image by applying the stabilization path to the selected frame.

According to another aspect of the present invention, there is provided an image processing apparatus including: a frame input unit receiving a selected one of frames of an input image; A frame decomposing unit for decomposing the frame based on the motion model of the correlation matrix for the selected frame to generate a camera path for each selected frame; A path optimizing unit for optimizing paths by applying line fitting or applying a smoothing filter to each of the frame-by-frame camera paths; A restoration unit for restoring the optimization paths to a matrix and generating a stabilization path; And a rendering unit for applying the stabilization path to the selected frame and outputting a stabilized resultant image.

As described above, according to the present embodiment, when the camera path is calculated based on the amount of change between frames, in order to solve the cumulative error, the route for each frame is calculated, optimized, and restored to stabilize the image It is effective.

FIGS. 1A and 1B are block diagrams schematically showing a video double speed playback apparatus according to the present embodiment.
FIG. 2 is a flowchart for explaining the video double speed playback process according to the present embodiment.
FIGS. 3A to 3C are diagrams for explaining a comparison between a general camera path and a stabilized camera path according to image double speed playback.
FIG. 4 is a flowchart for explaining a frame matching process according to the present embodiment.
5 is a flowchart illustrating a frame selection process according to the present embodiment.
6 is a flowchart for explaining a path stabilization process according to the present embodiment.
7 is a view for explaining a smooth path according to the present embodiment.
8 is a flowchart illustrating a rendering process according to the present embodiment.
9 is an exemplary diagram for explaining a cost matrix according to the present embodiment.
FIGS. 10A and 10B are diagrams for explaining a comparison between frame selection during normal speed reproduction and frame selection according to the present embodiment.

Hereinafter, the present embodiment will be described in detail with reference to the accompanying drawings.

The 'double speed reproduction' described in the present embodiment basically means a video which operates not only in a speed up video but also in a stable camera motion. In other words, the 'speed reproduction' described in this embodiment means a fast image (stable and fast reproduced image) moving in a stable camera path.

The 'stable image' described in this embodiment means that an image matching the camera path photographed by the expert is output. Normally, a specialist shoots a moving camera path that does not shake the image by using an imaging assistant when shooting while moving.

'Correlation matrix (homography)' described in this embodiment means a matrix including variation parameter between adjacent frames. "Correlation matrix" is x-axis shift amount (t _x) parameters, y-axis movement amount (t _y) parameters, the rotational angle (Angle) (θ) parameters, the size (Scale) parameter, the aspect ratio (Aspect Ratio) parameters, the front end (Shear ) ≪ / RTI > parameter and a perspective parameter. The 'correlation matrix' may be a 3 × 3 matrix. The 'correlation matrix' does not need to be recalculated once it is calculated and saved.

The 'cost matrix' described in this embodiment means a kind of scalar value obtained by comparing the correlation matrix of the reference frame with the correlation matrix of the remaining frames. The 'cost matrix' is newly calculated using the pre-stored 'correlation matrix' according to the speed that the user inputs. A chain rule may be applied to the correlation matrix between neighboring frames in order to speed up the computation speed in calculating the 'cost matrix'.

FIGS. 1A and 1B are block diagrams schematically showing a video double speed playback apparatus according to the present embodiment.

The video double speed playback apparatus 100 according to the present embodiment includes an input unit 110, a frame matching unit 120, a frame selecting unit 130, a path stabilizing unit 140, and a rendering unit 150. The constituent elements included in the image data playback device 100 are not limited thereto.

Each component included in the image data playback device 100 may be connected to a communication path connecting a software module or a hardware module in the device so as to operate organically with each other. These components communicate using one or more communication buses, signal lines, or wireless communications.

Each component of the video playback device 100 shown in FIG. 1 refers to a unit for processing at least one function or operation, and may be implemented as a software module, a hardware module, or a combination of software and hardware.

The video double speed playback apparatus 100 according to the present embodiment basically means a device that not only provides speed up video but also outputs a video (a video that is stably and quickly reproduced) in a stable camera path.

The video image playback apparatus 100 is provided with (i) a communication device such as a communication modem for performing communication with various devices or wired / wireless networks, (ii) a memory for storing various programs and data, (iii) And a microprocessor for controlling the microprocessor and the like. According to at least one embodiment, the memory may be a computer such as a random access memory (RAM), a read only memory (ROM), a flash memory, an optical disk, a magnetic disk, or a solid state disk Readable recording / storage medium. According to at least one embodiment, a microprocessor can be programmed to selectively perform one or more of the operations and functions described in the specification. In accordance with at least one embodiment, the microprocessor may be implemented in hardware, such as an Application Specific Integrated Circuit (ASIC), in wholly or partially of a particular configuration. If the components of the image-speed playback apparatus 100 are implemented as software modules, they can be stored in the memory.

The input unit 110 receives the original image.

The frame matching unit 120 calculates a correlation matrix by matching feature points of each frame of the original image with each other. The frame matching unit 120 includes a feature point extracting unit 122, a descriptor checking unit 124, and a correlation calculating unit 126.

The feature point extracting unit 122 extracts feature points for all the frames included in the original image. The descriptor confirmation unit 124 confirms a descriptor for each minutiae of each frame.

The correlation calculation unit 126 calculates correlation matrices between frames by matching descriptors among neighboring frames among all frames. Correlation matrix is a 3 × 3 matrix, a change amount parameter between adjacent frames, x-axis movement amount (t _x) parameters, y-axis movement amount (t _y) parameters, the angle of rotation (θ) parameters, size parameters, and the aspect ratio parameter, shear parameters And a projection parameter. The correlation calculation unit 126 may selectively remove an outlier using an algorithm for selecting a model having a maximum consensus in random sample data when matching the descriptors between neighboring frames. .

The frame selector 130 calculates a cost matrix comparing the associativity between frames based on the correlation matrix, and selects only the path frame having the highest correlation among the frames based on the cost matrix. The frame selection unit 130 includes a relevance calculation unit 132, a weight calculation unit 134, and a cost calculation unit 136. [

The relevancy calculator 132 calculates inter-frame associations based on the correlation matrix of the first frame and the correlation values of the correlation matrix up to the frame in which the frame is twice as large as the playback speed. The relevancy calculator 132 calculates inter-frame associations by multiplying the correlation matrix of the frame by the correlation matrix of the subsequent frames.

The weight calculation unit 134 calculates a value for satisfying the speed set by the user for each frame. The weight calculation section 134 to on the basis of the i-th frame index (i), j-th frame index (j), the user sets playback speed value (v), the threshold value (τ _s) satisfy the speed the user has preset (C _s ) for the first time.

The cost calculation unit 136 calculates the cost matrix by combining the inter-frame relevance and the values for satisfying the speed set by the user, and repeats the process of selecting the frame having the minimum value among the cost matrices as the path frame to the last frame . The cost calculation unit 136 calculates the amount of image movement and the amount of overlapping (C _m ) for the amount of overlapping, the weight (? _S ) for the value for keeping the speed, and the double speed satisfaction value (C _s ) for satisfying the speed set by the user Based on the cost matrix is calculated.

The path stabilizing unit 140 optimizes and stabilizes the camera path based on the path frame received from the frame selecting unit 130. Although the path stabilizing unit 140 is described as being included in the video playback device 100, the path stabilizing unit 140 is not necessarily limited to this. The path stabilizing unit 140 may be an independent module ). ≪ / RTI > Since the path stabilizing unit 140 performs stabilization based on the frame received from the frame selecting unit 130, the path stabilizing unit 140 calculates the correlation matrix of the frame matching unit 120 and the cost matrix of the frame selecting unit 130 It can operate independently of the calculation.

The path stabilization unit 140 includes a frame input unit 142, a path decomposition unit 144, a path optimization unit 146, and a path restoration unit 148. The frame input unit 142 receives some selected path frames from all of the frames from the frame selecting unit 130. [ The frame input unit 142 receives the path frames selected at variable intervals among all the frames from the frame selecting unit 130. [ The path decomposition unit 144 generates a frame-by-frame path for each frame. The path decomposition unit 144 decomposes the frame based on the motion model of the correlation matrix for the frame to generate a frame-by-frame path. The path decomposing unit 144 decomposes the number of path frames corresponding to the number of parameters included in the correlation matrix to generate a frame-by-frame path. The path optimization unit 146 generates optimization paths that are optimized for each frame-by-frame path. The path optimizing unit 146 calculates a smooth path, which is a result of performing line fitting or applying a smoothing filter to each frame for each frame. The path restoring unit 148 restores the optimization paths to a matrix, and then generates a stabilized path. The path restoration unit 148 calculates a warping matrix in which a smooth path optimized for each frame-specific path is restored into a matrix.

The rendering unit 150 renders an image for double speed playback based on the stabilized camera path. The rendering unit 150 multiplies all the pixel values of the path frame by the wavelet matrix B (t), and renders the image so as to move the pixel values of the frame to the output coordinates.

FIG. 2 is a flowchart for explaining the video double speed playback process according to the present embodiment.

The video-image playback device 100 receives the original video (captured video) (S210). For example, the image double speed playback apparatus 100 receives an original image (1,800 frames) of one minute of video having 30 frames per second (FPS) per second. In step S210, the original image refers to an image photographed by a general photographing apparatus, not a fixed type camera, without a separate stabilizing apparatus.

The image-speed playback apparatus 100 extracts feature points of each of the frames (1,800 frames) included in the original image. The image data playback device 100 performs matching between adjacent frames using the minutiae of each frame (S220). For example, the image double speed playback apparatus 100 may be configured to select one of the adjacent frames ('first to second frame', 'second to third frame', 'third to fourth frame' Frame '). The image quality playback apparatus 100 calculates a correlation matrix between adjacent frames (first to second frame, second to third frame, third to fourth frame, ..., 1,799 to 1,800 th frame) . Correlation matrices between adjacent frames need not be recalculated once calculated. In step S220, the image quality doubling playback apparatus 100 calculates correlation matrices by matching feature points of each frame of the original image with each other. A concrete operation method of calculating the correlation matrix will be described below with reference to FIG.

The image data playback device 100 selects a frame having the highest relevance among the matched frames as a path frame (S230). In step S230, the image quality playback apparatus 100 calculates a cost matrix in which associations among frames are compared based on a correlation matrix. The image-quality playback apparatus 100 selects a frame having the highest correlation among frames as a path frame by using a cost matrix. For example, when the double-speed reproduction is set to 16-times speed, the image-type double-speed playback apparatus 100 compares the first frame (reference frame) with the second through the 32nd frame (double speed x 2) The highest frame is selected as the route frame.

The video image playback device 100 calculates a cost matrix to check the association between the first frame (reference frame) and the second to 32 < th > frame (double speed x 2). For example, when the double speed reproduction is set to the 16x speed, the image double speed playback apparatus 100 calculates a cost matrix for comparing the first frame (reference frame) to the second to the 32nd frame (twice the speed).

First, the image data playback device 100 checks the amount of change between the 'first frame' and the 'second frame', checks the amount of change of the 'first frame' and the 'third frame' Check the amount of change of the 'fourth frame', and check the amount of change of the 'first frame' and the '32nd frame'. Then, the image data playback device 100 selects the frame having the minimum value among the variation amounts as the path frame having the highest correlation with the 'first frame'. A specific method of calculating the cost matrix will be described below with reference to FIG.

Since the image multiplier 100 calculates all of the correlation matrices between adjacent frames in step S220, only the cost matrix corresponding to the speed selected by the user is calculated in step S230, and the optimum path made of the frames having high relevance is easily Can be confirmed.

In order to speed up the computation speed, the image-based double-speed playback apparatus 100 may apply the chain rule to the correlation matrix between neighboring frames. For example, when it is desired to check how much the '1 st frame' and the '10 th frame' are different from each other, the image data playback apparatus 100 may calculate the correlation matrix of the 1 st and 2 nd frames, 1 < th > frame and the 10 < th > frame by applying a 'chaining principle' which multiplies all of the 'correlation matrix' of ' 'Can be calculated quickly.

Since the correlation matrix between neighboring frames does not need to be recalculated once calculated, only the cost matrix is re-calculated according to the speed set by the user, and the computation processing can be performed quickly each time the user controls the speed.

The image data rate reproduction apparatus 100 stabilizes the calculated optimal path based on the basic frame and the path frame (S240).

The image double speed playback apparatus 100 renders an image based on the stabilized path (S250). In step S250, the image-double-speed playback apparatus 100 may render an image for double-speed playback based on the path frame without passing through step S240.

Although it is described in Fig. 2 that steps S210 to S250 are sequentially executed, the present invention is not limited thereto. In other words, Fig. 2 is not limited to the time-series order, as it would be applicable to changing and executing the steps described in Fig. 2 or executing one or more steps in parallel.

As described above, the overall operation of the image double speed playback according to the embodiment described in FIG. 2 can be implemented by a program and recorded on a computer-readable recording medium. The program for realizing the overall operation of the image double speed reproduction according to the present embodiment is recorded, and the computer readable recording medium includes all kinds of recording devices for storing data that can be read by the computer system.

FIGS. 3A to 3C are diagrams for explaining a comparison between a general camera path and a stabilized camera path according to image double speed playback.

3A is a diagram showing a camera path when an image is photographed by a general photographing method. In the case of shooting everyday life, extreme situations, sports which can not be photographed, and the like, with a photographing device such as a handheld camera or a kohf hero, the image is too long, complex, and monotonous. In other words, a video shot with a common imaging device without professional assistant equipment has a shaky camera path, as shown in FIG. 3A, because there is no professional editing.

FIG. 3B is a view showing a camera path photographed in a general photographing mode by a single smooth camera path (Smooth Camera Path). Generally, when stabilizing an unstable camera path as shown in FIG. 3A, a smooth camera path is generated as shown in FIG. 3B.

3C is a diagram showing an example in which the shaky path is changed to have a stabilized hardness. The image double speed playback apparatus 100 according to the present embodiment does not simply sample a frame corresponding to a preset speed and reproduce it (image double speed playback), but performs matching by matching feature points of each frame of the original image with each other After calculating the correlation matrix and calculating the cost matrix in which the associations among the frames are compared based on the correlation matrix, only the path frame with the highest correlation among the frames is selected based on the cost matrix, , A smooth camera path is changed. For example, in general, it is possible to perform time stabilization (e.g., 10 times speed) after performing video stabilization on an image or perform image stabilization after rapidly reproducing an image at a predetermined speed. However, since unstable camera paths are largely stabilized Do not. However, in the present embodiment, instead of sampling only the frame corresponding to the user's input data, the cost matrix is calculated using the calculated correlation matrix, and then the cost matrix is calculated by considering the playback speed and the frame- Since the frame is selected at intervals, unstable camera paths are largely stabilized. In the case of this embodiment, the image double speed playback apparatus 100 does not need to perform a separate calibration to stabilize the unstable camera path or to check the focal length or the like of the camera.

FIG. 4 is a flowchart for explaining a frame matching process according to the present embodiment.

The image double speed playback apparatus 100 extracts minutiae for each of all the frames (1,800 frames) included in the original image (S410). For example, in step S410, the image double speed playback apparatus 100 can extract '500' feature points in the first frame, extract '495 feature points' in the second frame, The feature points of '497' can be extracted, and the feature points of '490' can be extracted from the 1,800th frame.

The image data playback device 100 confirms a descriptor for each minutiae of each frame (S420). The descriptor includes a kind of attribute information or feature information for the feature point. For example, in step S420, the image double-speed playback apparatus 100 confirms 500 descriptors of '500' feature points extracted from the first frame and extracts '500' descriptors of '495' feature points extracted from the second frame 497 descriptors of 497 feature points extracted from the third frame, and 497 feature descriptors of 490 feature points extracted from the 1,800th frame. Identify the 490 descriptors.

The video image playback apparatus 100 calculates a correlation matrix between neighboring frames by matching the descriptors among neighboring frames of all the frames in a method of comparing them (S430). For example, in step S430, the video playback apparatus 100 transmits the first frame and the second frame, which are adjacent frames, the second through third frames, the third through fourth frames, And '1,799 ~ 1,800th frame', and calculates and stores a correlation matrix for each frame. The image-quality playback apparatus 100 calculates and stores '1, 799' correlation matrices as a result of performing Step S430 for '1,800 frames'.

In step S430, the image-wattening reproduction apparatus 100 performs an algorithm (RANSAC (RANdom SAmple Consensu) algorithm) for selecting a model having the maximum consensus from the random sample data when matching the descriptors between adjacent frames in a matching manner And selectively removes an outlier from the image.

The correlation matrix is a 3x3 matrix and includes up to eight change value parameters excluding one in a 3x3 matrix. Correlation matrix is a 3 × 3 matrix, a change amount parameter between adjacent frames, x-axis movement amount (t _x) parameters, y-axis movement amount (t _y) parameters, the angle of rotation (θ) parameters, size parameters, and the aspect ratio parameter, shear parameters And a projection parameter.

Hereinafter, the correlation matrix H (t) will be described.

When the correlation matrix H (t) is composed of a translation motion model including 2 DOF (Degree of Freedom), the x-axis movement amount (t _x ) parameter and the y-axis movement amount (t _y ) .

Correlation matrix (H (t)) a geometric motion model comprising a 3 DOF (Euclidean Motion Model), x-axis movement amount (t _x) parameters, y-axis movement amount (t _y) parameters, the rotational angle (θ) have been made to Parameters.

Correlation matrix (H (t)) is the similarity motion model comprising a 4 DOF (Similarity Motion Model), x-axis movement amount (t _x) parameters, y-axis movement amount (t _y) parameters, the rotational angle (θ) have been made to Parameters, and size parameters.

If made of a correlation matrix (H (t)) the affine motion model including a 6 DOF (Affine Motion Model), x-axis movement amount (t _x) parameters, y-axis movement amount (t _y) parameters, the rotation angle (θ ) Parameters, size parameters, aspect ratio parameters, and shear parameters. Parallel lines are preserved for affine motion models.

Correlation matrix (H (t)) is the projection motion model (Perspective Motion Model), x-axis movement amount (t _x) parameters, y-axis movement amount (t _y) parameters, the rotational angle (θ) have been made by including a 8 DOF Parameters, size parameters, aspect ratio parameters, shear parameters, and projective parameters. Parallel lines are not preserved for the projective motion model.

4, steps S410 to S430 are sequentially executed. However, the present invention is not limited to this. In other words, Fig. 4 is not limited to the time-series order, since it would be applicable to changing or executing the steps described in Fig. 4 or executing one or more steps in parallel.

As described above, the frame matching process according to the present embodiment described in FIG. 4 can be implemented as a program and recorded in a computer-readable recording medium. A program for implementing the frame matching process according to the present embodiment is recorded, and a computer-readable recording medium includes all kinds of recording devices for storing data that can be read by a computer system.

5 is a flowchart illustrating a frame selection process according to the present embodiment.

The image-quality playback apparatus 100 calculates associations (C _m ) between frames using a correlation matrix between frames based on the correlation matrix (S510). In step S510, the image-quality-of-view playback apparatus 100 determines whether or not the correlation between the first frame and the subsequent frames based on the correlation matrix between the frames is the change value obtained by comparing the correlation matrix up to the frame having twice the maximum playback speed (C _m ). In general, the maximum reproduction speed has a value of 16 or 32. [

The relation between the frames is calculated by comparing the 'first frame' and the 'second to 32nd frame' when the image double speed playback apparatus 100 reproduces an image at 16 × speed.

In order to confirm the association between the 'first frame' and the 'tenth frame' in general, it is necessary to combine all the feature points and descriptor combinations extracted from the 'tenth frame' The matrices must be matched to calculate the matrices in a comparative manner.

However, in this embodiment, the cost matrix corresponding to the speed input by the user can be quickly calculated by using the correlation matrix of '1,799' previously calculated by performing step S430.

In step S510, the video double-speed playback apparatus 100 may apply a chaining rule to a correlation matrix between neighboring frames in order to increase the computation speed. For example, when it is desired to check how much the '1 st frame' and the '10 th frame' are different from each other, the image data playback apparatus 100 may calculate the correlation matrix of the 1 st and 2 nd frames, The correlation matrix for multiplying the correlation matrix of the ninth to tenth frames by the correlation law of the correlation matrix of the third to fourth frames, The difference can be calculated quickly. Since the correlation matrix between neighboring frames does not need to be recalculated once calculated, only the cost matrix is re-calculated according to the speed set by the user, and the computation processing can be performed quickly each time the user controls the speed.

In the case of calculation using a general frame comparison method, a correlation matrix is calculated by a method of comparing 'first to second frame', and a correlation matrix is calculated by matching the 'first to third frame' . In the case of calculating the correlation matrix in the above-described manner, the calculation amount becomes large. Therefore, the correlation matrix of '57, 568' should be calculated instead of calculating the correlation matrix of '1,799' as in the present embodiment. The image quality playback apparatus 100 may calculate the cost matrix of each frame by multiplying the correlation matrix of the first frame by the correlation matrix of the subsequent frame.

In step S510, a method for calculating the correlation (C _m ) between frames in the image-by-picture playback apparatus 100 is as shown in the following equation (1).

C _r: i image shift amount between the first frame and the j th frame, T: i-th frame and the correlation between the j-th frame related functions, i: i-th frame index (Index), j: j-th frame index, C _0: x ₀ is the center x coordinate value of the image, y ₀ is the center y coordinate value of the image, C _{m is} the correlation between the frames, and the amount of overlap between the image and the moving image is A value that determines the total movement level in terms of the amount, and Γ is the maximum threshold. For example, when the value of C _r (i, j) is larger than τ _c , the value of Γ is used as the maximum value. τ _c : Threshold

Generally, frames that have a high correlation between frames must be computed for all feature points in order to use the average value of geometric re-projection errors, and a chain rule for fast operations is applied can not do. Since the image double speed playback apparatus 100 according to the present embodiment uses the geometric dx and dy magnitude of the motion matrix as feature values, it is possible to perform fast operation by the 'chain rule'.

Double-speed video player 100 calculates a speed satisfying value (C _s) to satisfy the speed the user has preset for each frame (S520). In step S520, based on the i-th frame index i, the j-th frame index j, the user-preset double speed playback value v, and the threshold value? _S , (C _s ) for satisfying the preset speed. More specifically, the image double speed playback apparatus 100 calculates a double speed satisfaction value (C _s ) for satisfying the speed set by the user using the formula (2).

In step S520, when the image is reproduced at the 16x speed, the image quality doubling playback apparatus 100 calculates a double speed satisfaction value ( _Cs ) for satisfying the 16x speed. For example, if only the degree of similarity in the degree of skipping between frames is considered, it is difficult for the user to satisfy the desired speed. Therefore, the total cost is calculated using [Equation 2]. The combined cost function of the cost matrix computed by the image-speed-double playback apparatus 100 is expressed by Equation (2).

i: i-th frame index, j: j-th frame index, v: user-defined double speed playback value. For example, values of 2, 4, 8, and 16 are input. C _{m is} a value for determining the total movement level between the i-th image and the j-th image in terms of the amount of image movement and the amount of overlapping, and λ _s is a weight for keeping the speed, and a value between 0.0 and 1.0 is mainly used . C _s : Speed correction value to satisfy the speed set by the user. For example, when the user's desired speed is 8, the value of C _s is the smallest value when the difference between i and j is 8. The reason why the value of C _s is required is that the matching degree of the images is the highest in the nearest frame, and the user can not satisfy the desired speed if only the nearest frame is selected. C (i, j, v): The value used in the final cost matrix. The value of C (i, j, v) is a value including the degree of matching of the image and the frame jump (skip) level. The value of C (i, j, v) is used to find the value of i, j so as to minimize the value of C (i, j, v) τ _{s is a} threshold value, which is a preset upper limit value because it affects the determination of the frame if the weight value of the frame playback speed is too large. τ _s can generally be a value of 200, but specific values can be changed depending on the setting.

Video speed reproduction apparatus 100 is a combination of the speed values calculated at the correlation satisfies (C _m) and step S520 between the frames calculated at step S510 (C _s) and calculates the cost matrix. The image-speed-playback apparatus 100 calculates an optimal path consisting only of the path frame by repeating the process of selecting the frame having the minimum value among the cost matrix as the path frame until the last frame (S530). Double-speed image reproducing apparatus 100 in step S530 is a combination of a speed calculated from a value satisfying relationship (C _m), step S520 between the frames calculated at step S510 (C _s) and calculates the cost matrix. More specifically, the image-speed-double playback apparatus 100 calculates the cost matrix C _{(i, j, v)} using the equation (2).

The image data playback device 100 selects a frame having the minimum value among the cost matrices calculated in step S530 as a path frame. The image-speed-double playback apparatus 100 calculates an optimal path only with the path frame. For example, when the double speed reproduction is set to 16x speed, the image double speed playback apparatus 100 sets the frame inter-frame relevance (i.e., the frame rate of the first frame to the second frame to the 32nd frame C _m ).

First, the image data playback device 100 checks the amount of change between the 'first frame' and the 'second frame', checks the amount of change of the 'first frame' and the 'third frame' Check the amount of change of the 'fourth frame', and check the amount of change of the 'first frame' and the '32nd frame'.

The video-image playback device 100 calculates a double-speed satisfaction value (C _s ) for satisfying the speed set by the user for each frame. Then, the image-speed-increase playback apparatus 100 calculates inter-frame relevance (C _m ) for comparing the first frame with the second to the 32nd frame (twice the speed), the weight (λ _s ) And calculates a cost matrix based on a doubling satisfaction value (C _s ) for satisfying a predetermined double speed.

The image-quality playback apparatus 100 selects the frame having the minimum value in the cost matrix in which the first frame is compared with the second through the 32nd frames as the route frame having the highest correlation with the 'first frame'.

The image-speed playback apparatus 100 programs the optimal path calculated in operation 530 (Greedy Programming). A programming example is as follows.

The image double speed playback apparatus 100 proceeds to the optimization path calculation based on the cost matrix calculated by using Equation (2).

The image quality playback apparatus 100 finds an optimization path in a greedy manner unlike the DTW (Dynamic Time Warping) scheme. Due to the nature of greedy algorithms, it is not possible to calculate the perfect optimization value, but it is possible to find a value close to the optimization with high probability.

Although it is described in Fig. 5 that steps S510 to S530 are sequentially executed, the present invention is not limited thereto. In other words, Fig. 5 is not limited to a time series order, since it would be applicable to changing and executing the steps described in Fig. 5 or executing one or more steps in parallel.

As described above, the frame selection process according to the present embodiment described in FIG. 5 can be implemented as a program and recorded in a computer-readable recording medium. A program for implementing the frame selection process according to the present embodiment is recorded, and a computer-readable recording medium includes all kinds of recording devices for storing data that can be read by a computer system.

6 is a flowchart for explaining a path stabilization process according to the present embodiment.

The image data playback device 100 receives some selected path frames (path frames selected at variable intervals) of all the frames from the frame selecting unit 130 (S610).

The image-type double speed playback apparatus 100 generates a frame-by-frame path for each input frame (S620). In step S620, the image-double-speed playback apparatus 100 decomposes the frame based on the motion model of the correlation matrix for the frame to generate a frame-by-frame path. Correlation matrix to contain at least two or more parameters of the x-axis shift amount (t _x) parameters, y-axis movement amount (t _y) parameters, the angle of rotation (θ) parameters, size parameters, and the aspect ratio parameter, the front end parameters and the projection parameter, The image double speed playback apparatus 100 decomposes the number of path frames corresponding to the number of parameters included in the correlation matrix to generate a frame-by-frame path.

The image-speed-increase playback apparatus 100 extracts the minutiae between adjacent frames of all the frames of the input original image, and confirms the descriptor per minutiae. The image quality playback apparatus 100 calculates a correlation matrix between all the frames in the original image using a RANSAC algorithm for matching (matching and matching the corresponding points of the respective feature points) in a manner of comparing descriptors among adjacent frames.

if the correlation matrix between the t-th frame and the t-th frame is defined as H (t-1), then the image-for-picture double-speed playback apparatus 100 sets the camera path C ( t) is calculated using Equation (3).

C (t): Camera path of 't-th frame'

C (t-1): Camera path of the (t-1) th frame

H (t-1): a correlation matrix between the 't-1 th frame' and the 't th frame'

Total Path: Actually, a path is a set of digital values with one value per frame, not a continuous value. In other words, the entire path from the first frame to the t-th frame, which is represented by a 3 × 3 matrix and is a whole frame, of the camera path C (t) of the 't th frame' (0, 0), C (1), ..., C (t), for example, if the x- , (1,3) elements of C (t) in C (1) are represented by graphs. However, in the camera path C (t) The number depends on the 'motion model' used to calculate the correlation matrix.

For example, when using a transformed motion model that includes 2 DOFs when calculating the correlation matrix, two paths are created. In calculating the correlation matrix, four paths are generated when using a similarity motion model including four DOFs, and six paths are generated when an affine motion model including six DOFs is used. In the case of the projective motion including eight DOFs When using the model, eight paths are created.

The image-quality-speed playback apparatus 100 generates optimization paths that have been optimized for each frame-by-frame path (S630). In step S630, the image quality doubling playback apparatus 100 calculates a smooth path P (t), which is a result of applying line fitting for each frame or applying a smoothing filter to optimize each frame.

The image data playback apparatus 100 reconstructs the optimization paths into a matrix and generates a stabilized path (S640). In step S640, the image quality doubling playback apparatus 100 calculates a weighting matrix in which a smooth path optimized for each frame-specific path is restored into a matrix.

When the image-speed playback apparatus 100 applies the optimization method to the camera path C (t) calculated using Equation (3), a smooth path P (t) is obtained as shown in FIG. 7A .

The image-quality playback apparatus 100 finally applies the wobbing matrix B (t) calculated by the equation (4) to each frame (the path frame received from the frame selector 130) .

C (t), P (t), and B (t) are all 3 × 3 matrices.

(T): Smooth path of 't-th frame', C (t): 't (t): Warping matrix to be applied to the original image at rendering in' Th frame ', C ^-1 (t): Inverse of the camera path of the' t-th frame '

The warping matrix B (t) can be expressed as:

B (t) is the Wapping matrix of the '0th frame', B (1) is the Wapping matrix of the 1st frame, B (t- : soft path of '0th frame', P (1): smooth path of '1st frame', P (t-1) C ¹ (0): Inverse of the camera path of the '0 th frame', C ^-1 (1): '1 st frame' C ^-1 (t-1): Inverse of the camera path of the 't-1 th frame', C ^-1 (t): Inverse of the camera path of the 't th frame'

It is possible to calculate a smooth path P (t) using [Equation 4] in a general environment. However, if the interval between frames for which motion is to be sought for the double speed reproduction is large as in the present embodiment, An error will inevitably occur in the correlation matrix H (t), which is the estimated value (Motion Estimation).

Since the error in the correlation matrix H (t) is a structure in which the error is continuously multiplied by the cumulative error as shown in Equation (3), the camera path C (t) do. The calculated smooth path P (t) also contains a large number of errors because the smooth path P (t) is calculated using the camera path C (t) that has deviated due to the error. A large number of holes are generated in the image as shown in FIG. 7B when the final weighting matrix B (t) obtained by the smooth path P (t) including the error is applied to the image.

The image-quality playback apparatus 100 according to the present embodiment performs frame-by-frame path optimization to resolve holes generated in an image. The image quality doubling playback apparatus 100 according to the present embodiment calculates a path for each frame to solve an error accumulation problem of a global path generated in a general image processing method.

The image data playback apparatus 100 decodes the correlation matrix for each path frame input from the frame selection unit 130, optimizes and then compiles it to calculate a warping matrix, So that errors do not accumulate.

In other words, in the general image processing method, if one smooth path is obtained by using one optimum path (global path), the image-rearranging playback apparatus 100 according to the present embodiment sets N paths And calculates a smooth path P (t) using one representative value per path. Since N paths are generated for each of the N frames, when M motion models are used, a total of M x N = MN paths are generated in this embodiment. A general image processing method generates only M paths.

Let r be the custom window size (set how much you want to see back and forth, r will be more stabilized but more likely to cause holes, you can use 15 in actual implementation) The camera path Path _{t, r} can be defined as Equation (6).

Path _{t, r:} in each frame the camera path, C (t): the "t-th frame, the camera path, C (t-1): camera path" t-1 th frame ", C (t-2 ): Camera path of 't-2 nd frame', C (tr): Camera path of 'tr th frame', r: Size of user specified window, H ^-1 (t) "the inverse matrix of the correlation relationship between the ^{mattress, H -1 (t + 1)} : '+1 th frame t + 1 the inverse of the first frame, and" t + 2 the correlation between the second frame, between the mattress, H ^-1 (t + r -1: the inverse matrix of the correlation mattress, H (t + r-1): 't + r-1' th frame and t + r- (t + 1): correlation mattress between 't + 1' th frame and 't + 2 th frame', I _{3 × 3} : 3 × 3 Identity Matrix,

The image quality doubling playback apparatus 100 calculates the camera path C (t) of the 't-th frame' using Equation (7) using the frame-specific path optimization method.

In the image-aspect-fast playback apparatus 100, a maximum of eight paths in a 3 × 3 matrix can be applied to different optimization methods according to a motion estimation model. Since the method described below is only an embodiment, the number of actual paths and the optimization method can be changed.

When it is assumed that the image-speed playback apparatus 100 uses a geometric motion model including three DOFs when estimating the correlation matrix, the image-double-speed playback apparatus 100 sets the x-axis movement amount (t _x ) the y-axis movement amount (t _y ) parameter, and the rotation angle (?) parameter can be decomposed using the following equation (8).

Since the image is judged to be unstable by the user as the change in the rotation angle? Parameter is greater, the image double speed playback apparatus 100 sets the L1-absolute value? 1 for one path corresponding to the rotation angle? Apply smooth path (P (t)) by applying line fitting (L1-Norm Line Fitting).

However, in order to minimize the error caused by the incorrectly calculated camera path C (t), the image-speed-double playback apparatus 100 calculates n points (n < 2 × r + 1) is randomly selected, and an IRLS (Iteratively Reweight Least Squares) method is used.

In addition, the image double speed playback apparatus 100 multiplies the x position (Position) entering the line filtering by an appropriate interval (for example, 5) even if the difference of the actual frame is 1, thereby eliminating the influence due to the fine x interval .

In addition, x-axis movement amount (t _x) parameters, y-axis movement amount (t _y) parameters are due to its variation in size compared to the rotation (Rotation), imaging speed playback device 100 is a four Bits key smoothing filter-Golay filter (Savitzky-Golay Filter).

If the image-speed-increase playback apparatus 100 obtains the smooth path P (t) through optimization, the weighting matrix B (t) to be finally applied to the resultant image can be calculated using Equation (9).

, I_{3× 3}: 3×3 항등행렬 =

, P(t): ‘t번째 프레임’의 부드러운 경로, B(t): ‘t번째 프레임’에서 렌더링시 원본 영상에 적용할 와핑 매트릭스C (t): the camera path for the "t th frame ^{', C -1 (t):} ' t -th frame, the inverse matrix of the camera _{^{path, I 3 × 3 - 1:}} 3 × 3 identity matrix in the inverse matrix =

, I _{3 × 3} : 3 × 3 identity matrix =

, T (t): smooth path of 't-th frame', B (t)

In the video double speed playback apparatus 100, since a corner point extraction and an optical flow matching method, which are often used in general image processing methods, are not suitable in an environment in which motion is large during double speed reproduction In the present embodiment, the feature points of each frame are extracted quickly using an ORB (Oriented FAST and Rotated BRIEF) algorithm, and the neighboring frames are compared with each other by checking the descriptors according to the feature points.

Although it is described in Fig. 6 that steps S610 to S640 are sequentially executed, the present invention is not limited thereto. In other words, Fig. 6 is not limited to the time-series order, since it would be applicable to changing or executing the steps described in Fig. 6 or executing one or more steps in parallel.

As described above, the path stabilization process according to the present embodiment described in FIG. 6 can be implemented by a program and recorded on a computer-readable recording medium. A program for implementing the path stabilization process according to the present embodiment is recorded, and a computer-readable recording medium includes all kinds of recording devices for storing data that can be read by a computer system.

8 is a flowchart illustrating a rendering process according to the present embodiment.

The image data playback apparatus 100 receives the stabilization path received from the path stabilization unit 140 and performs decoding (S810).

The image double speed playback apparatus 100 renders an image for double speed playback based on the decoded stabilized camera path (S820). In step S820, the image-quality playback apparatus 100 multiplies all the pixel values of the path frame by the wavelet matrix B (t), and renders the image so as to move the pixel values of the frame to the output coordinates.

In step S820, the image playback apparatus 100 multiplies the corresponding frame by the smooth path P (t) having the same value as the weighting matrix B (t) using Equation (9) .

The image multiplier 100 multiplies all pixel values corresponding to the path frame by a 3 × 3 matrix of smooth paths P (t) and moves the pixel values of the frame to the output coordinates to output Video is output.

Step S820 is expressed by the following equation (10). Since we have confirmed that the smooth path P (t) and the wavelet matrix B (t) represent the same 3 × 3 matrix when generating a frame-by-frame path using Equation 9, Multiplying the warping matrix B (t) by the soft path P (t) to obtain the same result.

The image-aspect-fast playback apparatus 100 encodes a frame to which the stabilization path is applied (S830).

Although it is described in Fig. 8 that steps S810 to S830 are sequentially executed, it is not limited thereto. In other words, Fig. 8 is not limited to the time-series order, since it would be applicable to changing and executing the steps described in Fig. 8 or executing one or more steps in parallel.

As described above, the rendering process according to the present embodiment described in FIG. 8 can be implemented as a program and recorded in a computer-readable recording medium. A program for implementing the rendering process according to the present embodiment is recorded, and a computer-readable recording medium includes all kinds of recording devices for storing data that can be read by a computer system.

9 is an exemplary diagram for explaining a cost matrix according to the present embodiment.

The image-aspect-ratio playback apparatus 100 calculates motion information of all frames in the input original image. The general motion information calculation method is a method of calculating motion information using the RANSAC algorithm in the HARRIS + BRIEF method, but in the present embodiment, the motion information is calculated using the ORB (Oriented FAST and Rotated BRIEF) feature points and the RANSAC algorithm .

The image-speed-double playback apparatus 100 calculates a cost matrix based on the calculated motion information. The image quality doubling reproduction apparatus 100 performs calculation only on a frame required in a greedy scheme rather than a conventional dynamic programming scheme in cost matrix calculation. The image-speed-playback device 100 generates a stabilized result image based on the obtained path information.

The video image playback device 100 receives an original image (1,800 frames) of one minute of video having 30 frames per second (FPS) per second. The image-speed-increase playback apparatus 100 extracts feature points for all the frames (1,800 frames) included in the original image. The video image reproducing apparatus 100 can extract '500' feature points from the first frame, extract '495 feature points from the second frame, extract' 497 'images from the third frame, , And extracts feature points of '490' from the 1,800th frame.

The image data playback device 100 confirms the descriptors for each of the minutiae for each frame. The image double speed playback apparatus 100 confirms 500 descriptors for the 500 feature points extracted in the first frame and 495 descriptors for the 495 feature points extracted in the second frame 497 pieces of descriptors were extracted from the 497 pieces of feature points extracted from the 3rd frame and ... 490 pieces of descriptors were extracted from the 490 pieces of feature points extracted from the 1,800th frame Check.

The image data playback apparatus 100 matches the descriptors among neighboring frames of all the frames in a manner of comparing the neighboring frames, and outputs the neighboring frames ('1 st to 2 nd frame', '2 nd to 3 rd frame', '3 rd to 4 th frame' '1,799 to 1,800 th frame'). The image quality playback apparatus 100 calculates and stores '1, 799' correlation matrices by matching descriptors among adjacent frames among 1,800 frames. Correlation matrices do not need to be recalculated once they are calculated and stored. Then, according to the speed set by the user, the cost matrix can be immediately calculated according to the speed, and the image can be rendered. In the general scheme, the entire cost matrix for all the frames has to be calculated, but the present invention does not need to calculate the entire cost matrix.

Hereinafter, the cost matrix calculation method will be described.

As shown in FIG. 9, the image double-speed playback apparatus 100 determines a correlation (hereinafter, referred to as &quot; first frame &quot;) from a correlation matrix of a first frame to a frame The correlation between the frames (cost matrix) is calculated by the change values comparing the relation matrix.

As shown in FIG. 9, the image double-speed playback apparatus 100 may apply the chain rule to the correlation matrix between neighboring frames in order to quickly process a computation amount. For example, when it is desired to check how much the '1 st frame' and the '15 th frame' are different from each other, the image-wise playback apparatus 100 may calculate the correlation matrix of the 1 st and 2 nd frames, The correlation matrix for multiplying the correlation matrix of the 14th to the 15th frame is applied to calculate the correlation matrix of the first frame and the 15th frame The difference can be calculated quickly.

The image double speed playback apparatus 100 determines whether or not the correlation matrix of the '1 st frame' is used as a reference and the correlation matrix of the frame whose frame is twice the playback speed ('2 to 32 th frame' The frame having the minimum value ('15th frame') is recognized as a frame having the highest correlation with the '1 st frame', and is selected as a path frame.

Then, as shown in FIG. 9, the video double-speed playback apparatus 100 refers to the correlation matrix of the path frame ('15th frame'), ') Are calculated as cost matrices.

As shown in FIG. 9, the image double-speed playback apparatus 100 may apply the chain rule to the correlation matrix between neighboring frames in order to increase the computation speed. For example, when it is desired to check how much the '15th frame' and the '23rd frame' are different from each other, the image-rearrangement playback apparatus 100 may calculate the correlation matrix of the 15th to 16th frames, The correlation matrix for multiplying the correlation matrix of the 23rd to the 24th frame is applied to calculate the correlation matrix of the 15th frame and the 23rd frame The difference can be calculated quickly.

The image data playback apparatus 100 compares a correlation matrix from a correlation matrix of a path frame ('15th frame') to a frame ('16th to 47th frame') in which the frame is twice the playback speed ('23rd frame') having the minimum value of one change value is recognized as a frame having the highest correlation with the first selected route frame ('15th frame') in the cost matrix, and is selected as the second route frame.

The image double speed playback apparatus 100 according to the present embodiment is not a method of calculating a cost matrix for all frames but a method of calculating a cost matrix by comparing correlation matrices twice as fast as a selected frame from a selected frame. In the general method, when the 16-times speed is assumed, the process of comparing the 'second frame' and the 'third to 32nd frame' after comparing the 'first frame' and the ' Should be. That is, all comparisons must be performed. However, the image double-speed playback apparatus 100 according to the present embodiment compares the 'first frame' with the 'second to 32nd frame' (twice the speed) to calculate a cost matrix. Thereafter, the video playback apparatus 100 selects a frame having a high correlation with the 'first frame' as a path frame, and thereafter, there is a difference in that the computation amount can be greatly reduced in calculating the cost matrix of the selected path frame and the remaining frames do. For example, even if it is assumed that 1 MS (Milli-Second) is required to compare the amount of change between frames, performing 32 times requires 32 MSs, so that a large amount of resources are consumed for calculating the cost matrix, .

FIGS. 10A and 10B are diagrams for explaining a comparison between frame selection during normal speed reproduction and frame selection according to the present embodiment.

10A is a diagram for explaining comparison of selected frames during double speed reproduction of an image. The dotted line shown in FIG. 10A is a graph reconstructed based on a value obtained by observing how much the camera moved in the 'y-axis' direction when an actual person is running. Since the actual person is running, the 'y axis' shows motion in the pixel range of '-100 to 100' within '50 frames'. In other words, it can be seen that the image is greatly shaken by the dotted line graph.

The circular point (O) shown in FIG. 10A is a graph showing a case where a frame is selected at a normal speed. For example, in the case of time-lapse of '8x speed', in the case of selecting '16 frames', '24 frames', '32 frames' or the like, even if the frame is stabilized with the selected circle as shown in FIG. , The stabilization result is not good.

A star point (*) shown in FIG. 10A is a graph showing a case where a frame is selected in a manner according to the present embodiment. Even when the frame is selected for the double speed reproduction as in the star point (*) shown in FIG. 10A, even if the frame is selected as '8x', one frame is selected as '15 frames' So that a high frame is selected. That is, it is possible to select a frame positioned after the '8x speed', or to select a frame with a high correlation among frames by selecting a frame before the '8x speed'. In other words, when the frame is skipped, the video double-speed playback apparatus 100 according to the present embodiment does not select the frames at regular speed intervals, but selects frames variably.

10A is a result of selecting a frame at a normal speed and a star point (*) on a solid line graph is a result of selecting a frame at a variable interval.

In the case of comparing the graph with circular points shown in FIG. 10B and the graph with star points, it can be seen that stability is high when a frame is selected at variable intervals. When estimating a three-dimensional camera path, accuracy can be improved, but the speed is too slow due to too much computation. In this embodiment, double speed reproduction is implemented in a two-dimensional manner, but the calculation amount can be reduced to enable quick calculation.

In the present embodiment, the video double speed playback apparatus 100 calculates a cost matrix as shown in FIG. 10B in order to select a frame having a high correlation among frames. For example, in the case of a 16-times speed, the image-speed-doubling playback apparatus 100 compares the first frame with the second to the 32-th frame (twice the speed). The image-speed-up playback apparatus 100 calculates the amount of change with the frame up to twice the double speed based on the correlation matrix between adjacent frames calculated in advance as a cost matrix. Then, the image data playback device 100 confirms the amount of change of the 'first frame' and the 'second frame' using the cost matrix, checks the amount of change of the 'first frame' and the 'third frame' 1 &quot; and &quot; 4th frame &quot;, and confirms the amount of change of the 'first frame' and the '32nd frame'. At this time, the amount of change of each frame is already calculated by a correlation matrix between adjacent frames. That is, as a result of checking the first frame and the second through the 32nd frames of the image data playback apparatus 100, the frame having the smallest change amount is selected as the frame having the highest correlation.

As shown in FIG. 10B, when comparing a circular point graph when a frame is selected at a normal speed and a star point graph selected at a variable interval, the graph of the hatched line indicates' 16 Quot ;, it is possible to select a frame located before the &quot; 16x speed &quot;, or to select a frame after a longer time.

It can be confirmed that a stable result with little change in the y-axis movement amount is generated by only selecting the optimum frame as shown by the dotted line in FIG. 10A. That is, when selecting a frame for double speed reproduction, a frame having a high correlation can be selected by selecting a frame at a variable interval, and as a result, a stable result can be output at the time of double speed reproduction.

The foregoing description is merely illustrative of the technical idea of the present embodiment, and various modifications and changes may be made to those skilled in the art without departing from the essential characteristics of the embodiments. Therefore, the present embodiments are to be construed as illustrative rather than restrictive, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be construed according to the following claims, and all technical ideas within the scope of equivalents thereof should be construed as being included in the scope of the present invention.

As described above, the present embodiment is applied to the field of scene analysis in the field of computer graphics, and a useful invention which generates an effect of stabilizing images reproduced after being optimized by calculating a path for each frame, to be.

100:
110: input unit 120: frame matching unit
130: frame selecting unit 140: path stabilizing unit
150:
142: frame input unit 144:
146: Path optimizing unit 148: Path optimizing unit

Claims (8)

A method for image stabilization by a video double speed playback apparatus,
An input process of receiving some selected frames among frames of an input image;
A frame-by-frame path generation step of generating a frame-by-frame camera path for each of the selected frames by decomposing the frame based on a motion model of a correlation matrix for the selected frame;
An optimization process of optimizing paths by applying line fitting or applying a smoothing filter to each camera path for each frame;
A stabilization process for generating a stabilization path after the optimizing paths are restored to a matrix; And
And outputting a stabilized result image by applying the stabilization path to the selected frame
The image stabilization method comprising: The method according to claim 1,
The path generation process includes:
Wherein the selected camera frame is generated by decomposing the selected frame based on a motion model of a correlation matrix for the selected frame. 3. The method of claim 2,
The correlation matrix is the x-axis shift amount (t _x) parameters, y-axis movement amount (t _y) parameters, the rotational angle (Angle) (θ) parameters, the size (Scale) parameter, the aspect ratio (Aspect Ratio) parameters, the front end (Shear) Parameter and a perspective parameter, the parameter including at least two parameters,
Wherein the path generation step decomposes the selected frames into a number corresponding to the number of parameters included in the correlation matrix to generate the camera path for each frame. The method according to claim 1,
The optimization process includes:
And calculating a smooth path P (t) as a result of performing the optimization. 5. The method of claim 4,
In the stabilization process,
And a warping matrix (B (t)) obtained by restoring the smooth path P (t) optimized for each frame-by-frame camera path into a matrix is calculated. 6. The method of claim 5,
In the rendering process,
Wherein the image is rendered to multiply all the pixel values of the selected frame by the wiping matrix B (t) and move the pixel values of the selected frame to the output coordinates. Combined with hardware,
An input process of receiving some selected frames among frames of an input image;
A frame-by-frame path generation step of generating a frame-by-frame camera path for each of the selected frames by decomposing the frame based on a motion model of a correlation matrix for the selected frame;
An optimization process of optimizing paths by applying line fitting or applying a smoothing filter to each camera path for each frame;
A stabilization process for generating a stabilization path after the optimizing paths are restored to a matrix; And
And outputting a stabilized result image by applying the stabilization path to the selected frame
The computer program being stored on a recording medium. A frame input unit for receiving a selected one of the frames of the input image;
A frame decomposing unit for decomposing the frame based on the motion model of the correlation matrix for the selected frame to generate a camera path for each selected frame;
A path optimizing unit for optimizing paths by applying line fitting or applying a smoothing filter to each of the frame-by-frame camera paths;
A restoration unit for restoring the optimization paths to a matrix and generating a stabilization path; And
And outputting a stabilized result image by applying the stabilization path to the selected frame,
Wherein the video playback apparatus comprises:

Images