CN115457448B - Intelligent extraction system for video key frames - Google Patents

Abstract

The invention relates to the technical field of video key frame processing, and discloses an intelligent extraction system of a video key frame, which comprises: the frame extraction module is used for extracting frame units from the animation video; the preprocessing module is used for carrying out binarization processing on the image of the frame unit to obtain a binary image; the contour processing module is used for processing the binary image obtained by the preprocessing module to obtain the contour of the image of the frame unit, one binary image comprises more than one contour, and no intersected pixel point exists between the two contours; a sequence generation module that maps the image of a frame unit to a set of sequences based on its contour; the first screening module, the second screening module and the third screening module screen key frames from the extracted frame units based on the outlines based on a preset strategy; the key frame extraction system based on the image contour is designed based on the production characteristics of the animation video, and the effect superior to that of the key frame extraction method based on the content can be obtained aiming at the animation video.

Description

An Intelligent Extraction System of Video Key Frames

technical field

The invention relates to the technical field of animation video processing, and more specifically, it relates to an intelligent video key frame extraction system.

Background technique

There is a big difference between animation video and ordinary video in the production method and material source. The content contained in animation video is more imaginative and more beyond reality. Therefore, the content-based key frame extraction method applied to general video is applied to animation Video keyframe extraction doesn't work well.

Contents of the invention

The invention provides a video key frame intelligent extraction system, which solves the technical problem in the related art that the content-based key frame extraction method of general video is applied to the key frame of animation video and has poor effect.

According to one aspect of the present invention, a kind of video key frame intelligent extraction system is provided, comprising:

Frame extraction module, it is used for extracting frame unit from animation video;

A preprocessing module, which performs binarization processing on the image of the frame unit to obtain a binary image;

A contour processing module, which obtains the contour of the image of the frame unit based on the binary image processing obtained by the preprocessing module, a binary image includes more than one contour, and there is no intersecting pixel between the two contours;

a sequence generation module that maps frame-unit images to sequence collections based on their contours;

The first screening module, which is used to calculate the distance between the outline of the image of the frame unit and the outline of the image border of the frame unit, deletes the outline of the image of the frame unit whose distance is less than the first threshold, and generates the first outline set;

a second screening module for calculating distances of contours within the first contour set;

The first contour set is sorted to generate the second contour set through the following steps:

Step S101, generating a first intermediate set from the first contour set, the distance between the contours in the first intermediate set and more than one contour in the same first intermediate set is less than a second threshold;

Step S102, the first intermediate set is processed, the first intermediate set belongs to the same episode of an animation drama series, and only one of the contours whose time value difference is smaller than the third threshold value is retained, and all the contours retained in the first intermediate set are The set is the second contour set;

The third screening module generates an inner contour set and an outer contour set based on the second contour set, and the conditions to be satisfied by the contour of the inner contour set are:

For an A contour of the inner contour set, there are more than two contours in the image of the frame unit associated with the A contour, and there are more than one contours outside the A contour in the image;

The conditions to be met by the contours of the outer contour set are:

For a B contour of the outer contour set, there are more than two contours in the image of the frame unit associated with the B contour, and there is no contour outside the B contour in the image;

The first frame unit processing module generates the first frame set after extracting the frame unit associated with the second contour set;

Extracting frame units that do not contain contours of the outer contour set and the inner contour set from the first frame set to generate a second frame set;

Extracting frame units that do not belong to the second frame set from the first frame set to generate a third frame set;

Extract the contours satisfying the following conditions from the outer contour set to generate the third contour set:

The number of contours whose distance to the extracted contour is smaller than the sixth threshold in the outer contour set is greater than the fifth threshold;

Extract the outline of the image belonging to the same frame unit as the outline of the third outline set from the inner layer outline set to generate a fourth outline set;

A small set is generated from the fourth contour set, the distance between the contours in the small set and more than one other contours of the same small set is less than the seventh threshold, and the distance between the contours in the small set and the contours outside the small set is greater than or equal to the first seven thresholds;

A profile is randomly selected from each small set in the fourth profile set to generate a fifth profile set;

Extracting frame units associated with contours of the fifth contour set from the third frame set to generate a fourth frame set;

A key frame generation module, which combines the second frame set and the fourth frame set to obtain a key frame set, and the frame unit in the key frame set is the extracted key frame.

Further, when the contour processing module generates the contour of the image of the frame unit, the contour whose distance from the border contour of the frame unit of the image is smaller than a reference threshold is deleted, and the reference threshold is smaller than the first threshold.

Further, the sequence generation module maps the outline of the image frame of the frame unit to a sequence set in advance.

Further, a sequence set of a contour is A={a ₁ ,a ₂ ... a _n }; a _n represents a pixel of the contour.

Further, there is no intersection between the outer contour set and the inner contour set, and the union of the outer contour set and the inner contour set is less than or equal to the second contour set.

Further, the second screening module further processes the second contour set, including the following steps:

Step S201, generating a second intermediate set from the second contour set, the distance between the contours in the second intermediate set and more than one contour in the same second intermediate set is less than a second threshold;

Step S202, process the second intermediate set, and only keep one contour in a second intermediate set;

The set of all the remaining contours in the second intermediate set is used as the new second contour set.

Further, the difference between the time values in step S102 is less than 1s.

Further, the method for calculating the distance between the first screening module and the second screening module includes:

Calculate the distance between the sequence units of the two contour sequence sets to obtain the first distance matrix;

The element u _ij of the first distance matrix represents the distance between the i-th sequence unit of a profile sequence set and the j-th sequence unit of another profile sequence set;

The formula for calculating the distance of a contour is as follows:

Among them, n and m respectively represent the total number of sequence units of the sequence sets of the two profiles, and d _nm represents the value of the element in the nth row and mth column of the first distance matrix.

Further, the calculation formula for the distance between the i-th sequence unit of a profile sequence set and the j-th sequence unit of another profile sequence set is as follows:

Where x _i and y _i are the two coordinates of the i-th sequence unit, x _j and y _j are the two coordinates of the j-th sequence unit;

d _ij is the value of element u _ij of the first distance matrix.

Further, a core frame extraction module is also included, which is used to perform the following steps to generate a core frame set:

，其中R为关键帧集的关键帧的数量；Step S301, establishing an N*N matrix,

, where R is the number of keyframes in the keyframe set;

Step S302, randomly selecting N*N key frames from the key frame set as elements of the matrix;

Step S303, randomly select a key frame that does not belong to an element of the matrix from the key frame set, then calculate the similarity between the key frame and the matrix element, and select the matrix element with the largest similarity as the central element;

Mark the central element and the elements in the matrix whose distance from the central element is less than 2 as the marked element;

The calculation formula for the distance of the elements of the matrix is as follows:

Among them, u _H1 and u _H2 are row values of two matrix elements respectively, u _L1 and u _L2 are column values of two matrix elements respectively;

Step S304, updating the attribute values of the tag elements in sequence, the updating formula is as follows:

表示更新之后的标记元素的第k项属性的值，

表示更新之前的标记元素的第k项属性的值，

表示步骤S303中提取的关键帧的第k项属性的值；where t represents the number of times the tagged element is updated,

Indicates the value of the kth attribute of the tag element after the update,

Indicates the value of the kth attribute of the tag element before updating,

Represent the value of the kth item attribute of the key frame extracted in step S303;

k≤3, the three attributes of the key frame are the first attribute, the second attribute and the third attribute; the value of the first attribute is the number of contours contained in the key frame; the value of the second attribute is the number of contours contained in the key frame The number of inner contours; the value of the third attribute is the playback time value of the key frame;

，R为关键帧集的关键帧的数量；Step S305, iteratively execute step S303 and step S304, the number of iterative execution is

, R is the number of keyframes in the keyframe set;

Step S306 , extracting key frames in the key frame set whose element similarity with the matrix is greater than the eighth threshold as core frames, and generating a core frame set.

The beneficial effects of the present invention are:

The present invention designs a key frame extraction system based on image contours based on the production characteristics of animation videos, and can obtain key frame extraction effects better than content-based key frame extraction methods for animation videos.

Description of drawings

Fig. 1 is a module schematic diagram one of a kind of video key frame intelligent extraction system of the present invention;

Fig. 2 is the flow chart that arranges the first outline set and generates the second outline set according to the present invention;

Fig. 3 is the flow chart that the second screening module of the present invention further processes the second contour set;

Fig. 4 is the module schematic diagram two of a kind of video key frame intelligent extraction system of the present invention;

Fig. 5 is a flowchart of generating a core frame set by the core frame extraction module of the present invention.

In the figure: frame extraction module 101, preprocessing module 102, contour processing module 103, sequence generation module 104, first screening module 105, second screening module 106, third screening module 107, first frame unit processing module 108, key Frame generation module 109, core frame extraction module 110.

Detailed ways

The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and realize the subject matter described herein, and is not intended to limit the protection scope, applicability or examples set forth in the claims. Changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as needed. Additionally, features described with respect to some examples may also be combined in other examples.

Embodiment one

As shown in Figures 1-3, a video key frame intelligent extraction system includes:

Frame extraction module 101, it is used for extracting frame unit from animation video;

A preprocessing module 102, which performs binarization processing on the image of the frame unit to obtain a binary image;

Contour processing module 103, it obtains the contour of the image of the frame unit based on the binary image processing obtained by the preprocessing module 102, a binary image includes more than one contour, and there is no intersecting pixel point between the two contours;

a sequence generation module 104, which maps the frame-unit images to sequence sets based on their contours;

For example, the sequence set of a contour is A={a ₁ ,a ₂ ... a _n }; a _n represents a pixel of the contour;

The first screening module 105, which is used to calculate the distance between the outline of the image of the frame unit and the outline of the image border of the frame unit, delete the outline of the image of the frame unit whose distance is less than the first threshold, and generate the first outline set;

A second screening module 106, which is used to calculate the distance of the contours in the first contour set;

The first contour set is sorted to generate the second contour set through the following steps:

Step S101, generating a first intermediate set from the first contour set, the distance between the contours in the first intermediate set and more than one contour in the same first intermediate set is less than a second threshold;

Step S102, the first intermediate set is processed, the first intermediate set belongs to the same episode of an animation drama series, and only one of the contours whose time value difference is smaller than the third threshold value is retained, and all the contours retained in the first intermediate set are The set is the second contour set;

In one embodiment of the present invention, the second screening module 106 further processes the second contour set, including the following steps:

Step S201, generating a second intermediate set from the second contour set, the distance between the contours in the second intermediate set and more than one contour in the same second intermediate set is less than a second threshold;

Step S202, process the second intermediate set, and only keep one contour in a second intermediate set;

The set of all the remaining contours in the second intermediate set is used as the new second contour set.

Aiming at the characteristics of reusing the original paintings of animation series, the above strategy is used to delete the frame units generated based on the reuse of the original paintings, but still retain a relatively similar outline; here, the time value is standardized, for example, 10 minutes 23 of a certain episode second, then its time value is 623.03, and the 20th frame of 10 minutes and 22 seconds of a certain episode, then its time value is 622.20, the difference between the time values in step S102 is generally selected to be less than 1, generally the first The second threshold is set to be small, and this condition is to clean up the repeated frame insertion to meet the format of the 24-frame image.

The third screening module 107, which generates an inner contour set and an outer contour set based on the second contour set, the conditions to be met by the contour of the inner contour set are:

For an A contour of the inner contour set, there are more than two contours in the image of the frame unit associated with the A contour, and there are more than one contours outside the A contour in the image;

The conditions to be met by the contours of the outer contour set are:

For a B contour of the outer contour set, there are more than two contours in the image of the frame unit associated with the B contour, and there is no contour outside the B contour in the image;

There is no intersection between the outer contour set and the inner contour set, and the union of the outer contour set and the inner contour set is less than or equal to the second contour set.

The first frame unit processing module 108, which generates the first frame set after extracting the frame unit associated with the second contour set;

Extracting frame units that do not contain contours of the outer contour set and the inner contour set from the first frame set to generate a second frame set;

Extracting frame units that do not belong to the second frame set from the first frame set to generate a third frame set;

Extract the contours satisfying the following conditions from the outer contour set to generate the third contour set:

The number of contours whose distance to the extracted contour is smaller than the sixth threshold in the outer contour set is greater than the fifth threshold;

Extract the outline of the image belonging to the same frame unit as the outline of the third outline set from the inner layer outline set to generate a fourth outline set;

A small set is generated from the fourth contour set, the distance between the contours in the small set and more than one other contours of the same small set is less than the seventh threshold, and the distance between the contours in the small set and the contours outside the small set is greater than or equal to the first seven thresholds;

A profile is randomly selected from each small set in the fourth profile set to generate a fifth profile set;

Extracting frame units associated with contours of the fifth contour set from the third frame set to generate a fourth frame set;

A key frame generation module 109, which combines the second frame set and the fourth frame set to obtain a key frame set, and the frame unit in the key frame set is the extracted key frame;

The method for calculating the distance between the first screening module 105 and the second screening module 106 includes:

Calculate the distance between the sequence units of the two contour sequence sets to obtain the first distance matrix;

The element u _ij of the first distance matrix represents the distance between the i-th sequence unit of a profile sequence set and the j-th sequence unit of another profile sequence set;

The formula for calculating the distance of a contour is as follows:

Among them, n and m respectively represent the total number of sequence units of the sequence sets of the two profiles, and d _nm represents the value of the element of the nth row and mth column of the first distance matrix;

The values of D(n-1, m), D(n, m-1), and D(n-1, m-1) are iteratively calculated with reference to the above formula;

For the magnification lens commonly used in animation, it can also calculate the similarity of the outline well;

In one embodiment of the present invention, the formula for calculating the distance between the i-th sequence unit of one profile sequence set and the j-th sequence unit of another profile sequence set is as follows:

Where x _i and y _i are the two coordinates of the i-th sequence unit, x _j and y _j are the two coordinates of the j-th sequence unit;

d _ij is the value of element u _ij of the first distance matrix;

In one embodiment of the present invention, the sequence generation module 104 maps the image frame outline of the frame unit to a sequence set in advance. When extracting the outline of the image of the frame unit, the image frame outline is not included, and the images of all frame units The bounding box outlines should be the same, so the image bounding box outlines for frame units can be pre-generated individually.

In one embodiment of the present invention, when the outline processing module 103 generates the outline of the image of the frame unit, it deletes the outline whose distance from the image border outline of the frame unit is less than the reference threshold, the reference threshold is less than the first threshold, and the pixels belonging to the image border outline The points are distributed on the border of the image of the frame unit;

In one embodiment of the present invention, the profile is stored in the form of chain code.

As shown in Fig. 4 and Fig. 5, in one embodiment of the present invention, a kind of video key frame intelligent extraction system also comprises core frame extraction module 110, and it is used to perform the following steps to generate core frame set:

，其中R为关键帧集的关键帧的数量；Step S301, establishing an N*N matrix,

, where R is the number of keyframes in the keyframe set;

Step S302, randomly selecting N*N key frames from the key frame set as elements of the matrix;

Step S303, randomly select a key frame that does not belong to an element of the matrix from the key frame set, then calculate the similarity between the key frame and the matrix element, and select the matrix element with the largest similarity as the central element;

Mark the central element and the elements in the matrix whose distance from the central element is less than 2 as the marked element;

The calculation formula for the distance of the elements of the matrix is as follows:

Among them, u _H1 and u _H2 are row values of two matrix elements respectively, u _L1 and u _L2 are column values of two matrix elements respectively;

Step S304, updating the attribute values of the tag elements in sequence, the updating formula is as follows:

表示更新之后的标记元素的第k项属性的值，

表示更新之前的标记元素的第k项属性的值，

表示步骤S303中提取的关键帧的第k项属性的值；where t represents the number of times the tagged element is updated,

Indicates the value of the kth attribute of the tag element after the update,

Indicates the value of the kth attribute of the tag element before updating,

Represent the value of the kth item attribute of the key frame extracted in step S303;

k≤3, the three attributes of the key frame are the first attribute, the second attribute and the third attribute; the value of the first attribute is the number of contours contained in the key frame; the value of the second attribute is the number of contours contained in the key frame The number of inner contours; the value of the third attribute is the playback time value of the key frame;

，R为关键帧集的关键帧的数量；Step S305, iteratively execute step S303 and step S304, the number of iterative execution is

, R is the number of keyframes in the keyframe set;

Step S306, extracting the key frames in the key frame set whose element similarity with the matrix is greater than the eighth threshold as core frames to generate a core frame set;

It should be noted that the key frame selected in step S303 should not be selected in the previous iterative execution steps;

The matrix element described in step S306 refers to the updated matrix after step S305 iteration;

Among them, the formula for calculating the similarity between keyframes and matrix elements is as follows:

Among them, u ₁₁ , u ₁₂ , and u ₁₃ are respectively the values of the first attribute, the second attribute, and the third attribute of the matrix elements, and u ₂₁ , u ₂₂ , and u ₂₃ are respectively the values of the first attribute, the second attribute, and the third attribute of the key frame. The value of the three attributes;

The key frame set generated by the key frame generation module 109 still has a large amount of data for a larger video with a total number of frames, and the core frame extraction module 110 performs further processing based on the characteristics of the key frame set generated by the key frame generation module Scale down for fewer core frames.

The embodiment of this embodiment has been described above, but this embodiment is not limited to the above-mentioned specific implementation, the above-mentioned specific implementation is only illustrative, not restrictive, those of ordinary skill in the art Inspired by the embodiment, without departing from the gist of the embodiment and the protection scope of the claims, many forms can be made, all of which are within the protection of the embodiment.

Claims (10)

1. A video key frame intelligent extraction system is characterized in that, comprising: Frame extraction module, it is used for extracting frame unit from animation video; A preprocessing module, which performs binarization processing on the image of the frame unit to obtain a binary image; A contour processing module, which obtains the contour of the image of the frame unit based on the binary image processing obtained by the preprocessing module, a binary image includes more than one contour, and there is no intersecting pixel between the two contours; a sequence generation module that maps frame-unit images to sequence collections based on their contours; The first screening module, which is used to calculate the distance between the outline of the image of the frame unit and the outline of the image border of the frame unit, deletes the outline of the image of the frame unit whose distance is less than the first threshold, and generates the first outline set; a second screening module for calculating distances of contours within the first contour set; The first contour set is sorted to generate the second contour set through the following steps: Step S101, generating a first intermediate set from the first contour set, the distance between the contours in the first intermediate set and more than one contour in the same first intermediate set is less than a second threshold; Step S102, the first intermediate set is processed, the first intermediate set belongs to the same episode of an animation drama series, and only one of the contours whose time value difference is smaller than the third threshold value is retained, and all the contours retained in the first intermediate set are The set is the second contour set; The third screening module generates an inner contour set and an outer contour set based on the second contour set, and the conditions to be satisfied by the contour of the inner contour set are: For an A contour of the inner contour set, there are more than two contours in the image of the frame unit associated with the A contour, and there are more than one contours outside the A contour in the image; The conditions to be met by the contours of the outer contour set are: For a B contour of the outer contour set, there are more than two contours in the image of the frame unit associated with the B contour, and there is no contour outside the B contour in the image; The first frame unit processing module generates the first frame set after extracting the frame unit associated with the second contour set; Extracting frame units that do not contain contours of the outer contour set and the inner contour set from the first frame set to generate a second frame set; Extracting frame units that do not belong to the second frame set from the first frame set to generate a third frame set; Extract the contours satisfying the following conditions from the outer contour set to generate the third contour set: The number of contours whose distance to the extracted contour is smaller than the sixth threshold in the outer contour set is greater than the fifth threshold; Extract the outline of the image belonging to the same frame unit as the outline of the third outline set from the inner layer outline set to generate a fourth outline set; A small set is generated from the fourth contour set, the distance between the contours in the small set and more than one other contours of the same small set is less than the seventh threshold, and the distance between the contours in the small set and the contours outside the small set is greater than or equal to the first seven thresholds; A profile is randomly selected from each small set in the fourth profile set to generate a fifth profile set; Extracting frame units associated with contours of the fifth contour set from the third frame set to generate a fourth frame set; A key frame generation module, which combines the second frame set and the fourth frame set to obtain a key frame set, and the frame unit in the key frame set is the extracted key frame. 2. a kind of video key frame intelligent extraction system according to claim 1, is characterized in that, when contour processing module generates the contour of the image of frame unit, deletes and the distance of the image frame contour of frame unit is less than the contour of benchmark threshold value, benchmark The threshold is less than the first threshold. 3. A video key frame intelligent extraction system according to claim 2, characterized in that the sequence generation module maps the outline of the image frame of the frame unit to a sequence set in advance. 4. A kind of video key frame intelligent extraction system according to claim 1, is characterized in that, the sequence set of a profile is A={a ₁ , a ₂ ... a _n }; a _n represents a pixel of the profile . 5. A kind of video key frame intelligent extraction system according to claim 1, is characterized in that, there is not intersection between outer layer outline set and inner layer outline set, and the union of outer layer outline set and inner layer outline set is less than or equal to the second contour set. 6. A kind of video key frame intelligent extraction system according to claim 1, is characterized in that, the second screening module carries out further processing to the second outline set, comprises the following steps: Step S201, generating a second intermediate set from the second contour set, the distance between the contours in the second intermediate set and more than one contour in the same second intermediate set is less than a second threshold; Step S202, process the second intermediate set, and only keep one contour in a second intermediate set; The set of all the remaining contours in the second intermediate set is used as the new second contour set. 7. A video key frame intelligent extraction system according to claim 1, characterized in that the difference between the time values in step S102 is less than 1s. 8. A kind of video key frame intelligent extraction system according to claim 1, is characterized in that, the method for the distance of the first screening module and the second screening module calculation contour comprises: Calculate the distance between the sequence units of the two contour sequence sets to obtain the first distance matrix; The element u _ij of the first distance matrix represents the distance between the i-th sequence unit of a profile sequence set and the j-th sequence unit of another profile sequence set; The formula for calculating the distance of a contour is as follows:

Among them, n and m respectively represent the total number of sequence units of the sequence sets of the two profiles, and d _nm represents the value of the element in the nth row and mth column of the first distance matrix. 9. A kind of video key frame intelligent extraction system according to claim 8, is characterized in that, the calculation of the distance of the ith sequence unit of the sequence set of a profile and the j sequence unit of another profile sequence set The formula is as follows:

Where x _i and y _i are the two coordinates of the i-th sequence unit, x _j and y _j are the two coordinates of the j-th sequence unit; d _ij is the value of element u _ij of the first distance matrix. 10. a kind of video key frame intelligent extraction system according to claim 1, is characterized in that, also comprises core frame extraction module, and it is used for performing the following steps to generate core frame set: Step S301, establishing an N*N matrix,

, where R is the number of keyframes in the keyframe set; Step S302, randomly selecting N*N key frames from the key frame set as elements of the matrix; Step S303, randomly select a key frame that does not belong to an element of the matrix from the key frame set, then calculate the similarity between the key frame and the matrix element, and select the matrix element with the largest similarity as the central element; Mark the central element and the elements in the matrix whose distance from the central element is less than 2 as the marked element; The calculation formula for the distance of the elements of the matrix is as follows:

Among them, u _H1 and u _H2 are row values of two matrix elements respectively, u _L1 and u _L2 are column values of two matrix elements respectively; Step S304, updating the attribute values of the tag elements in sequence, the updating formula is as follows:

where t represents the number of times the tagged element is updated,

Indicates the value of the kth attribute of the tag element after the update,

Indicates the value of the kth attribute of the tag element before updating,

Represent the value of the kth item attribute of the key frame extracted in step S303; k≤3; The three attributes of the key frame are the first attribute, the second attribute and the third attribute; the value of the first attribute is the number of contours contained in the key frame; the value of the second attribute is the number of inner contours contained in the key frame ;The value of the third attribute is the play time value of the key frame; Step S305, iteratively execute step S303 and step S304, the number of iterative execution is

, R is the number of keyframes in the keyframe set; Step S306 , extracting key frames in the key frame set whose element similarity with the matrix is greater than the eighth threshold as core frames, and generating a core frame set.

Images